diff --git a/.claude/skills/smoke-regression/SKILL.md b/.claude/skills/smoke-regression/SKILL.md new file mode 100644 index 000000000..c0e9ab21d --- /dev/null +++ b/.claude/skills/smoke-regression/SKILL.md @@ -0,0 +1,88 @@ +--- +name: smoke-regression +description: >- + Run the smoke and regression pass for this stack after a change: the hermetic + suites, the turn-matrix script that drives real agent turns and asserts every + SSE code path (frame order, interrupt/resume, 409, Stop, preview, restore, + cache stability), and the in-app-browser checklist for rendered surfaces. + Use when the user says "smoke test", "regression test", "run the smoke pass", + "verify this on dev", "test all the response paths", "did we break + streaming", or before a release. Points at docs/testing/smoke-regression.md + as the single source of truth for rows, fixtures and failure reading. +--- + +# Smoke / regression pass + +The matrix, its fixtures and how to read a failure live in +[docs/testing/smoke-regression.md](../../../docs/testing/smoke-regression.md). +Read §2.2 before running L1 and §3 before L2. This skill is the procedure. + +## 0. Decide the layers + +From the diff, pick per §4 of the doc: + +- **L0** always, for the packages touched. +- **L1** when the change touches `inference_api/chat/*`, `agents/main_agent/**`, + `apis/shared/harness/**`, the SSE table, or any interrupt flow. +- **L2** when the change touches a rendered surface (composer, loader, tool + rail, pickers, banners) or an interrupt flow. + +Say which layers you are running and why in one line before starting. + +## 1. L0 — hermetic + +From the branch's checkout. In a worktree `PYTHONPATH` is mandatory or you test +`develop`: + +```bash +cd /backend && PYTHONPATH=$PWD/src AWS_EC2_METADATA_DISABLED=true
/backend/.venv/bin/python -m pytest tests/ -q +``` + +Frontend `cd frontend/ai.client && npx ng test --watch=false`; infra +`cd infrastructure && npx tsc --noEmit && npx jest`. Never pass dummy AWS +credentials to make a stall go away (see the doc §1). + +## 2. L1 — turn matrix + +Pick the target in this order and state which one you used: + +1. **Deployed dev through the Runtime** (always available to an agent with the + `dev-ai` profile and an active headless grant): + ```bash + cd backend && AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant --user-id --prefix dev-boisestateai-v2 --json /tmp/smoke.json + ``` + The grant's owner id is in `dev-boisestateai-v2-bff-sessions` under + `PK=HEADLESS-GRANT#…` (`grant_user_id`). If the script reports no active + grant, the user must sign in and run a headless run once; do not try to + mint tokens another way (the doc explains why the other routes are dead ends). + This mode skips the app-api-only rows (409, Stop, steer). It deletes the + sessions it created; after a crash or `--keep`, pass `--delete-sessions id,id`. +2. **Local stack** when the branch is not deployed yet and you need the + app-api rows: app-api on :8000 with `SKIP_AUTH=true`, inference-api on + :8001, then `--base-url http://127.0.0.1:8000`. Do not flip `SKIP_AUTH` in + the user's `backend/src/.env` without asking; use a worktree copy. +3. **Deployed through app-api** with a Cognito password user + (`--auth cognito`), when one exists for the environment. + +Add `--with-attachments` when attachments are in scope, and the fixture flags +(`--approval-tool`, `--skill-id`, `--agent-id`) when those rows matter. Never +pass `--allow-prod`. + +Paste the report table into your summary. Treat SKIP as "not covered here", +OBSERVED as information, WARN as a model-quality note. A FAIL is a finding: +quote its detail, check the three traps in the doc §2.2 before calling it a +bug, then fix or report. + +## 3. L2 — in-app browser + +Use the built-in browser pane on `https://dev.boisestate.ai`; it carries the +user's session. Install the frame-capture snippet from the doc §3 with +`javascript_tool` first, then work through the checklist rows that apply. +Verify with `read_page` and `window.__frames`, not screenshots, except for a +final visual proof. Mirror the CSRF cookie into `X-CSRF-Token` for any manual +non-GET call. Delete the smoke conversations when done. + +## 4. Report + +One table per layer run, a one-line verdict, and the exact commands used so +the run is reproducible. If a layer was skipped, say which and why. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 64cb14e39..81a8fba0b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,9 +3,22 @@ name: CI # Pre-merge test gate. The deploy workflows (backend.yml, platform.yml, # frontend-deploy.yml) only run on push to develop/main — they run the # test jobs as a gate *before deploying*, so unit tests never ran on the -# PR itself. This workflow runs the full test suites on every pull request -# into develop/main, with no deploy/build/AWS steps, so regressions are -# caught before merge. Deploys remain push-only. +# PR itself. This workflow runs the test suites on every pull request into +# develop/main, with no deploy/build/AWS steps, so regressions are caught +# before merge. Deploys remain push-only. +# +# Which suites run is decided per PR by the `changes` job from the PR's file +# list (scripts/ci/classify-changes.sh): a docs-only PR runs nothing, a +# frontend-only PR runs the two SPA jobs, an infrastructure PR runs jest plus +# the backend suites that read the CDK tree. The mapping is fail-open — an +# unrecognised path, a workflow_dispatch, or a failed file-list call runs +# everything — and is pinned by backend/tests/supply_chain/test_ci_path_filter.py. +# +# The filtering lives on the JOBS, not on the trigger's `paths:`. A workflow +# that does not trigger leaves a required status check "Expected" forever +# and blocks the merge; a job that is skipped reports as skipped and does +# not. develop has no required checks today, but this shape survives adding +# them. on: workflow_dispatch: pull_request: @@ -23,25 +36,71 @@ concurrency: cancel-in-progress: true jobs: + # Classify the PR's changed files into the suites that can observe them. + # The file list comes from the REST API rather than a checkout diff: no + # fetch-depth games, no third-party action, and it is exact for renames + # (both names are fed through). Any failure to obtain the list yields an + # empty stdin, which the classifier treats as "run everything". + changes: + name: Classify changed paths + runs-on: ubuntu-24.04 + permissions: + contents: read + pull-requests: read + outputs: + backend: ${{ steps.classify.outputs.backend }} + backend_contracts: ${{ steps.classify.outputs.backend_contracts }} + frontend: ${{ steps.classify.outputs.frontend }} + infra: ${{ steps.classify.outputs.infra }} + load: ${{ steps.classify.outputs.load }} + scripts: ${{ steps.classify.outputs.scripts }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + sparse-checkout: scripts/ci + - name: Classify + id: classify + env: + GH_TOKEN: ${{ github.token }} + EVENT_NAME: ${{ github.event_name }} + REPO: ${{ github.repository }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + files="" + if [ "$EVENT_NAME" = "pull_request" ]; then + files=$(gh api "repos/$REPO/pulls/$PR_NUMBER/files" --paginate \ + --jq '.[] | .filename, (.previous_filename // empty)') || files="" + echo "$(printf '%s\n' "$files" | grep -c .) changed path(s) on PR #$PR_NUMBER" + else + echo "$EVENT_NAME is not a pull request: running every suite" + fi + printf '%s\n' "$files" | bash scripts/ci/classify-changes.sh | tee -a "$GITHUB_OUTPUT" + tests: name: Tests + needs: changes permissions: contents: read uses: ./.github/workflows/tests.yml with: - run_backend: true - run_frontend: true + run_backend: ${{ needs.changes.outputs.backend == 'true' }} + # The supply-chain/architecture subset, for PRs that touch what those + # tests read (CDK, scripts, workflows) without touching backend/. A + # no-op inside tests.yml when run_backend is also true. + run_backend_contracts: ${{ needs.changes.outputs.backend_contracts == 'true' }} + run_frontend: ${{ needs.changes.outputs.frontend == 'true' }} # The frontend suite again under --coverage — the nightly's invocation. # Coverage changes how the specs are built (see tests.yml), so specs can # pass here and fail there. Only the PR gate pays for it; the deploy # workflows keep running the plain suite. - run_frontend_coverage: true - run_infra: true + run_frontend_coverage: ${{ needs.changes.outputs.frontend == 'true' }} + run_infra: ${{ needs.changes.outputs.infra == 'true' }} # Pure-logic tests for tests/load. Deliberately only on the PR gate: the # deploy workflows run test suites to protect a deploy, and the load # suite has no bearing on one. - run_load_unit: true + run_load_unit: ${{ needs.changes.outputs.load == 'true' }} # Offline unit tests for scripts/restore-data. PR gate only, like the # load suite: a deploy gains nothing from them. - run_scripts: true + run_scripts: ${{ needs.changes.outputs.scripts == 'true' }} diff --git a/.github/workflows/nightly-deploy-pipeline.yml b/.github/workflows/nightly-deploy-pipeline.yml index 7b94c786c..408ed3fa7 100644 --- a/.github/workflows/nightly-deploy-pipeline.yml +++ b/.github/workflows/nightly-deploy-pipeline.yml @@ -11,7 +11,8 @@ # 3. deploy-*-code — API-driven code deploys (ECS / AgentCore Runtime # / Lambda) onto the freshly-deployed stack. # 4. deploy-frontend — build SPA + S3 sync + CloudFront invalidation. -# 5. smoke-test / e2e-tests — validate the deployment. +# 5. smoke-test / e2e-tests — validate the deployment (e2e-tests also runs +# the turn-path smoke matrix, scripts/nightly/turn-smoke.sh, non-blocking). # 6. teardown — delete the ephemeral stack + buckets/secrets/logs. # # Ephemeral deploys run with NO custom domain (CDK_DOMAIN_NAME=""); the SPA, @@ -504,6 +505,17 @@ jobs: aws-role-arn: ${{ secrets.AWS_ROLE_ARN }} aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + # uv is needed before the E2E run: e2e-test.sh seeds the E2E users and + # (via backend/scripts/seed_e2e_role.py) the e2e_user RBAC role, and the + # turn smoke below reuses the same environment. + - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + with: + version: '0.7.12' + enable-cache: true + cache-dependency-glob: 'backend/uv.lock' + - name: Sync backend deps + working-directory: backend + run: uv sync --extra agentcore --frozen - name: Run E2E tests run: bash scripts/nightly/e2e-test.sh - name: Upload Playwright report @@ -513,6 +525,31 @@ jobs: name: playwright-report-${{ inputs.label }} path: frontend/ai.client/playwright-report/ retention-days: 30 + # ── Turn-path smoke matrix (docs/testing/smoke-regression.md §2) ── + # Real agent turns through the stack's app-api as the E2E user, asserting + # the SSE contract, interrupt/resume, 409, Stop, preview, attachments, + # restore and cache-hash stability. Runs after the Playwright step because + # e2e-test.sh has by then patched CORS and the Cognito callback list for + # the CloudFront URL, which the scripted Hosted UI login depends on. + # Non-blocking (continue-on-error) until it has a green track record on + # the nightly stack; promote by removing that line. The E2E user's + # `calculator` / `ask_user_question` grants come from the e2e_user role + # that e2e-test.sh seeds; a row that cannot run reports SKIP, never FAIL. + - name: Run turn smoke matrix + id: turn-smoke + if: always() + continue-on-error: true + env: + TURN_SMOKE_REPORT: ${{ github.workspace }}/turn-smoke-report.json + run: bash scripts/nightly/turn-smoke.sh + - name: Upload turn smoke report + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: turn-smoke-report-${{ inputs.label }} + path: turn-smoke-report.json + retention-days: 30 + if-no-files-found: ignore # ── Auto-teardown ────────────────────────────────────────────────────────── # ALWAYS runs (unless skip-teardown), even if any deploy/test job above failed diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 422505cc2..fba4bb96f 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -16,6 +16,11 @@ on: required: false default: false type: boolean + run_backend_contracts: + description: 'Run only the backend suites that read the CDK tree, scripts and workflows (supply_chain, architecture); a no-op when run_backend is also set' + required: false + default: false + type: boolean run_frontend: description: 'Run frontend test suite' required: false @@ -67,10 +72,13 @@ jobs: run: uv sync --extra agentcore --extra dev - name: Run pytest working-directory: backend - # `-n auto` fans the suite across all runner cores (pytest-xdist). - # Verbosity/--tb live in backend/pytest.ini. The nightly coverage run - # (scripts/backend/test.sh) is deliberately left serial. - run: uv run pytest tests/ -n auto + # `-n logical` fans the suite across every vCPU (pytest-xdist). Not + # `auto`: with psutil installed, `auto` counts PHYSICAL cores, and the + # 4-vCPU runner reports 2, so it started 2 workers for ~11k tests + # (`created: 2/2 workers` in the log). Verbosity/--tb live in + # backend/pytest.ini. The nightly coverage run (scripts/backend/test.sh) + # is deliberately left serial. + run: uv run pytest tests/ -n logical # The repo-ROOT supply-chain suite (not backend/tests/supply_chain): # cross-package contracts that read CDK, the backup/restore scripts and # the python env-var readers side by side, so it lives above backend/. @@ -81,6 +89,38 @@ jobs: - name: Run root supply-chain pytest run: uv run --project backend --no-sync pytest tests/supply_chain/ -v + # The subset of the backend suite that reads OUTSIDE backend/: the + # supply-chain and architecture tests assert against the CDK tree, the + # scripts and the workflows side by side with the Python that consumes + # them, so an infrastructure-only PR still needs them — and nothing else in + # backend/tests/. About a hundred tests, well under a minute, against seven + # minutes for the full suite. Skipped whenever run_backend is set, because + # the full run already covers it. + test-backend-contracts: + if: ${{ inputs.run_backend_contracts && !inputs.run_backend }} + name: Test backend contracts (pytest) + runs-on: ubuntu-24.04 + permissions: + contents: read + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + ref: ${{ inputs.ref }} + persist-credentials: false + - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + with: + version: '0.7.12' + enable-cache: true + cache-dependency-glob: 'backend/uv.lock' + - name: Sync deps + working-directory: backend + run: uv sync --extra agentcore --extra dev + - name: Run backend contract suites + working-directory: backend + run: uv run pytest tests/supply_chain/ tests/architecture/ -v + - name: Run root supply-chain pytest + run: uv run --project backend --no-sync pytest tests/supply_chain/ -v + test-frontend: if: ${{ inputs.run_frontend }} name: Test frontend diff --git a/CHANGELOG.md b/CHANGELOG.md index 2412bf560..87e8f64df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,76 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.26.0] - 2026-10-03 + +The first token arrives sooner. Every stage between the request and the first model output is now mapped and timed, and the largest stages we own were taken off the critical path: a cold agent build drops from ~870ms to ~370ms, a long conversation no longer pays ~390ms to count its history, external MCP servers load concurrently, and an agent's knowledge-base search overlaps the build. Voice mode is **priced for the first time**, per modality, and users can choose which Nova 2 Sonic voice answers and which microphone listens. Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna and GPT-5.5 join the catalog. Managed knowledge bases stop mis-counting imports and stop serving stale synced files. A smoke and regression pass now drives real turns and runs nightly. Strands moves to 1.57.2 with bedrock-agentcore 1.24.0. **A CDK deploy is required, and voice stays unmetered until an admin adds the Nova 2 Sonic catalog row** (see the release notes). + +### 🚀 Added + +- **Voice and microphone choice** — the composer's voice button becomes a split control with a menu: all 16 Nova 2 Sonic voices grouped by language (new `voiceId` user setting, unknown ids 422) and the input device (per browser, shared with dictation) (#1399) +- **Voice pricing per modality** — optional `speechInputPricePerMillionTokens` / `speechOutputPricePerMillionTokens` on catalog rows, `CostCalculator.calculate_voice_cost` billing text and speech buckets separately, and a curated **Nova 2 Sonic** row; models whose output includes speech are hidden from the chat picker (#1399) +- **New curated models** — Claude Opus 5.5 (cache read 0.05×), GPT-6 Sol, GPT-6 Luna and GPT-5.5; Claude Opus 4.7, GPT-5.6 Sol and GPT-5.6 Terra move to the picker's More models submenu (#1358) +- **Turn-path smoke and regression pass** — `backend/scripts/smoke_turns.py` drives real turns through `POST /chat/stream` and asserts frame order, restore, interrupts and resume, the single-flight 409, Stop, steering, attachments, citations and prompt-cache stability; runbook in `docs/testing/smoke-regression.md` (#1405) +- `repair_managed_kb_byte_counters.py --settle-unexplained` — opt-in, per-agent repair for a quiet managed KB whose counters drifted from its ledger (#1361) + +### ✨ Improved + +- **Tool rail** shows a tool's full result in a bounded scroll box instead of cutting it at 200 characters; labels sit above values, input renders one `key: value` per line, and scroll boxes are keyboard-focusable with AA-contrast labels (#1394) +- **Conversation titles appear sooner** — polled every 100ms during the stream, persisted in the background, and written with one keyed `update_item` (#1374) +- Dark-mode sidebar date headers, bold text and headings in responses are brighter (#1378) +- On phones, conversation text aligns with the composer instead of sitting ~48px from the edge (#1382) +- **Turn-latency observability** — `agent_build.tools` split into four timed sub-stages with per-server MCP timings (#1396); the turn clock runs to the first model output (`turn_first_token`, `FirstTokenMs`) (#1397, #1398) + +### ⚡ Performance + +- One process-wide boto3 session for the agent build, built at container warm-up: cold `agent_build` 869 → 372ms median on dev. Kill switch `AGENT_BUILD_SHARED_SESSION_ENABLED` (#1377, #1395) +- The per-turn history count is read on a worker thread from the head of the turn: 388 → 0ms at 100+ stored events, first token 1560 → 1092ms on dev. Kill switch `HISTORY_COUNT_PREFETCH_ENABLED` (#1403) +- The long-term-memory lookup overlaps the two message writes: `pre_model` 450 → 270ms warm, 664 → 352ms cold. Kill switch `MEMORY_RETRIEVAL_PREFETCH_ENABLED` (#1400) +- An agent's external MCP servers pre-flight concurrently, so tool loading tracks the slowest server rather than the sum; tool order is unchanged. Kill switch `MCP_PARALLEL_PREFLIGHT_ENABLED` (#1406) +- An agent turn's knowledge-base search starts before the agent build (kill switch `KB_SEARCH_AHEAD_ENABLED`) (#1411), its follow-up reads run off the event loop (#1415), and the KB record is read once per search on a cached table (#1416) +- The agent-binding "does this thread have messages?" check is a one-event existence read and no longer writes a `SESSION` event: first-agent-turn `rag` 955 → 571ms (#1413) +- Side-channel Bedrock callers (tool summaries, compaction, document digest, embeddings) reuse one client instead of building one per call (#1375) + +### 🐛 Fixed + +- Imports, crawls and syncs on managed knowledge bases committed bytes they never reserved, so `totalBytes` never moved and `reservedBytes` could go negative (#1361) +- A synced Drive file or crawled page that changed was never re-ingested on a managed KB, which kept serving the import-time version (#1365) +- A KB sync whose S3 stage failed left its change gates advanced, so the change was never retried (#1373) +- Stop pressed after the turn's `done` marked a finished turn interrupted (#1366); a page-hide in the same window stamped it `navigated_away` (#1370) +- Side-channel calls (compaction summary and extraction, tool summaries, document abstracts, titles) accepted guardrail, content-filter and unknown stop reasons as valid results (#1368) +- Inline spreadsheets and decks from headless callers were dropped while the guidance note said the tools could reach them; they are now stored as session files. Kill switch `INLINE_ATTACHMENT_PERSIST_ENABLED` (#1409) +- The context-meter panel drew under the announcement pill (#1369) +- A failed announcement acknowledgement showed a global error toast and was lost; it is now silent and retried once on a network drop or gateway error (#1383) +- The model picker's chevron never rotated when the menu opened (#1393) +- Voice sessions' `message_count` was inflated about fourfold (#1367) + +### 🔒 Security + +- The voice agent no longer writes the client-supplied `sample_rate` into its logs, where a crafted value could forge a log line (CodeQL `py/log-injection`) (#1425) + +### ⚠️ Changed + +- Imports, crawls and syncs on a managed knowledge base now count against the per-owner and per-KB byte caps like uploads: one that would breach a cap fails with the cap message instead of slipping past (#1361) + +### 📦 Dependencies + +- Backend: `strands-agents` / `strands-agents[bidi]` 1.55.0 → 1.57.2, `bedrock-agentcore` 1.21.0 → 1.24.0, `boto3`/`botocore` 1.43.68 → 1.43.103, `mcp` 1.28.1 → 1.30.0 (still held below 2); voice adapted to the graduated Bidi API with the WebSocket contract unchanged (#1367, #1410) + +### 🏗️ Infrastructure + +- Turn-latency dashboard gains first-token, after-prelude and `agent_build.tools` sub-stage widgets (#1396, #1397) + +### 🔧 CI/CD + +- The PR gate runs only the suites a PR's paths can reach, with 4 xdist workers on the backend job (#1392) +- The nightly pipeline runs the turn-path smoke matrix after E2E, non-blocking, with its report uploaded (#1412, #1414) + +### 📚 Docs + +- `docs/specs/turn-path-ttft.md` maps every stage from the SPA's send to the first token, with the regression plan and dev readouts; the chat route's phases are extracted to match (#1388, #1402, #1404, #1408, #1418) +- Complete feature-flag reference for fork operators, and the rule that feature switches stay while rollout switches retire, with a Kind column (#1384, #1420) +- Specs: AgentCore Runtime V2 migration plan (#1376), background handoffs (#1371), conversation rewind and fork (#1419) + ## [1.25.1] - 2026-09-26 A single SPA fix on top of 1.25.0, which carries this week's features. **Upgrading from 1.24.x? Follow the 1.25.0 deployment notes** (CDK deploy plus post-deploy scripts). 1.25.1 adds no steps of its own. diff --git a/CLAUDE.MD b/CLAUDE.MD index 0bacb1ad3..f4db8aeed 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -31,6 +31,12 @@ npx cdk synth npx cdk deploy {prefix}-PlatformStack ``` +### Smoke / regression pass +```bash +cd backend && uv run python scripts/smoke_turns.py --help # L1 turn matrix; see docs/testing/smoke-regression.md +``` +The `/smoke-regression` skill runs the full pass (hermetic suites → turn matrix on dev or a local stack → in-app-browser checklist). Run it after any change to the turn path or an SSE event, and once per release. + ## Key Conventions - **Single CDK stack.** All AWS resources live in `PlatformStack`. There are no cross-stack dependencies to manage and no other CDK stack to deploy in a particular order. Application code is shipped out-of-band by `backend.yml` via AWS APIs (ECR push → ECS service update / Lambda code update / AgentCore Runtime update), not via CDK. @@ -41,6 +47,7 @@ npx cdk deploy {prefix}-PlatformStack - **Signal-based state** throughout frontend (`signal()`, `computed()`) - **Prompt-cache stability is a contract.** Bedrock prompt caching is exact-prefix-match: any list that reaches the system prompt or `toolConfig` (skills, tools, models, MCP tool listings) must be deterministically ordered at its source, and restored conversation history must be byte-stable between compaction-state changes (see the truncation anchor in `TurnBasedSessionManager`). An order flip or history mutation between turns silently re-writes a 30k–150k-token prefix at the cache-write premium, which is **1.25× the model's own base input rate** (2× at 1h TTL; a cache read is 0.1×) — there is no flat per-MTok figure, so price it against the model actually in play: $1.375/MTok on our default Haiku 4.5, $4.125 on Sonnet 4.6. Our `us.*` ids are **Regional (CRIS)** inference profiles and price ~10% above `global.*` for the same model; rates live in `curated-models.ts`. Per-model AWS **model cards** stay the primary source — they carry context windows, caching support, parameter tables and cutoffs, which no pricing API does — but the **Price List API now corroborates almost all of them**, and a rate worth trusting should agree in both. **Which offer file matters more than which vendor:** `--service-code AmazonBedrockFoundationModels` (372 us-west-2 SKUs) carries Claude, Cohere, Palmyra, TwelveLabs, Luma, Stability; `--service-code AmazonBedrock` (1052) carries Nova, xAI, Google, DeepSeek, Qwen and Moonshot. Querying the wrong one returns nothing and reads exactly like an unpublished model — which is how an earlier revision of this note concluded the API held "10 Claude SKUs, none newer than Claude 3." **That is no longer true** (re-verified 2026-09-21): every Claude model we curate is present and matches its card to the cent — Haiku 4.5 $1.10/$5.50 Regional and $1.00/$5.00 Global, Sonnet 4.6 $3.30/$16.50, Opus 4.7 $5.50/$27.50, Sonnet 5 $2.00/$10.00 Global, Fable 5.1 $10/$50 Global with a cache read of $0.25 that independently confirms its 0.025× multiplier. The 1.100× Regional premium reproduces throughout. **Two traps remain.** The `model` attribute is gone from the product schema — match on `servicename` (FoundationModels) or `usagetype` (AmazonBedrock), or every filter silently returns zero. And the hosted OpenAI family (`us.openai.gpt-5.6-*`, `us.openai.gpt-6-astra`, `openai.gpt-5.4`) is genuinely absent from **both** offer files, so its cards remain the only source; the old explanation that Marketplace billing is uncovered is wrong — Claude's rows are `MP:` usagetypes and are covered. `PROMPT_CACHE_OBSERVABILITY_ENABLED=false` disables the observability layer (fingerprint hook, cacheStatus derivation, EMF metrics) — the caching itself stays on - **Token cost effectiveness is a design tenet — engineer against waste, not against context.** Before merging a change that touches the model call path, answer: what does this add to the prompt, on every turn, for the life of every session? (1) Anything in the cacheable prefix (system prompt, `toolConfig`, restored history) must be deterministic and append-only — see the prompt-cache contract above. (2) Per-turn payloads (tool results, MCP responses, retrieved documents) should be bounded or offloaded, never unbounded pass-through. (3) Don't guess — verify: `cacheStatus` + fingerprint hashes on the session's `C#` rows, `GET /admin/costs/sessions/{id}/calls`, and the `AgentCoreStack/PromptCache` EMF metrics exist to prove a change's cost impact. The balance: "cost effective" means eliminating waste (avoidable cache re-writes, duplicated context, oversized payloads) — it never means stripping context the model needs for a quality answer. When cost and answer quality genuinely conflict, quality wins; look for the cheaper path to the *same* quality, not a cheaper answer +- **The turn path is mapped, stage by stage, in `docs/specs/turn-path-ttft.md`** — every stage from the SPA's send to the first token, with what runs where, what is network, what blocks the event loop, and which `turn_prelude` mark times it. Read it before touching `inference_api/chat/routes.py`, `chat/service.py`, `base_agent.py`, `session_factory.py` or the stream coordinator; the route's phases (`_resolve_turn_attachments`, `_prepare_session_state`, `_check_turn_quota`, `_resolve_turn_model`, `_resolve_effective_tools`, `_build_turn_tools`) are named for that map. It also carries the regression plan for this path (§6): the invariants, the tests per change, and the dev recipes that settle a timing claim - **Time to first token (TTFT) is a hard budget — add no latency before the first token.** That path is everything between a request arriving and the first streamed token: request handling in `inference_api/chat/routes.py`, agent build/cache lookup, MCP pre-flight, and every `BeforeInvocationEvent` / `BeforeModelCallEvent` hook — the latter fires before *every* model call, so its cost repeats on each tool round. Default pattern: a hook captures raw facts only, and anything derived from them is computed after the model answers, memoized (e.g. `get_context_breakdown(agent, itemized=True)` itemizes on read, not in `ContextAttributionHook`). If a change genuinely cannot avoid adding latency there, measure it at realistic scale (dozens of tools, a production-sized system prompt — not a toy fixture), state the number in the PR, and get the developer driving the change to accept it explicitly before merging. A millisecond or two sounds free; the preamble was cut from 455ms to 22–37ms, so each such addition is several percent of that budget, and they compound - **One session can be served by more than one agent — never cache session state on an agent instance.** The agent cache keys on *configuration* (system prompt, tools, model, skills), so an `@`-mention turn builds a second `Agent`, and each `Agent` builds its own `TurnBasedSessionManager`. Both write the same DynamoDB session row and neither knows the other exists, while `initialize()` never re-runs on a cache hit — so anything a manager loads once and holds goes stale silently. This has bitten twice: conversation history (#741, fixed by aliasing the message list in `_adopt_session_conversation`) and compaction state (#751, fixed by re-reading it every turn in `_adopt_persisted_compaction_state`). Per-session state must be aliased across instances or re-read per turn, and must never move backwards — a clobbered checkpoint or truncation anchor is a prompt-cache **cost** bug before it is a correctness one - **All dependencies use exact version pins** — no `^`, `~`, or `>=` @@ -61,6 +68,21 @@ A feature has up to two switches, one per side, set per environment: - **Keep the two in step per environment.** If they drift, the damage is cosmetic: the UI offers a feature whose API 404s, or hides a working one. - **Adding a flag:** add a field to `FeatureFlags` and all three environment files (in-development: `false` locally and in prod, `true` in dev only if the dev environment has turned the backend on), plus the backend flag, CDK config and test, and the `platform.yml` variable. +### Feature switches stay; rollout switches retire + +A default-on flag is one of two kinds, and only one is permanent. The docs-site page (`docs-site/src/content/docs/configuration/feature-flags.md`) records which kind each flag is in its **Kind** column. + +| Kind | What it turns off | Lifetime | +|---|---|---| +| **Feature switch** | A capability a deployment may reasonably not want: it costs money, widens what users can do, or is a product choice (`TOOL_SUMMARIES_ENABLED`, `BROWSER_TAKEOVER_ENABLED`, `DICTATION_ENABLED`) | Permanent. It is configuration | +| **Rollout switch** | A change meant to be invisible (a reorder on the turn path, a fix, a cache), kept only so a bad deploy can be undone without a revert (`KB_SEARCH_AHEAD_ENABLED`, `CONFIG_CACHE_ENABLED`) | Retires once the change is proven | + +Rollout switches retire because the off path costs something every day it exists. It doubles a code path, often on the turn path, and its tests have to cover both sides. Nobody runs it, so it breaks without anyone noticing. A kill switch that has not been flipped in months gives false confidence. + +- **When adding a rollout switch**, write its retirement condition in the reader's docstring and in the docs-site row. The default condition is: shipped in a production release, then at least two weeks there with the switch never flipped and the change's readout recorded. Use a stricter condition when the risk only shows up in a situation prod hasn't reached yet (for example, `AGENT_BUILD_SHARED_SESSION_ENABLED` waits for Runtime V2, because its known risk is a V2 snapshot). +- **Retiring one** is a cleanup PR, batched once per release cycle. Delete the reader, the off path and its tests, any env or CDK wiring, and the docs row. Add a CHANGELOG `Removed` entry naming the variable, because a deployment that set it `=false` loses that opt-out. +- **Promoting an in-development flag to default-on** also means picking its kind. A finished feature usually stays a feature switch. Any rollout switch that guarded its internals can retire on the usual condition. + Current in-development flags: **Shared Projects** (`PROJECTS_ENABLED` / `CDK_PROJECTS_ENABLED`, `features.projects`). It's on in the `development` GitHub environment and in `environment.development.ts`, and off in production. ## SSE Event Types @@ -75,7 +97,7 @@ Current in-development flags: **Shared Projects** (`PROJECTS_ENABLED` / `CDK_PRO | `ui_tool_input_partial` | Streamed partial tool input for a UI tool (SEP-1865 `ui/notifications/tool-input-partial`) — payload `{type, toolUseId, arguments}`. Emitted repeatedly while the model is still streaming a UI tool's arguments (after the early `ui_resource` mount); `arguments` is the streamed prefix server-side "healed" into a valid object (`apis/shared/mcp_apps/partial_json.py`). The SPA relays each to the App via `ui/notifications/tool-input-partial` so a progressively-rendering App (e.g. Excalidraw's guided camera tour) animates as args arrive; the complete `tool-input` follows once the input is final. Same gating as `ui_resource` | | `agent_status` | What the agent is doing right now — payload `{type, sessionId, phase, cycle, toolName?, toolUseId?, durationMs?, ok?}`. Emitted from `AgentStatusHook` (`BeforeModelCall` / `Before`+`AfterToolCall`) and drained in `stream_coordinator` exactly like `steering_applied`, before the event it precedes, so "Using list_assignments" reaches the client while that tool is running rather than after its result. Phases: `thinking` (one per event-loop cycle — a three-tool turn reports it four times, and `cycle` distinguishes them), `tool_start`, `tool_end` (carries Strands' own measured `durationMs`, and `ok=false` for BOTH a raised exception and a result with `status: "error"`). There is deliberately **no "responding" phase** — the SPA already knows text is streaming from the deltas, and a backend-derived duplicate of a fact the client holds first-hand would only disagree at the edges. Durations are live-only and deliberately NOT persisted: a reloaded conversation shows summaries without timings, where a client-invented number would be one the user could not trust. Costs nothing against the model — nothing it produces reaches the prompt, so the cacheable prefix is untouched. Gated by `AGENT_STATUS_ENABLED` (default on with a kill switch); while off the hook is registered but every callback returns immediately and the SPA stays on its cycling phrases | | `tool_group_summary` | Model-generated one-line summary of a finished tool batch — payload `{type, sessionId, batchId, toolUseIds, summary}`, e.g. "Found the Syllabus Acknowledgment assignment in BIO 101". Produced by a Nova Micro **side-channel** task (`apis/shared/tool_summaries/summarizer.py`) structured exactly like `session_title`: its own Bedrock call on its own messages, concurrent with the agent stream, so it **never appends to the conversation** and adds nothing to the cacheable prefix. Spend is one bounded call per batch (inputs/results truncated at capture in the hook AND again in the summarizer). Lands mid-turn, out of band with the content stream, so it can arrive after the rail that shows it has rendered; the SPA keys it by `toolUseId` (not batch — client-side grouping need not match backend batches, and a group spanning two batches shows the first batch's line). Persisted as `TSUM#` rows in sessions-metadata, reusing the `SessionLookupIndex` GSI — zero new infra — and replayed on `GET /messages` as `toolSummaries`, because the event never re-streams. Deliberately NOT written onto the message content blocks: that is the Converse payload and the cacheable prefix, so a display string there would be paid at model rates every subsequent turn. Gated by `TOOL_SUMMARIES_ENABLED` (default on with a kill switch); while off the SPA's deterministic client-side formatter still renders ("Listed 4 assignments"), so absence is a downgrade in specificity, never a blank | -| `session_title` | Server-generated conversation title on a session's FIRST turn — payload `{type, sessionId, title}`. Title generation (Nova Micro) runs as an asyncio task concurrent with the agent stream; the finished title is interleaved between agent events (non-blocking done-check in `stream_with_quota_warning`), so the sidebar/top-nav rename while the response is still pending. Emitted at most once per stream, possibly after `done` (the SPA parser allowlists it past Completed-state gating); never carries the "New Conversation" placeholder. Best-effort: a stream that finishes before generation emits nothing — the SPA's post-close metadata refresh (`refreshTitleFromServer`) is the fallback, reading the title the task also persisted via `update_session_title` | +| `session_title` | Server-generated conversation title on a session's FIRST turn — payload `{type, sessionId, title}`. Title generation (Nova Micro) runs as an asyncio task concurrent with the agent stream, so the sidebar/top-nav rename while the response is still pending. The finished title is picked up by a non-blocking one-shot check (`_session_title_sse` in `stream_with_quota_warning`) that is polled from two places: the coordinator's live status merge (`poll_side_frame`, every 100ms, so a title that lands during the model's time-to-first-token or a long tool call goes out at once rather than waiting for the agent's next event) and between agent events (its only route while `AGENT_STATUS_LIVE_DRAIN_ENABLED=false`). The task returns the moment Nova answers and persists the title in the background (one keyed `update_item` on the static SK), so the DynamoDB write never sits between generation and the user seeing it. Emitted at most once per stream, possibly after `done` (the SPA parser allowlists it past Completed-state gating); never carries the "New Conversation" placeholder. Best-effort: a stream that finishes before generation emits nothing, and deliberately does not hold the connection open to wait (the SPA clears loading on close, not on `done`) — the SPA's post-close metadata refresh (`refreshTitleFromServer`, up to 5 retries at 400ms) is the fallback, reading the persisted title | | `quota_session_notice` | **This conversation** has reached the tier's session-notice share of the monthly limit — payload `{type, sessionId, sessionCost, quotaLimit, sessionPercentageOfLimit, thresholdPercentage, message}`. Emitted at the head of the stream right after `quota_warning`, and re-emitted every turn while over the share (dismissal is client-side, same contract as `quota_warning`). `sessionCost` is the session's **lifetime** cost — the `totalCost` aggregate on its metadata row — deliberately not period-scoped: a conversation that opened last month and is spending this month's budget is exactly the one worth surfacing. Share is tier-configurable (`sessionNoticePercentage`, default 25%, 0 disables); the whole runway rides the `QUOTA_RUNWAY_ENABLED` kill switch (default on), which also gates the 50%/75% `quota_warning` rungs. The SPA scopes it to the conversation it names — never shown above another thread's composer | | `agent_notice` | A Shared Project's harness is running this turn **without** some of its setup — payload `{type, sessionId, agentId, projectId?, message, unavailableModelId?, unavailableTools, unavailableSkills, unavailableMemory?}`. Only a project harness degrades (shared-projects §9.6): a member missing a bound tool, skill, model or memory space still gets the turn, with that piece dropped — tools keyed by *server* so no later scoped ref of a denied server slips through, an unavailable model falling through to the member's default, and an all-dropped toolset still meaning "no tools", never the request's. Every other shared agent keeps block-with-message (D5, `stream_error`). Emitted before `message_start`, beside `quota_session_notice`. **SSE only, never the prompt** — the model is not told what was dropped, so the cacheable prefix is unaffected — and not persisted, because the next turn re-derives it. The SPA ignores it until the Projects UI (PR-1.8) renders it | | `model_retry` | Backend is retrying a failed model call instead of surfacing it — payload `{type, attempt, delaySeconds}`. Emitted from Strands' `EventLoopThrottleEvent`; `attempt` is 1-based and counted per turn in `stream_processor` (the raw event carries only the delay). **Timing caveat:** Strands sleeps the backoff *inside* its hook and yields the event afterwards, so it lands as the next attempt BEGINS, not when the wait starts — `delaySeconds` describes the gap just endured, it is not a countdown. It also cannot cover the failing model call itself, which is indistinguishable from a slow healthy one. The SPA swaps the loading indicator's cycling phrases for a fixed amber notice, cleared on `message_start`/`done` | @@ -167,7 +189,7 @@ Three things that look optional and are not: - **Tool not appearing:** Check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry - **Session not persisting:** Check AgentCore Memory config, session_id, TurnBasedSessionManager flush - **SSE stream disconnecting:** Check 600s timeout, client connection, quota exceeded events -- **Usage at $0 / not counted against quota:** the call's model id has no catalog row, so it priced to `None`. The `UnmeteredModelCall` EMF metric (PromptCache namespace, alarm threshold 0) fires per such call; the dashboard's "Unmetered calls by model" widget names the id. A turn that names no model runs on the user's saved default, then the catalog's `isDefault` row (`_resolve_fallback_model`); the hard-coded `Defaults.MODEL_ID` is reached only when the catalog has no enabled default, and prod has no row for it +- **Usage at $0 / not counted against quota:** the call's model id has no catalog row, so it priced to `None`. The `UnmeteredModelCall` EMF metric (PromptCache namespace, alarm threshold 0) fires per such call; the dashboard's "Unmetered calls by model" widget names the id. A turn that names no model runs on the user's saved default, then the catalog's `isDefault` row (`_resolve_fallback_model`); the hard-coded `Defaults.MODEL_ID` is reached only when the catalog has no enabled default, and prod has no row for it. **Voice** prices only when the catalog has a row for `amazon.nova-2-sonic-v1:0` (curated as "Nova 2 Sonic" on the Bedrock tab): its `input`/`output` are the TEXT rates and `speechInput`/`speechOutputPricePerMillionTokens` the speech ones, billed per bucket by `CostCalculator.calculate_voice_cost`. The row's SPEECH output modality keeps it out of the chat picker - **Cost spike / cache miss:** Check `cacheStatus` + `toolConfigHash`/`systemPromptHash`/`historyHash` on the session's `C#` rows in sessions-metadata (or `GET /admin/costs/sessions/{id}/calls`) — the hash that changed between consecutive calls names the cache-buster. Rule out the explained case first: `agentSwitched` on the row means an `@`-mention ran that turn on a different Agent, which re-writes the prefix on purpose and looks identical to the regression (`agentSwitchMissCount`/`agentSwitchUsd` are a subset of the totals — subtract for unexplained waste). **A `hit` is not proof the prefix was cached** — `partial_miss` marks the calls that read a leading segment (tools+system) while re-writing the history behind it, which costs like a full miss; its tell is a flat `cacheReadInputTokens` next to a `cacheWriteInputTokens` that tracks the whole conversation, and its dollars are in `wastedUsd` with `partialMissCount`/`partialMissUsd` naming the subset. Rows written before that status shipped still say `hit` — nothing is backfilled ## Coding Standards diff --git a/README.md b/README.md index cc80e1ca2..16dd52b12 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.25.1-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.26.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -296,7 +296,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.25.1 +**Current release:** v1.26.0 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index e37d65d47..606a28a95 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,237 @@ +# Release Notes — v1.26.0 + +**Release Date:** October 3, 2026 +**Previous Release:** v1.25.1 (September 26, 2026) + +--- + +> 🏗️ **A CDK deploy is required.** The deploy order is unchanged: `platform.yml` → `backend.yml` → `frontend-deploy.yml`. The only infrastructure change is new widgets on the turn-latency dashboard. There are **no new tables, no GSI changes and no `backfill_*.py` step**. +> +> 🎙️ **Voice stays unmetered until an admin adds the Nova 2 Sonic catalog row.** Nothing seeds it. Add it from **Admin → Models → catalog** in each environment after the deploy. See Deployment notes. +> +> 📦 **Strands moves from 1.55.0 to 1.57.2**, paired with bedrock-agentcore 1.24.0. Voice mode was adapted to the new Bidi API. The voice WebSocket contract the SPA uses is unchanged. + +--- + +## Highlights + +**The first token arrives sooner.** This release maps every stage between a request arriving and the model's first output, puts a timer on each one, and then takes the largest stages we own off the critical path. On dev: +- a cold agent build drops from **869ms to 372ms** (median); +- a long conversation no longer spends **~390ms** counting its stored messages before every turn, and its first token went from **1560ms to 1092ms**; +- the long-term-memory lookup overlaps the message writes, cutting `pre_model` from **450ms to 270ms** on a warm turn; +- an agent's external MCP servers load concurrently, so a first turn waits for the slowest server instead of all of them in a row; +- an agent's knowledge-base search runs alongside the build instead of before it. + +**Voice is priced, and users can choose the voice.** Voice sessions had been counting $0 against quota. They are now billed per modality, at separate text and speech rates. The voice button opens a menu with all 16 Nova 2 Sonic voices and a microphone picker. + +The catalog adds **Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna and GPT-5.5**. Managed knowledge bases stop mis-counting imports and stop serving stale synced files. A new **smoke and regression pass** drives real turns against a deployed stack and runs every night. + +**Action required:** a CDK deploy, the Nova 2 Sonic catalog row, and one optional knowledge-base repair, all under Deployment notes. + +--- + +## Faster to the first token + +A turn is the most complicated process in the app, and until this release nobody had one map of it. `docs/specs/turn-path-ttft.md` now covers every stage from the SPA's send to the first token: what runs, what is network, what blocks the event loop, which timer measures it, and what it costs. The chat route was refactored into named phases that match the map (#1388). Every change below was measured on dev before and after, and each readout is recorded in the spec. + +### Measurement + +- **`agent_build.tools` is split into four sub-stages** (`filter`, `gateway`, `mcp`, `extra`), with a timing line per external MCP server (#1396). +- **The turn clock now runs to the first model output.** `head_of_turn`, `pre_model` and `model` stages follow the existing `turn_prelude`, and a `FirstTokenMs` metric is published (#1397). `firstTokenMs` is read at the model mark, so the stages add up to it exactly (#1398). +- **The turn-latency dashboard** gains first-token and after-prelude (p90) widgets. + +### What got faster + +| Change | Before → after (dev) | Kill switch | +|---|---|---| +| One process-wide boto3 session for the agent build, built at container warm-up (#1377, #1395) | cold `agent_build` 869 → 372ms; client first token 4211 → 3564ms (median, 15-turn A/B per arm) | `AGENT_BUILD_SHARED_SESSION_ENABLED` | +| Per-turn history count read on a worker thread from the head of the turn (#1403) | `history_count` 388 → 0ms at 100+ events; first token 1560 → 1092ms | `HISTORY_COUNT_PREFETCH_ENABLED` | +| Long-term-memory lookup overlapped with the two message writes (#1400) | `pre_model` 450 → 270ms warm, 664 → 352ms cold | `MEMORY_RETRIEVAL_PREFETCH_ENABLED` | +| External MCP servers pre-flighted concurrently (#1406) | tool loading tracks the slowest server, not the sum | `MCP_PARALLEL_PREFLIGHT_ENABLED` | +| Knowledge-base search started before the agent build (#1411) | first-agent-turn `rag` 1404 → 955ms | `KB_SEARCH_AHEAD_ENABLED` | +| Binding history check reduced to a one-event existence read (#1413) | first-agent-turn `rag` 955 → 571ms | — | +| KB search's follow-up reads moved off the event loop (#1415) | residual `rag_wait` ~130 → ~10ms | — | +| KB record read once per search, on a cached table (#1416) | DynamoDB time around `Retrieve` ~180 → ~20ms | — | +| One Bedrock client reused across side-channel calls (#1375) | ~6ms of event-loop work per call in the Runtime; ~250ms on app-api's first document digest | — | + +What reaches the model is unchanged in every case. Tool order, the system prompt, the augmented message and the citations are byte-identical, so none of this rewrites the prompt-cache prefix. The MCP change keeps clients registered in catalog order. The history-count read counts only messages created before the turn began, so it returns the same index the inline read did. + +**Session titles also arrive sooner (#1374).** The finished title is checked every 100ms during the stream instead of only between agent events. The DynamoDB write runs in the background, and it is now a single keyed `update_item`. + +### Kill switches + +The five switches in the table are **rollout switches**: they exist so a bad deploy can be undone without a revert, and they are on by default. Under the rule added in this release (#1420), each one will be removed once it has run cleanly in production for two weeks. Track them in #1422. Don't build on their off position. + +--- + +## Voice: priced, and your choice of voice + +Voice sessions had been running **unmetered**. With no catalog row for `amazon.nova-2-sonic-v1:0`, every session priced to `None` and counted $0 against the user's quota. A plain catalog row would still have mispriced it, because Nova 2 Sonic bills speech tokens at about ten times its text tokens, and Strands 1.55 discarded the speech/text split Nova reports (#1399). + +### Backend + +- **Catalog rows** gain optional `speechInputPricePerMillionTokens` and `speechOutputPricePerMillionTokens`. On a speech model, `input` and `output` are the **text** rates. +- **`CostCalculator.calculate_voice_cost`** bills the four buckets separately. When the split is missing, all tokens are priced at the speech rate, so an error can only over-count. +- **`NovaSonicModelWithUsageDetails`** keeps Nova's `usageEvent.details.total` split. A test pins the provider method it overrides, including its 1.57 signature, so an SDK change fails loudly. +- **A curated Nova 2 Sonic row** on the Bedrock tab carries us-west-2 rates from the Price List API: text $0.319 / $2.651 and speech $3.00 / $12.00 per million tokens. +- **Speech models are hidden from the chat picker**, because Sonic answers no chat API. +- **The voice choice** is stored as a new `voiceId` user setting. An unknown id returns 422, and a blank one clears the setting. It rides the voice WebSocket's `config` frame. For languages that inflect for gender, the voice prompt gains one line naming the voice's form. + +### Frontend + +- **The composer's voice button** is now a split control. A chevron opens a menu with **Voice**, **Microphone** and **Start voice conversation**. Choices are fixed at connect, so the chevron hides while a session is live. +- **Voices:** all 16 Nova 2 Sonic voices, grouped by language. Tiffany stays the default, because only Tiffany and Matthew speak every language. +- **Microphone:** the device is stored per browser and shared with dictation. A device that has disappeared falls back to the system default. +- **The admin model form** shows the speech rates under "Speech pricing". + +**Voice stays unmetered until the catalog row exists in each environment.** See Deployment notes. + +--- + +## New models + +Four newly released Bedrock models join the curated catalog (#1358). + +| Model | Model ID | Input / output ($/MTok) | Cache write / read | Featured | +|---|---|---|---|---| +| Claude Opus 5.5 | `us.anthropic.claude-opus-5-5` | 4.40 / 22.00 | 5.50 / 0.22 (0.05×) | yes | +| GPT-6 Sol | `us.openai.gpt-6-sol` | 2.20 / 11.00 | 2.75 / 0.22 | yes | +| GPT-6 Luna | `us.openai.gpt-6-luna` | 0.11 / 0.55 | 0.1375 / 0.011 | yes | +| GPT-5.5 | `us.openai.gpt-5.5` | 5.50 / 33.00 | 0 / 0.55 | no | + +Claude Opus 4.7, GPT-5.6 Sol and GPT-5.6 Terra move to the picker's **More models** submenu. They are still available and are not retired. The curated rows use Regional (`us.*`) ids. A deployment that prefers Global CRIS should edit the id when it adds a row. + +--- + +## Smoke and regression pass + +`backend/scripts/smoke_turns.py` drives real agent turns through the same `POST /chat/stream` the SPA uses and asserts the response contract on the wire (#1405). It targets: +- a local `SKIP_AUTH` stack; +- a deployed environment, as a Cognito user; +- the AgentCore Runtime directly, through a headless grant. + +The matrix covers: +- SSE head and tail frame order; +- second-turn prompt-cache hash stability; +- restore from `GET /messages`; +- a tool turn; +- `ask_user_question` pause and resume, and tool-approval resume; +- the single-flight 409; +- Stop and lease recovery; +- preview sessions and mid-turn steering; +- inline PDF and diverted CSV attachments; +- skills, knowledge-base citations and quota refusal. + +The runbook is `docs/testing/smoke-regression.md`, and the `/smoke-regression` skill runs the full pass. + +The nightly pipeline now runs the matrix after its E2E step and uploads the report (#1412, #1414). It is non-blocking for now. The first run against dev found the inline-attachment bug fixed below. + +--- + +## 🐛 Bug fixes + +**Managed knowledge bases** +- **Imports didn't count against the storage cap.** Every Google Drive import, web crawl or sync on a managed KB committed bytes it had never reserved. `totalBytes` didn't move, and `reservedBytes` could go negative. Imports now reserve their real size at ingestion, so they are capped like uploads (#1361). See Changed. +- **Changed synced files were never re-ingested.** When a sync found a changed Drive file or crawled page on a managed KB, the ingestion consumer mistook the overwrite for a duplicate event. The KB kept serving the version from import time, and a later delete refunded the old size. The sync now records the version it staged, and the consumer re-ingests whenever that differs from what it last ingested (#1365). +- **A failed sync could block a file for good.** When the S3 stage failed, the sync had already advanced the file's change markers, so every later run treated the file as unchanged. A failed stage now rolls those markers back, conditionally, so a newer run's write is left alone (#1373). + +**Conversations** +- **Finished turns marked as interrupted.** On a first turn the loading state can outlast the server's `done`. Pressing Stop in that gap marked a finished turn interrupted, and the next prompt carried a false interruption note (#1366). A refresh or tab close in the same window had the same effect (#1370). Both now check whether `done` has already arrived. +- **Side-channel results accepted on a refusal.** The compaction summary and extraction, tool-batch summaries, document abstracts and titles accepted a guardrail stop, content-filter refusal or unknown stop reason as a valid result. A refused compaction summary would have become the checkpoint in the cacheable history. Each now accepts only a positive completion, and otherwise uses the fallback it already had (#1368). +- **Inline spreadsheets and decks dropped.** A headless caller (the harness, scheduled runs, API clients) that sent a CSV, XLSX or PPTX as inline base64 got a note saying the tools could reach the file, but the file was never stored. It is now written to S3 and registered as a session file, the same way an upload is. The SPA, which uploads first, was not affected (#1409). + +**Interface** +- **The context-meter panel** drew under the announcement pill (#1369). +- **Acknowledging an announcement on a flaky mobile connection** showed a global error toast and lost the acknowledgement. Both announcement requests are now silent on failure, and a network drop or gateway error is retried once (#1383). +- **The model picker's chevron** never rotated, because the handlers were bound to an output `cdkMenu` doesn't have (#1393). +- **Voice sessions' `message_count`** is no longer inflated about fourfold (#1367). + +## 🔒 Security + +- **Voice log injection.** `VoiceAgent.send_audio` logged the `sample_rate` the voice WebSocket passes through without validation, so a client could forge a log line. It now logs only the configured rate. The line is debug-level and came in with the Strands 1.57 adaptation. CodeQL `py/log-injection` found it on the release PR (#1425). + +## ⚠️ Changed + +- **Imports now count against the managed-KB byte caps.** A Google Drive import, web crawl or sync that would breach the per-owner or per-KB cap now fails with the same cap message an upload gets, instead of slipping past it (#1361). Owners near their cap may see an import refused that would have gone through on 1.25.x. + +## ✨ Also improved + +- **Tool rail (#1394):** + - a tool's full result now shows in a bounded scroll box, instead of being cut at 200 characters behind a toggle; + - labels sit above their values; + - input renders one `key: value` per line; + - scroll boxes are keyboard-focusable, and the labels now meet WCAG AA in both themes. +- **Dark mode (#1378):** sidebar date headers are brighter, and bold text and headings in responses are white instead of only heavier. +- **Mobile (#1382):** conversation text now lines up with the composer at the 16px inset, instead of sitting about 48px from the edge. + +## 🏗️ Infrastructure + +- **Turn-latency dashboard:** new widgets for first token (handler entry to first model output), the after-prelude stages at p90, and the `agent_build.tools` sub-stages (#1396, #1397). No new resources, tables or IAM changes. + +## 🔧 CI/CD + +- **Path-scoped PR gate (#1392).** A `changes` job maps a PR's paths to the suites they can reach, so a docs-only PR no longer runs the full 7–8 minute matrix. The backend pytest job runs on 4 xdist workers. +- **Nightly turn-smoke matrix**, non-blocking, with its report uploaded (#1412, #1414). + +## 📚 Docs + +- **`docs/specs/turn-path-ttft.md`:** the turn-path map, its regression plan, and every dev readout (#1388, #1402, #1404, #1408, #1418). +- **Feature flags for fork operators.** A complete flag reference with fork defaults and cost (#1384). It now has a **Kind** column, plus the rule that feature switches are permanent while rollout switches retire (#1420). +- **New specs:** + - the AgentCore Runtime V2 migration plan (#1376); + - background handoffs with in-place review (#1371); + - conversation rewind and fork (#1419). + +## 📦 Dependencies + +| Package | From | To | +|---|---|---| +| `strands-agents` / `strands-agents[bidi]` | 1.55.0 | 1.57.2 | +| `bedrock-agentcore` | 1.21.0 | 1.24.0 | +| `boto3` / `botocore` (locked) | 1.43.68 | 1.43.103 | +| `mcp` (transitive, locked) | 1.28.1 | 1.30.0 (still held below 2 by a declared constraint) | +| `awscrt` (transitive, `[bidi]`) | 0.32.0 | 0.32.2 | + +Strands 1.56–1.57 rewrote the Bidi API that voice uses (#1367, #1410): +- **Events.** The provider constructor, the input shape and every output event changed. A `VoiceWireAdapter` translates the new events back to the existing WebSocket contract, so the SPA and `voice_routes.py` are unchanged. +- **Session manager.** A `BidiAgent` now drives the ordinary session hooks. Guards keep the text agent's compaction checkpoint and long-term-memory retrieval away from voice history. +- **Transcripts.** They now arrive through `MessageUpdatedEvent` and are persisted from there. +- **Pins.** A test enforces the strands ↔ agentcore pairing and the `mcp<2` constraint across `pyproject.toml`, `uv.lock` and the Lambda requirements files. + +## 🚀 Deployment notes + +**1. Deploy in the usual order.** `platform.yml` → `backend.yml` → `frontend-deploy.yml`. The GSI check passes, because no index changes, and there is no backfill. + +**2. Add the Nova 2 Sonic catalog row (required for voice billing).** In each environment, after the backend deploy: **Admin → Models → catalog → Nova 2 Sonic**, and save it with: +- text $0.319 / $2.651; +- speech $3.00 / $12.00 per million tokens. + +It doesn't need to be enabled for chat, because speech models are filtered out of the picker. Until the row exists, voice keeps counting $0 and the `unmetered-model-call` alarm keeps firing for it. + +An environment that already has a hand-made row from the 1.25.0 notes should check it. A row with $3 / $12 in the **text** fields and no speech rates over-counts voice about twofold. + +**3. Settle a managed KB the 1.25.0 repair left for review (if you have one).** The 1.25.0 byte-counter repair could not explain KBs that had imports after promotion. Deploy first, because the old import path would break the counters again. Then report, and apply per agent: + +```bash +AWS_PROFILE= backend/.venv/bin/python backend/scripts/repair_managed_kb_byte_counters.py \ + --project-prefix --region +AWS_PROFILE= backend/.venv/bin/python backend/scripts/repair_managed_kb_byte_counters.py \ + --project-prefix --region --settle-unexplained --agent +AWS_PROFILE= backend/.venv/bin/python backend/scripts/repair_managed_kb_byte_counters.py \ + --project-prefix --region --settle-unexplained --agent --apply +``` + +Only settle a KB the report notes as *explained exactly by … document(s) committed without a request-time reservation*. The settle refuses a KB with anything in flight. A re-run of the report should then list no notes for it. + +**4. New kill switches (all on by default, runtime-only).** `AGENT_BUILD_SHARED_SESSION_ENABLED`, `HISTORY_COUNT_PREFETCH_ENABLED`, `MEMORY_RETRIEVAL_PREFETCH_ENABLED`, `MCP_PARALLEL_PREFLIGHT_ENABLED`, `KB_SEARCH_AHEAD_ENABLED` and `INLINE_ATTACHMENT_PERSIST_ENABLED`. Set one to `false` on the Runtime out of band only to undo a problem, and please open an issue if you need to. These are rollout switches and will be removed (#1422). + +**5. Expected on the first turns after deploy.** +- **No prompt-cache rewrites.** Nothing in this release changes what reaches the model, so cache hashes should stay flat across the deploy. +- **Fewer `SESSION` events.** The agent-binding check no longer creates a `SESSION` event in an empty thread. +- **One extra warning per process** on environments with GPT-6 Astra or GPT-5.6 rows. The Strands context-window table now resolves those ids to 1.05M, while their curated `maxInputTokens: 272000` cap still wins. Make sure no OpenAI-family row lacks `maxInputTokens`, or compaction would cut past the 272K price tier. + +--- + # Release Notes — v1.25.1 **Release Date:** September 26, 2026 diff --git a/VERSION b/VERSION index d905a6d1d..5ff8c4f5d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.25.1 +1.26.0 diff --git a/backend/Dockerfile.kb-migration b/backend/Dockerfile.kb-migration index a243cb44f..e05d82a67 100644 --- a/backend/Dockerfile.kb-migration +++ b/backend/Dockerfile.kb-migration @@ -55,6 +55,9 @@ RUN pip install --no-cache-dir -r /tmp/requirements.txt # implicit namespace packages, exactly as the kb-sync image does. COPY backend/src/apis/shared/__init__.py ${LAMBDA_TASK_ROOT}/apis/shared/__init__.py COPY backend/src/apis/shared/timestamps.py ${LAMBDA_TASK_ROOT}/apis/shared/timestamps.py +# aws_clients.py — kb_backend/records.py reads the assistants table through its +# process-cached DynamoDB resource (stdlib + boto3, a single file). +COPY backend/src/apis/shared/aws_clients.py ${LAMBDA_TASK_ROOT}/apis/shared/aws_clients.py # observability/emf.py — the metrics helpers publish through it. Copied as the # package so its `__init__` resolves. diff --git a/backend/Dockerfile.kb-sync b/backend/Dockerfile.kb-sync index 8c6985b7a..ace332312 100644 --- a/backend/Dockerfile.kb-sync +++ b/backend/Dockerfile.kb-sync @@ -34,6 +34,9 @@ RUN pip install --no-cache-dir -r /tmp/requirements.txt COPY backend/src/apis/shared/__init__.py ${LAMBDA_TASK_ROOT}/apis/shared/__init__.py COPY backend/src/apis/shared/timestamps.py ${LAMBDA_TASK_ROOT}/apis/shared/timestamps.py COPY backend/src/apis/shared/dynamo_errors.py ${LAMBDA_TASK_ROOT}/apis/shared/dynamo_errors.py +# aws_clients.py — kb_backend/records.py and assistants/rag_service.py read the +# assistants table through its process-cached DynamoDB resource. +COPY backend/src/apis/shared/aws_clients.py ${LAMBDA_TASK_ROOT}/apis/shared/aws_clients.py # feature_flags.py + caching/ — oauth/provider_repository.py memoizes its # provider list in the process-wide config cache, which reads the # CONFIG_CACHE_ENABLED kill switch. Both are module-level imports, so a miss diff --git a/backend/Dockerfile.rag-ingestion b/backend/Dockerfile.rag-ingestion index 960c837ae..b07a6b2bb 100644 --- a/backend/Dockerfile.rag-ingestion +++ b/backend/Dockerfile.rag-ingestion @@ -175,6 +175,9 @@ COPY backend/src/apis/app_api/documents/ingestion/ ${LAMBDA_TASK_ROOT} # so this adds files, not dependencies. COPY backend/src/apis/shared/__init__.py ${LAMBDA_TASK_ROOT}/apis/shared/__init__.py COPY backend/src/apis/shared/timestamps.py ${LAMBDA_TASK_ROOT}/apis/shared/timestamps.py +# aws_clients.py — kb_backend/records.py reads the assistants table through its +# process-cached DynamoDB resource (stdlib + boto3, a single file). +COPY backend/src/apis/shared/aws_clients.py ${LAMBDA_TASK_ROOT}/apis/shared/aws_clients.py COPY backend/src/apis/shared/embeddings/ ${LAMBDA_TASK_ROOT}/apis/shared/embeddings/ COPY backend/src/apis/shared/kb_backend/ ${LAMBDA_TASK_ROOT}/apis/shared/kb_backend/ RUN touch ${LAMBDA_TASK_ROOT}/apis/__init__.py diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 7956a0b39..2986c1c7a 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-stack" -version = "1.25.1" +version = "1.26.0" requires-python = ">=3.10" description = "Multi-agent conversational AI system with AWS Bedrock AgentCore" readme = "README.md" @@ -17,7 +17,7 @@ dependencies = [ "uvicorn[standard]==0.46.0", # AWS and cloud services - "boto3==1.43.68", + "boto3==1.43.103", # Utilities "python-dotenv==1.2.2", @@ -67,11 +67,11 @@ dependencies = [ [project.optional-dependencies] # AgentCore-specific dependencies (for inference_api) agentcore = [ - "strands-agents==1.55.0", + "strands-agents==1.57.2", "strands-agents-tools==0.8.8", "aws-opentelemetry-distro==0.19.0", - "bedrock-agentcore==1.21.0", + "bedrock-agentcore==1.24.0", # Multi-provider LLM support "openai==2.32.0", # For OpenAI models "google-genai==1.73.1", # For Google Gemini models @@ -82,7 +82,7 @@ agentcore = [ # Voice/BidiAgent dependencies (Nova Sonic speech-to-speech) bidi = [ - "strands-agents[bidi]==1.55.0", + "strands-agents[bidi]==1.57.2", ] # Document ingestion pipeline dependencies (for Lambda deployment) @@ -125,11 +125,11 @@ all = [ # Guard rail on a TRANSITIVE dependency — this is not a loosened pin. # # `mcp` arrives only via `strands-agents`; we do not depend on it directly and -# deliberately do not promote it to a direct dependency. `strands-agents==1.55.0` -# — the version pinned above, today — already declares `mcp>=1.23.0,<2.2`, so -# the resolver is free to cross into `mcp` 2.x whenever it gets the chance. +# deliberately do not promote it to a direct dependency. `strands-agents==1.57.2` +# — the version pinned above, today — declares `mcp>=1.23.0,<2.2`, so the +# resolver is free to cross into `mcp` 2.x whenever it gets the chance. # -# The ONLY thing holding us on the 1.x line right now is incidental: `mcp` 2.x +# Without this constraint the only thing holding us on 1.x is incidental: `mcp` 2.x # pulls `httpx2>=2.5.0`, which needs `idna>=3.18`, and we pin `idna==3.15` for # an unrelated Dependabot alert. A routine security bump of `idna` would quietly # unblock the major crossing — nobody would be choosing it, and nothing in the @@ -147,6 +147,7 @@ all = [ # A constraint, not a dependency: it bounds the version IF `mcp` is in the # resolution graph, without adding it to our own dependency metadata. Lift it # deliberately, together with a test pass over that `ClientSession` patch. +# Asserted by backend/tests/supply_chain/test_strands_agentcore_pairing.py. constraint-dependencies = ["mcp<2"] [tool.setuptools] diff --git a/backend/scripts/repair_managed_kb_byte_counters.py b/backend/scripts/repair_managed_kb_byte_counters.py index 2d65b718a..1e29b8afb 100644 --- a/backend/scripts/repair_managed_kb_byte_counters.py +++ b/backend/scripts/repair_managed_kb_byte_counters.py @@ -30,12 +30,43 @@ ``totalBytes == storedBytes + reservedBytes``. Conditioned on the counters being exactly what this run expects, so a concurrent upload or delete makes the write refuse (re-run it) rather than race. +* **settle-unexplained** (opt-in, ``--settle-unexplained`` with ``--agent``) — for a + quiet knowledge base whose counters no document explains, trust the ledger: + every unsettled ``complete`` document is claimed as committed, and the counters + are set to ``storedBytes = totalBytes = ledger``, ``reservedBytes = 0``. See + below for the shape that needs it and why it is safe. + +THE SHAPE --settle-unexplained EXISTS FOR +----------------------------------------- +File-source imports, crawls and syncs reserve nothing at request time, but write +their real ``sizeBytes`` before the consumer reads the row. The consumer took that +size for a reservation and committed it out of ``reservedBytes`` it never added to +(fixed in ``byte_cap.reserved_at_request``). Each such document left +``reservedBytes`` and ``totalBytes`` short by its size. On a migrated knowledge base +still holding its corpus as a reservation, that reads as:: + + reservedBytes == unsettled complete bytes - bytes of imports committed since + +which the adopt rule rightly refuses; on one with nothing else reserved it reads as +a negative ``reservedBytes``. The report names that diagnosis when the numbers fit. + +Once nothing is in flight, no reservation is legitimately outstanding, so the +counters the ledger implies are the right ones whatever produced the drift. The +mode still refuses unless the knowledge base is provably quiet: no upload in +flight, no row part-way through a commit or a refund, and the same documents seen +again after ``--quiet-seconds`` (an upload reserves before its ``DOC#`` row is +written, so a reservation read with no row yet must get time to show itself). The +claims and the counter write are ONE DynamoDB transaction, conditioned on every +counter being exactly as read and every claimed row being unchanged, so a +concurrent upload, delete, consumer commit or reconciler re-anchor cancels the +whole repair rather than interleaving with half of it, and a re-run finds nothing +left to do. SAFETY ------ * **Read-only by default.** Nothing is written without ``--apply``, and ``--apply`` requires ``--confirm-prefix`` equal to ``--project-prefix``. -* ``--agent`` limits a run to named agent ids. +* ``--agent`` limits a run to named agent ids. ``--settle-unexplained`` requires it. * Every write is conditional; a refused one is reported, never forced. Run (a human, not CI; production is read-only from agents):: @@ -43,6 +74,8 @@ AWS_PROFILE=dev-ai backend/.venv/bin/python backend/scripts/repair_managed_kb_byte_counters.py \\ --project-prefix dev-boisestateai-v2 --region us-west-2 # report only ... --apply --confirm-prefix dev-boisestateai-v2 # repair + ... --settle-unexplained --agent # preview one + ... --settle-unexplained --agent --apply --confirm-prefix ... # settle one The table name is derived from the prefix (``{prefix}-rag-assistants``) and can be overridden. The report names agent ids, document ids and byte counts only. @@ -55,6 +88,7 @@ import json import os import sys +import time from dataclasses import dataclass, field from datetime import datetime, timezone from decimal import Decimal @@ -68,6 +102,14 @@ #: pre-migration leftover that never reserved anything. IN_FLIGHT_STATUSES = frozenset({"provisioning", "uploading"}) +#: DynamoDB's cap on one TransactWriteItems call. A settle claims every unsettled +#: or un-backfilled document and writes the ``KB#`` record in one transaction. +MAX_TRANSACTION_ITEMS = 100 + +#: How long a settle waits, from reading the ``KB#`` record, before re-reading the +#: documents. See "THE SHAPE --settle-unexplained EXISTS FOR" above. +DEFAULT_QUIET_SECONDS = 10.0 + def _int(value: Any) -> int: try: @@ -91,6 +133,13 @@ class KbPlan: in_flight: int = 0 ledger: int = 0 reanchor: bool = False + #: The counters are not explained by the documents; left for review unless + #: ``--settle-unexplained`` makes ``settle`` true. + unexplained: bool = False + settle: bool = False + #: ``time.monotonic()`` when the ``KB#`` record was read; a settle waits out + #: its quiet window from here. + read_at: float = 0.0 notes: List[str] = field(default_factory=list) @property @@ -99,7 +148,11 @@ def stored_after_adopt(self) -> int: @property def has_work(self) -> bool: - return bool(self.adopt or self.backfill or self.reanchor) + return bool(self.adopt or self.backfill or self.reanchor or self.settle) + + def fingerprint(self) -> Tuple[Any, ...]: + """What a settle writes. Two reads that agree on this settle identically.""" + return (self.settle, sorted(self.adopt), sorted(self.backfill), self.ledger, self.in_flight) def summary(self) -> Dict[str, Any]: return { @@ -116,21 +169,45 @@ def summary(self) -> Dict[str, Any]: if self.reanchor else None ), + "settle": ( + { + "storedBytes": [self.stored, self.ledger], + "reservedBytes": [self.reserved, 0], + "totalBytes": [self.total, self.ledger], + } + if self.settle + else None + ), "notes": self.notes, } -def plan_kb(agent_id: str, record: Dict[str, Any], rows: Sequence[Dict[str, Any]]) -> KbPlan: +def plan_kb( + agent_id: str, + record: Dict[str, Any], + rows: Sequence[Dict[str, Any]], + settle_unexplained: bool = False, + read_at: Optional[float] = None, +) -> KbPlan: """What this knowledge base needs, from one read of its record and ``DOC#`` rows.""" plan = KbPlan( agent_id=agent_id, stored=_int(record.get("storedBytes")), reserved=_int(record.get("reservedBytes")), total=_int(record.get("totalBytes")), + read_at=time.monotonic() if read_at is None else read_at, ) unsettled_complete: List[Tuple[str, int]] = [] committed = 0 + #: Ledger bytes of documents that never reserved at request time (they carry a + #: ``sourceAdapterKey``). Before the fix each committed without a reservation. + import_bytes = 0 + import_documents = 0 + #: Rows whose bytes are in the counters but not in the ledger yet (a consumer + #: between its commit and ``complete``) or no longer (a delete between + #: ``deleting`` and its refund). A settle would count them wrong. + mid_settlement: List[str] = [] for row in rows: status = row.get("status") settled = bool(row.get("byteCapSettled")) @@ -138,23 +215,32 @@ def plan_kb(agent_id: str, record: Dict[str, Any], rows: Sequence[Dict[str, Any] if status in IN_FLIGHT_STATUSES and not settled: plan.in_flight += 1 if status != "complete": + if row.get("committedBytes") is not None and not row.get("byteCapRefunded"): + mid_settlement.append(_document_id(row)) continue + ledger_bytes: Optional[int] = None if not settled: if size > 0: unsettled_complete.append((_document_id(row), size)) elif row.get("committedBytes") is not None: if not row.get("byteCapRefunded"): - committed += _int(row.get("committedBytes")) + ledger_bytes = _int(row.get("committedBytes")) + committed += ledger_bytes elif row.get("retrievableAt"): plan.backfill.append((_document_id(row), size)) + ledger_bytes = size else: plan.notes.append(f"settled complete document {_document_id(row)} has no completion stamp; not counted") + if ledger_bytes and row.get("sourceAdapterKey"): + import_bytes += ledger_bytes + import_documents += 1 unsettled_bytes = sum(n for _, n in unsettled_complete) if unsettled_complete: if plan.in_flight == 0 and plan.reserved == unsettled_bytes: plan.adopt = unsettled_complete else: + plan.unexplained = True plan.notes.append( f"reservedBytes {plan.reserved} is not explained by {len(unsettled_complete)} unsettled " f"complete document(s) of {unsettled_bytes} bytes with {plan.in_flight} in flight; " @@ -168,15 +254,50 @@ def plan_kb(agent_id: str, record: Dict[str, Any], rows: Sequence[Dict[str, Any] if plan.in_flight == 0 and settled_everything and reserved_after == 0: plan.reanchor = plan.ledger != plan.stored_after_adopt or plan.total != plan.stored_after_adopt elif plan.in_flight == 0 and not unsettled_complete and plan.reserved: + plan.unexplained = True plan.notes.append( f"reservedBytes {plan.reserved} with no upload in flight is a leaked reservation; " f"left for review" ) elif plan.total != plan.stored + plan.reserved: plan.notes.append("totalBytes != storedBytes + reservedBytes, but the record is not quiet; not re-anchored") + + if plan.unexplained and import_bytes and plan.reserved == unsettled_bytes - import_bytes: + plan.notes.append( + f"explained exactly by {import_documents} imported/crawled/synced document(s) of " + f"{import_bytes} bytes committed without a request-time reservation (pre-fix consumer); " + f"--settle-unexplained repairs it" + ) + + if settle_unexplained and plan.unexplained: + _plan_settle(plan, unsettled_complete, committed, mid_settlement) return plan +def _plan_settle( + plan: KbPlan, + unsettled_complete: List[Tuple[str, int]], + committed: int, + mid_settlement: List[str], +) -> None: + """Turn an unexplained plan into a settle, if the knowledge base is quiet.""" + blockers = [] + if plan.in_flight: + blockers.append(f"{plan.in_flight} upload(s) in flight") + if mid_settlement: + blockers.append(f"document(s) part-way through a commit or refund: {', '.join(sorted(mid_settlement))}") + items = len(unsettled_complete) + len(plan.backfill) + 1 + if items > MAX_TRANSACTION_ITEMS: + blockers.append(f"{items} writes exceed one {MAX_TRANSACTION_ITEMS}-item transaction") + if blockers: + plan.notes.append(f"not settled: {'; '.join(blockers)}") + return + plan.settle = True + plan.adopt = unsettled_complete + plan.reanchor = False + plan.ledger = committed + sum(n for _, n in plan.backfill) + sum(n for _, n in plan.adopt) + + # ── AWS ────────────────────────────────────────────────────────────────────── def managed_records(table) -> List[Dict[str, Any]]: """Every ``KB#`` record on the managed engine that no worker step owns.""" @@ -216,12 +337,108 @@ def document_rows(table, agent_id: str) -> List[Dict[str, Any]]: kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"] -def apply_plan(table, plan: KbPlan) -> Dict[str, Any]: +def _counter_condition(name: str, value: int, values: Dict[str, Any]) -> str: + """``name`` is exactly ``value``; an absent counter reads as 0, as ``_int`` does.""" + placeholder = f":was_{name}" + values[placeholder] = Decimal(value) + if value == 0: + return f"(attribute_not_exists({name}) OR {name} = {placeholder})" + return f"{name} = {placeholder}" + + +def settle_transaction(table_name: str, plan: KbPlan) -> List[Dict[str, Any]]: + """The settle as one TransactWriteItems request. + + Plain Python values: it is sent through the table resource's own client + (``table.meta.client``), which serializes them the way the resource does. + """ + agent_id = plan.agent_id + + def update(key: Dict[str, str], expression: str, condition: str, values: Dict[str, Any], + names: Optional[Dict[str, str]] = None) -> Dict[str, Any]: + item: Dict[str, Any] = { + "TableName": table_name, + "Key": key, + "UpdateExpression": expression, + "ConditionExpression": condition, + "ExpressionAttributeValues": values, + } + if names: + item["ExpressionAttributeNames"] = names + return {"Update": item} + + items = [] + for document_id, n_bytes in plan.adopt: + items.append(update( + {"PK": f"AST#{agent_id}", "SK": f"DOC#{document_id}"}, + "SET byteCapSettled = :true, committedBytes = :n", + "attribute_exists(PK) AND attribute_not_exists(byteCapSettled) " + "AND #s = :complete AND sizeBytes = :n", + {":true": True, ":n": Decimal(n_bytes), ":complete": "complete"}, + {"#s": "status"}, + )) + for document_id, n_bytes in plan.backfill: + items.append(update( + {"PK": f"AST#{agent_id}", "SK": f"DOC#{document_id}"}, + "SET committedBytes = :n", + "attribute_exists(PK) AND attribute_exists(byteCapSettled) " + "AND attribute_exists(retrievableAt) AND attribute_not_exists(committedBytes) " + "AND attribute_not_exists(byteCapRefunded) AND #s = :complete AND sizeBytes = :n", + {":n": Decimal(n_bytes), ":complete": "complete"}, + {"#s": "status"}, + )) + values: Dict[str, Any] = {":ledger": Decimal(plan.ledger), ":zero": Decimal(0)} + condition = " AND ".join( + ["attribute_exists(PK)"] + + [ + _counter_condition(name, value, values) + for name, value in ( + ("storedBytes", plan.stored), + ("reservedBytes", plan.reserved), + ("totalBytes", plan.total), + ) + ] + ) + items.append(update( + {"PK": f"AST#{agent_id}", "SK": f"KB#{agent_id}"}, + "SET storedBytes = :ledger, reservedBytes = :zero, totalBytes = :ledger", + condition, + values, + )) + return items + + +def apply_settle(table, plan: KbPlan, quiet_seconds: float = DEFAULT_QUIET_SECONDS) -> Dict[str, Any]: + """Carry out a settle: re-check after the quiet window, then one transaction.""" + from botocore.exceptions import ClientError + + wait = plan.read_at + quiet_seconds - time.monotonic() + if wait > 0: + time.sleep(wait) + record = {"storedBytes": plan.stored, "reservedBytes": plan.reserved, "totalBytes": plan.total} + again = plan_kb(plan.agent_id, record, document_rows(table, plan.agent_id), settle_unexplained=True) + if again.fingerprint() != plan.fingerprint(): + return {"settle": "refused: the documents changed during the quiet window; re-run"} + + try: + table.meta.client.transact_write_items(TransactItems=settle_transaction(table.name, plan)) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") != "TransactionCanceledException": + raise + reasons = [r.get("Code", "None") for r in exc.response.get("CancellationReasons", [])] + return {"settle": f"refused: something changed since the read ({reasons}); re-run"} + return {"settle": "done", "adopted": len(plan.adopt), "backfilled": len(plan.backfill)} + + +def apply_plan(table, plan: KbPlan, quiet_seconds: float = DEFAULT_QUIET_SECONDS) -> Dict[str, Any]: """Carry out one plan. Every write is conditional; nothing is forced.""" from botocore.exceptions import ClientError from apis.shared.kb_backend import byte_cap + if plan.settle: + return apply_settle(table, plan, quiet_seconds) + agent_id = plan.agent_id result: Dict[str, Any] = collections.Counter() @@ -288,6 +505,17 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace: p.add_argument("--confirm-prefix", default=None, help="Required with --apply; must equal --project-prefix") p.add_argument("--table", default=None, help="Override {prefix}-rag-assistants") p.add_argument("--out", default=None, help="Write the JSON report here") + p.add_argument( + "--settle-unexplained", + action="store_true", + help="For the --agent knowledge bases only: settle counters no document explains to the ledger", + ) + p.add_argument( + "--quiet-seconds", + type=float, + default=DEFAULT_QUIET_SECONDS, + help="How long a settle waits after reading a record before re-checking its documents", + ) return p.parse_args(argv) @@ -296,6 +524,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int: if args.apply and args.confirm_prefix != args.project_prefix: print("--apply requires --confirm-prefix equal to --project-prefix", file=sys.stderr) return 2 + if args.settle_unexplained and not args.agent: + print("--settle-unexplained requires --agent: it is decided one knowledge base at a time", file=sys.stderr) + return 2 if args.profile: os.environ["AWS_PROFILE"] = args.profile table_name = args.table or f"{args.project_prefix}-rag-assistants" @@ -307,6 +538,7 @@ def main(argv: Optional[Sequence[str]] = None) -> int: table = boto3.resource("dynamodb", region_name=args.region).Table(table_name) + read_at = time.monotonic() records = managed_records(table) if args.agent: wanted = set(args.agent) @@ -316,10 +548,13 @@ def main(argv: Optional[Sequence[str]] = None) -> int: results = [] for record in records: agent_id = str(record["PK"]).split("#", 1)[1] - plan = plan_kb(agent_id, record, document_rows(table, agent_id)) + plan = plan_kb( + agent_id, record, document_rows(table, agent_id), + settle_unexplained=args.settle_unexplained, read_at=read_at, + ) summary = plan.summary() if args.apply and plan.has_work: - summary["result"] = apply_plan(table, plan) + summary["result"] = apply_plan(table, plan, args.quiet_seconds) results.append(summary) print(json.dumps(summary), flush=True) diff --git a/backend/scripts/seed_e2e_role.py b/backend/scripts/seed_e2e_role.py new file mode 100644 index 000000000..9e37a39ac --- /dev/null +++ b/backend/scripts/seed_e2e_role.py @@ -0,0 +1,131 @@ +"""Seed the RBAC role the nightly E2E user runs under. + +The nightly stack never runs the bootstrap data seeding, so its ``default`` +role is the auto-seeded one that grants nothing. That is fine for the +Playwright suite, which exercises chrome and a tool-free chat turn, but the +turn-path smoke matrix (``scripts/nightly/turn-smoke.sh``) needs two tools — +``calculator`` for the tool rows and ``ask_user_question`` for the +interrupt/resume row — and a model to run them on. + +This writes one ``e2e_user`` role through ``AppRoleRepository`` (the same +writer the admin console uses, so the DEFINITION row, the ``JWT_MAPPING#`` +row the resolver's GSI reads, and the ``TOOL_GRANT#`` / ``MODEL_GRANT#`` rows +all land together), mapped to the Cognito group ``scripts/nightly/seed-e2e-users.sh`` +puts the regular E2E user in. Idempotent: create on first run, update after. + +Usage (from ``backend/``, with credentials for the stack's account): + + uv run python scripts/seed_e2e_role.py --prefix nightly-e2e-develop + uv run python scripts/seed_e2e_role.py --prefix --dry-run # print, write nothing + +The table name comes from ``DYNAMODB_APP_ROLES_TABLE_NAME`` when set, else the +``//rbac/app-roles-table-name`` SSM parameter. Grants go through +``granted_*`` AND ``effective_permissions`` — the access checks read the +latter, and a role with only the former grants nothing (CLAUDE.md, RBAC). +""" + +from __future__ import annotations + +import argparse +import asyncio +import dataclasses +import json +import logging +import os +import sys +from typing import List, Optional + +logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") +logger = logging.getLogger("seed_e2e_role") + +ROLE_ID = "e2e_user" +DEFAULT_GROUP = "e2e-users" +DEFAULT_TOOLS = ["calculator", "ask_user_question"] +DEFAULT_MODELS = ["*"] + + +def build_role(*, group: str, tools: List[str], models: List[str]): + """The role as the repository will write it. Pure, so the test can pin it.""" + from apis.shared.rbac.models import AppRole, EffectivePermissions + + tools = sorted(set(tools)) + models = sorted(set(models)) + return AppRole( + role_id=ROLE_ID, + display_name="E2E Test User", + description=( + "Nightly end-to-end test account. Grants the tools the turn-path smoke " + "matrix exercises and every model; seeded by scripts/nightly/seed-e2e-users.sh." + ), + jwt_role_mappings=[group], + inherits_from=[], + granted_tools=tools, + granted_models=models, + granted_skills=[], + effective_permissions=EffectivePermissions(tools=tools, models=models, skills=[], quota_tier=None), + priority=10, + is_system_role=False, + enabled=True, + created_by="nightly-seed", + ) + + +def resolve_table_name(prefix: Optional[str], region: str) -> str: + env = os.environ.get("DYNAMODB_APP_ROLES_TABLE_NAME") + if env: + return env + if not prefix: + raise SystemExit("set DYNAMODB_APP_ROLES_TABLE_NAME or pass --prefix") + import boto3 + + ssm = boto3.client("ssm", region_name=region) + return ssm.get_parameter(Name=f"/{prefix}/rbac/app-roles-table-name")["Parameter"]["Value"] + + +async def upsert(role, repository) -> str: + """Create the role, or update it in place. Returns ``"created"`` or ``"updated"``.""" + existing = await repository.get_role(role.role_id) + if existing is None: + await repository.create_role(role) + return "created" + await repository.update_role(role) + return "updated" + + +async def main_async(args: argparse.Namespace) -> int: + role = build_role(group=args.group, tools=args.tools.split(","), models=args.models.split(",")) + if args.dry_run: + print(json.dumps(dataclasses.asdict(role), indent=2, default=str)) + return 0 + + table = resolve_table_name(args.prefix, args.region) + os.environ["DYNAMODB_APP_ROLES_TABLE_NAME"] = table + os.environ.setdefault("AWS_REGION", args.region) + from apis.shared.rbac.repository import AppRoleRepository + + outcome = await upsert(role, AppRoleRepository(table_name=table)) + logger.info( + "%s role %s in %s: group=%s tools=%s models=%s", + outcome, + role.role_id, + table, + args.group, + role.granted_tools, + role.granted_models, + ) + return 0 + + +def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--prefix", help="CDK project prefix, for the SSM table-name lookup") + p.add_argument("--region", default=os.environ.get("AWS_REGION", "us-west-2")) + p.add_argument("--group", default=DEFAULT_GROUP, help="Cognito group the role maps to") + p.add_argument("--tools", default=",".join(DEFAULT_TOOLS), help="comma-separated tool ids") + p.add_argument("--models", default=",".join(DEFAULT_MODELS), help="comma-separated model ids ('*' = all)") + p.add_argument("--dry-run", action="store_true", help="print the role, write nothing") + return p.parse_args(argv) + + +if __name__ == "__main__": + sys.exit(asyncio.run(main_async(parse_args()))) diff --git a/backend/scripts/smoke_turns.py b/backend/scripts/smoke_turns.py new file mode 100644 index 000000000..66a4c803a --- /dev/null +++ b/backend/scripts/smoke_turns.py @@ -0,0 +1,1436 @@ +"""Smoke / regression matrix for the chat turn path — every response code path. + +Drives real agent turns through the same `POST /chat/stream` the SPA uses and +asserts the response contract on the wire: SSE frame order (head → body → tail), +the interrupt/resume round trip, the single-flight 409, Stop + recovery, the +preview session, attachments, and that every turn restores from `GET /messages` +with the same content. Prompt-cache stability on the second turn is read back +from `GET /admin/costs/sessions/{id}/calls`. + +The canonical matrix, what each row proves, and the in-app-browser rows this +script cannot cover live in `docs/testing/smoke-regression.md`. Keep the two +in step: a row added here gets a line there, and vice versa. + +Targets (pick one): + + # Local stack — app-api :8000 with the SKIP_AUTH bypass on, proxying to + # inference-api :8001. No credentials. SKIP_AUTH_ROLES=system_admin is + # what makes the cache-stability row readable (admin endpoint). + cd backend + uv run python scripts/smoke_turns.py --base-url http://127.0.0.1:8000 + + # Deployed environment through app-api, as a Cognito password user + # (Hosted UI scripted login, same mechanism as tests/load). Needs a user + # with a permanent password; the nightly pipeline's E2E users qualify. + SMOKE_USERNAME=... SMOKE_PASSWORD=... uv run python scripts/smoke_turns.py \ + --base-url https:///api --auth cognito \ + --cognito-domain https://.auth..amazoncognito.com + + # Deployed environment straight at the AgentCore Runtime, as the owner of + # an active headless grant (apis/shared/harness/grants.py). Bypasses + # app-api, so the 409 / Stop / steer rows are skipped; persistence is read + # and sessions are deleted in-process through the shared services, with + # the AWS profile. + AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant \ + --user-id --prefix dev-boisestateai-v2 + +Every run spends real model dollars and writes real session rows for the user +it runs as. Prompts are deliberately tiny, `--model-id` defaults to the user's +default model, and sessions are deleted at the end unless `--keep`. The script +refuses a `--prefix` or `--base-url` that looks like production unless +`--allow-prod` is passed — this is a dev/local tool. + +Exit status: 0 when no row FAILED (SKIP and OBSERVED do not fail a run), 1 +otherwise, 2 for a setup error before any turn ran. +""" + +from __future__ import annotations + +import argparse +import asyncio +import base64 +import json +import logging +import os +import sys +import time +import uuid +from dataclasses import dataclass, field +from html.parser import HTMLParser +from typing import Any, Awaitable, Callable, Dict, List, Optional, Protocol, Tuple +from urllib.parse import urljoin, urlsplit + +import httpx + +from apis.shared.harness.sse import iter_sse_events + +logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") +logging.getLogger("httpx").setLevel(logging.WARNING) +logger = logging.getLogger("smoke") + +# ============================================================ +# Contract constants — mirror CLAUDE.md "SSE Event Types" + turn-path-ttft §6A +# ============================================================ + +# Frames that may legitimately precede `message_start` (turn-path-ttft I1 head). +# `agent_status` (preparing/prepared/thinking) and `session_title` are side +# channels that can land anywhere; the raw Strands passthroughs are harmless. +HEAD_ALLOWED = frozenset( + { + "quota_warning", + "quota_session_notice", + "agent_notice", + "citation", + "agent_status", + "session_title", + "model_retry", + "event", + "init_event_loop", + "start_event_loop", + } +) + +# Frames allowed after the final `message_stop` (I1 tail). `done` must be last. +# `event` and `message` are Strands passthroughs the coordinator relays as bare +# `data:` frames (the parser names them from their single top-level key); the +# SPA ignores both. +TAIL_ALLOWED = frozenset( + { + "oauth_required", + "tool_approval_required", + "user_question_required", + "browser_login_required", + "metadata", + "metadata_summary", + "compaction", + "session_title", + "tool_group_summary", + "agent_status", + "artifact", + "ui_resource", + "event", + "message", + "result", + "complete", + "done", + } +) + +# Every event name the backend is known to emit. Anything else is reported as +# a WARN (a new event type that CLAUDE.md's table does not list yet), not a FAIL. +KNOWN_EVENTS = ( + HEAD_ALLOWED + | TAIL_ALLOWED + | frozenset( + { + "message_start", + "content_block_start", + "content_block_delta", + "content_block_stop", + "message_stop", + "tool_use", + "tool_result", + "tool_error", + "tool_stream_event", + "ui_tool_input_partial", + "steering_applied", + "stream_error", + "error", + "quota_exceeded", + "citation_start", + "citation_end", + "reasoning", + "message", + } + ) +) + +INTERRUPT_EVENTS = frozenset( + { + "oauth_required", + "tool_approval_required", + "user_question_required", + "browser_login_required", + } +) + +# Observed-only: deterministic triggering is not possible from outside, so the +# report counts them but never fails on their absence. +OBSERVED_ONLY = ("model_retry", "compaction", "tool_group_summary", "steering_applied") + +PLACEHOLDER_TITLE = "New Conversation" +PREVIEW_PREFIX = "preview-" +SESSION_PREFIX = "smoke-" + +# Hosts that are production. The dev site is a subdomain of the prod apex, so +# this is an exact-host check, never a substring one. +PROD_HOSTS = frozenset({"boisestate.ai", "www.boisestate.ai"}) +PROD_PREFIX_MARKERS = ("prod", "production") + + +# ============================================================ +# Result types +# ============================================================ + + +@dataclass +class Frame: + name: str + data: Dict[str, Any] + t_ms: float + + +@dataclass +class TurnResult: + status_code: int + frames: List[Frame] = field(default_factory=list) + error: Optional[str] = None + aborted: bool = False + elapsed_ms: float = 0.0 + + @property + def names(self) -> List[str]: + return [f.name for f in self.frames] + + def first(self, name: str) -> Optional[Frame]: + return next((f for f in self.frames if f.name == name), None) + + def all(self, name: str) -> List[Frame]: + return [f for f in self.frames if f.name == name] + + @property + def final_text(self) -> str: + """Text of the last assistant message (same rule as the harness accumulator).""" + messages: List[str] = [] + current: List[str] = [] + for f in self.frames: + if f.name == "message_start": + if "".join(current).strip(): + messages.append("".join(current)) + current = [] + elif f.name == "content_block_delta": + if f.data.get("type") == "text" and f.data.get("text"): + current.append(str(f.data["text"])) + elif f.name == "message_stop": + if "".join(current).strip(): + messages.append("".join(current)) + current = [] + if "".join(current).strip(): + messages.append("".join(current)) + return messages[-1] if messages else "" + + @property + def first_token_ms(self) -> Optional[float]: + f = self.first("content_block_delta") + return f.t_ms if f else None + + +@dataclass +class RowResult: + row: str + status: str # PASS | FAIL | SKIP | OBSERVED + detail: str = "" + warnings: List[str] = field(default_factory=list) + data: Dict[str, Any] = field(default_factory=dict) + + +class RowFailure(Exception): + """A row's assertion failed. The message is the whole detail.""" + + +class RowSkip(Exception): + """A row cannot run here (fixture or transport missing). Not a failure.""" + + +# ============================================================ +# Transports +# ============================================================ + +OnFrame = Callable[[Frame], Awaitable[None]] + + +class Transport(Protocol): + label: str + via_app_api: bool + + async def stream_turn( + self, + payload: Dict[str, Any], + *, + on_frame: Optional[OnFrame] = None, + abort_after_deltas: Optional[int] = None, + timeout_s: float = 300.0, + ) -> TurnResult: ... + + async def get_messages(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: ... + + async def get_calls(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: ... + + async def list_session_ids(self) -> Optional[List[str]]: ... + + async def delete_session(self, session_id: str) -> int: ... + + async def interrupt(self, session_id: str, reason: str) -> int: ... + + async def steer(self, session_id: str, text: str) -> int: ... + + async def close(self) -> None: ... + + +async def _drain_sse( + response: httpx.Response, + started: float, + *, + on_frame: Optional[OnFrame], + abort_after_deltas: Optional[int], +) -> Tuple[List[Frame], bool]: + frames: List[Frame] = [] + deltas = 0 + aborted = False + async for name, data in iter_sse_events(response.aiter_lines()): + frame = Frame(name=name, data=data, t_ms=(time.monotonic() - started) * 1000) + frames.append(frame) + if on_frame is not None: + await on_frame(frame) + if name == "content_block_delta": + deltas += 1 + if abort_after_deltas is not None and deltas >= abort_after_deltas: + aborted = True + await response.aclose() + break + return frames, aborted + + +class AppApiTransport: + """Talks to app-api exactly as the SPA does: cookie session + CSRF header.""" + + via_app_api = True + + def __init__(self, base_url: str, label: str) -> None: + self.base_url = base_url.rstrip("/") + self.label = label + self._client = httpx.AsyncClient( + base_url=self.base_url, + timeout=httpx.Timeout(300.0, connect=30.0), + follow_redirects=False, + ) + self._csrf: Optional[str] = None + + # -- auth ------------------------------------------------------------- + + async def ensure_anonymous_ok(self) -> None: + """SKIP_AUTH local stack: /auth/session must answer 200 with no cookie.""" + r = await self._client.get("/auth/session") + if r.status_code != 200: + raise RuntimeError( + f"GET {self.base_url}/auth/session -> {r.status_code}. For a local run the " + "app-api needs SKIP_AUTH=true in backend/src/.env (and SKIP_AUTH_ROLES=" + "system_admin for the cache-stability row). For a deployed target pass " + "--auth cognito or --auth headless-grant." + ) + body = r.json() + self._csrf = body.get("csrf_token") or body.get("csrfToken") + user = body.get("user") or {} + logger.info("signed in (bypass) as %s", user.get("email") or user.get("user_id") or "?") + + async def login_cognito(self, username: str, password: str, cognito_domain: str) -> None: + """Hosted UI scripted login — the tests/load flow, ported to httpx.""" + r = await self._client.get("/auth/login") + if r.status_code != 302 or "/oauth2/authorize" not in r.headers.get("location", ""): + raise RuntimeError(f"GET /auth/login -> {r.status_code}; expected a 302 to /oauth2/authorize") + authorize_url = r.headers["location"] + if cognito_domain and not authorize_url.startswith(cognito_domain): + logger.warning("authorize redirect host differs from --cognito-domain; following the redirect") + page = await self._client.get(authorize_url, follow_redirects=True) + if page.status_code != 200: + raise RuntimeError(f"Cognito authorize page -> {page.status_code}") + form = _find_login_form(page.text) + payload = dict(form.fields) + payload[_pick_username_field(form)] = username + payload[form.password_field] = password # type: ignore[index] + action = urljoin(str(page.url), form.action or "") + post = await self._client.post(action, data=payload) + if post.status_code == 200: + raise RuntimeError( + "Cognito re-rendered the login form: bad credentials, unconfirmed user, or a " + "forced password change (FORCE_CHANGE_PASSWORD blocks scripted login)." + ) + location = post.headers.get("location", "") + if post.status_code == 302 and "code=" not in location: + # Cognito bounces a rejected login back to its own /login page. + raise RuntimeError( + "Cognito redirected without an authorization code: wrong credentials, an unconfirmed " + f"user, or a forced password change (-> {location[:100]})" + ) + if post.status_code != 302: + raise RuntimeError(f"Cognito login POST -> {post.status_code}, expected a 302 with ?code=") + callback = post.headers["location"] + cb = await self._client.get(callback) + if cb.status_code not in (302, 303): + raise RuntimeError(f"GET /auth/callback -> {cb.status_code}; the state cookie binding failed") + s = await self._client.get("/auth/session") + if s.status_code != 200: + raise RuntimeError(f"GET /auth/session -> {s.status_code} right after callback") + body = s.json() + self._csrf = body.get("csrf_token") or body.get("csrfToken") + if not self._csrf: + raise RuntimeError("GET /auth/session returned no csrf_token") + logger.info("signed in (cognito) as %s", (body.get("user") or {}).get("email", "?")) + + def _headers(self) -> Dict[str, str]: + h = {"Content-Type": "application/json"} + if self._csrf: + h["X-CSRF-Token"] = self._csrf + return h + + # -- turn path ---------------------------------------------------------- + + async def stream_turn( + self, + payload: Dict[str, Any], + *, + on_frame: Optional[OnFrame] = None, + abort_after_deltas: Optional[int] = None, + timeout_s: float = 300.0, + ) -> TurnResult: + started = time.monotonic() + try: + async with self._client.stream("POST", "/chat/stream", json=payload, headers=self._headers(), timeout=timeout_s) as response: + if response.status_code >= 400: + body = (await response.aread()).decode("utf-8", errors="replace") + return TurnResult( + status_code=response.status_code, + error=body[:500], + elapsed_ms=(time.monotonic() - started) * 1000, + ) + frames, aborted = await _drain_sse(response, started, on_frame=on_frame, abort_after_deltas=abort_after_deltas) + return TurnResult( + status_code=response.status_code, + frames=frames, + aborted=aborted, + elapsed_ms=(time.monotonic() - started) * 1000, + ) + except httpx.TimeoutException: + return TurnResult(status_code=0, error=f"stream exceeded {timeout_s:.0f}s") + except httpx.HTTPError as exc: + return TurnResult(status_code=0, error=f"transport: {type(exc).__name__}: {exc}") + + async def get_messages(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: + r = await self._client.get(f"/sessions/{session_id}/messages") + return r.status_code, (r.json() if r.status_code == 200 else None) + + async def get_calls(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: + r = await self._client.get(f"/admin/costs/sessions/{session_id}/calls") + return r.status_code, (r.json() if r.status_code == 200 else None) + + async def list_session_ids(self) -> Optional[List[str]]: + r = await self._client.get("/sessions", params={"limit": 200}) + if r.status_code != 200: + return None + body = r.json() + items = body.get("sessions") or body.get("items") or [] + return [str(s.get("sessionId") or s.get("session_id") or "") for s in items] + + async def delete_session(self, session_id: str) -> int: + r = await self._client.delete(f"/sessions/{session_id}", headers=self._headers()) + return r.status_code + + async def interrupt(self, session_id: str, reason: str) -> int: + r = await self._client.post(f"/sessions/{session_id}/interrupt", json={"reason": reason}, headers=self._headers()) + return r.status_code + + async def steer(self, session_id: str, text: str) -> int: + r = await self._client.post( + f"/sessions/{session_id}/steer", + json={"text": text, "entryId": f"steer-{uuid.uuid4().hex[:8]}"}, + headers=self._headers(), + ) + return r.status_code + + async def close(self) -> None: + await self._client.aclose() + + +class RuntimeTransport: + """Straight at the AgentCore Runtime data plane with a headless-grant bearer. + + Reuses the harness's URL builder, affinity header and auth strategy so the + request is byte-for-byte what a scheduled run sends. Persistence is read + in-process through the shared repositories (env resolved from SSM), which + is also why this mode needs an AWS profile for the target account. + """ + + via_app_api = False + + def __init__(self, user_id: str, prefix: str, region: str, invocations_base_url: Optional[str] = None) -> None: + from apis.shared.harness.auth import CognitoRefreshBearerAuth + from apis.shared.harness.runner import apply_runtime_session_header, build_invocations_url + + self.label = f"runtime:{prefix}" + self.user_id = user_id + self._apply_header = apply_runtime_session_header + self._url = build_invocations_url(invocations_base_url or os.environ["INFERENCE_API_URL"]) + if invocations_base_url: + self.label = f"runtime:{prefix} via {invocations_base_url}" + self._auth = CognitoRefreshBearerAuth() + self._client = httpx.AsyncClient(timeout=httpx.Timeout(300.0, connect=30.0)) + self._bearer: Optional[str] = None + self._bearer_at = 0.0 + + async def _bearer_token(self) -> str: + # Cognito access tokens live an hour; re-mint well inside that. + if self._bearer is None or time.monotonic() - self._bearer_at > 20 * 60: + self._bearer = await self._auth.mint_bearer_for_user(self.user_id) + self._bearer_at = time.monotonic() + return self._bearer + + async def stream_turn( + self, + payload: Dict[str, Any], + *, + on_frame: Optional[OnFrame] = None, + abort_after_deltas: Optional[int] = None, + timeout_s: float = 300.0, + ) -> TurnResult: + bearer = await self._bearer_token() + headers = self._apply_header( + {"Content-Type": "application/json", "Authorization": f"Bearer {bearer}"}, + payload.get("session_id"), + ) + started = time.monotonic() + try: + async with self._client.stream("POST", self._url, json=payload, headers=headers, timeout=timeout_s) as response: + if response.status_code >= 400: + body = (await response.aread()).decode("utf-8", errors="replace") + return TurnResult(status_code=response.status_code, error=body[:500]) + frames, aborted = await _drain_sse(response, started, on_frame=on_frame, abort_after_deltas=abort_after_deltas) + return TurnResult( + status_code=response.status_code, + frames=frames, + aborted=aborted, + elapsed_ms=(time.monotonic() - started) * 1000, + ) + except httpx.TimeoutException: + return TurnResult(status_code=0, error=f"stream exceeded {timeout_s:.0f}s") + except httpx.HTTPError as exc: + return TurnResult(status_code=0, error=f"transport: {type(exc).__name__}: {exc}") + + async def get_messages(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: + try: + from apis.shared.sessions.messages import get_messages + + resp = await get_messages(session_id, self.user_id) + return 200, resp.model_dump(by_alias=True) + except Exception as exc: # noqa: BLE001 — reported, never raised past the row + logger.warning("in-process get_messages failed: %s", exc) + return 0, None + + async def get_calls(self, session_id: str) -> Tuple[int, Optional[Dict[str, Any]]]: + try: + from apis.app_api.admin.costs.service import AdminCostService + + anatomy = await AdminCostService().get_session_cost_anatomy(session_id) + return 200, anatomy.model_dump(by_alias=True) + except Exception as exc: # noqa: BLE001 + logger.warning("in-process get_session_cost_anatomy failed: %s", exc) + return 0, None + + async def list_session_ids(self) -> Optional[List[str]]: + return None + + async def delete_session(self, session_id: str) -> int: + # The same soft delete app-api's DELETE /sessions/{id} performs (S#ACTIVE# + # -> S#DELETED#, cost rows kept), run in-process with the AWS profile. + try: + from apis.app_api.sessions.services.session_service import SessionService + + ok = await SessionService().delete_session(user_id=self.user_id, session_id=session_id) + return 204 if ok else 404 + except Exception as exc: # noqa: BLE001 + logger.warning("in-process delete_session(%s) failed: %s", session_id, exc) + return 0 + + async def interrupt(self, session_id: str, reason: str) -> int: + return 0 + + async def steer(self, session_id: str, text: str) -> int: + return 0 + + async def close(self) -> None: + await self._client.aclose() + + +# -- Hosted UI form parsing (ported from tests/load/agentcore_load/auth.py) --- + + +class _HtmlForm: + def __init__(self, action: Optional[str], method: str) -> None: + self.action = action + self.method = method.lower() + self.fields: Dict[str, str] = {} + self.password_field: Optional[str] = None + self.text_fields: List[str] = [] + + +class _FormParser(HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self.forms: List[_HtmlForm] = [] + self._current: Optional[_HtmlForm] = None + + def handle_starttag(self, tag: str, attrs: List[Tuple[str, Optional[str]]]) -> None: + attr = {k.lower(): (v or "") for k, v in attrs} + if tag == "form": + self._current = _HtmlForm(attr.get("action"), attr.get("method", "get")) + self.forms.append(self._current) + return + if tag != "input" or self._current is None or not attr.get("name"): + return + name = attr["name"] + kind = attr.get("type", "text").lower() + if kind == "password": + self._current.password_field = name + self._current.fields[name] = "" + elif kind in {"hidden", "text", "email", "tel"}: + self._current.fields[name] = attr.get("value", "") + if kind != "hidden": + self._current.text_fields.append(name) + + def handle_endtag(self, tag: str) -> None: + if tag == "form": + self._current = None + + +def _find_login_form(html: str) -> _HtmlForm: + p = _FormParser() + p.feed(html) + forms = [f for f in p.forms if f.password_field] + if not forms: + raise RuntimeError( + "No password form on the Cognito page. Managed login (branding v2) renders " + "client-side and cannot be scripted — use --auth headless-grant instead." + ) + return forms[0] + + +def _pick_username_field(form: _HtmlForm) -> str: + for c in ("username", "email", "signInFormUsername"): + if c in form.fields: + return c + if len(form.text_fields) == 1: + return form.text_fields[0] + raise RuntimeError(f"Could not pick the username field from {form.text_fields!r}") + + +# ============================================================ +# Contract assertions +# ============================================================ + + +RESUME_HEAD_EXTRA = frozenset({"tool_use", "tool_result", "tool_error", "message", "metadata", "tool_group_summary", "ui_resource"}) + + +def assert_frame_contract(turn: TurnResult, *, allow_interrupt: bool = False, resume: bool = False) -> List[str]: + """Invariant I1 (turn-path-ttft §6A) on one turn's frames. Returns warnings. + + ``resume=True`` widens the head: a resumed turn finishes the paused tool + first, so its ``tool_result`` (and the passthroughs around it) arrive before + the next model call's ``message_start``. + """ + warnings: List[str] = [] + names = turn.names + head_allowed = HEAD_ALLOWED | RESUME_HEAD_EXTRA if resume else HEAD_ALLOWED + if turn.status_code != 200: + raise RowFailure(f"HTTP {turn.status_code}: {turn.error}") + if not names: + raise RowFailure("stream carried no frames") + if "message_start" not in names: + raise RowFailure(f"no message_start; frames={names[:12]}") + + first_ms = names.index("message_start") + head_bad = [n for n in names[:first_ms] if n not in head_allowed] + if head_bad: + raise RowFailure(f"frames before message_start that the head contract forbids: {head_bad}; head={names[:first_ms]}") + + first_delta = next((i for i, n in enumerate(names) if n == "content_block_delta"), None) + if first_delta is not None and first_delta < first_ms: + raise RowFailure("content_block_delta arrived before message_start") + + if names[-1] != "done": + raise RowFailure(f"last frame is {names[-1]!r}, expected done") + if names.count("done") != 1: + raise RowFailure(f"done emitted {names.count('done')} times") + + if "message_stop" not in names: + raise RowFailure("no message_stop before done") + last_stop = len(names) - 1 - names[::-1].index("message_stop") + tail_bad = [n for n in names[last_stop + 1 :] if n not in TAIL_ALLOWED] + if tail_bad: + raise RowFailure(f"frames after the final message_stop that the tail contract forbids: {tail_bad}; tail={names[last_stop:]}") + + if not any(n in ("metadata", "metadata_summary") for n in names[last_stop:]): + warnings.append("no metadata/metadata_summary between the final message_stop and done") + + titles = turn.all("session_title") + if len(titles) > 1: + raise RowFailure(f"session_title emitted {len(titles)} times (max 1)") + if titles and (titles[0].data.get("title") or "").strip() == PLACEHOLDER_TITLE: + raise RowFailure("session_title carried the 'New Conversation' placeholder") + + errors = [f for f in turn.frames if f.name in ("stream_error", "error")] + if errors: + raise RowFailure(f"stream carried an error frame: {errors[0].data}") + + interrupts = [n for n in names if n in INTERRUPT_EVENTS] + if interrupts and not allow_interrupt: + raise RowFailure(f"unexpected interrupt event(s) {interrupts}") + + unknown = sorted({n for n in names if n not in KNOWN_EVENTS and not n.startswith("_")}) + if unknown: + warnings.append(f"event names not in the known set (update CLAUDE.md's table?): {unknown}") + unparseable = names.count("_unparseable") + if unparseable: + raise RowFailure(f"{unparseable} frame(s) carried unparseable JSON") + return warnings + + +def _tools_used(turn: TurnResult) -> List[str]: + return sorted({(f.data.get("tool_use") or f.data).get("name", "") for f in turn.all("tool_use")} - {""}) + + +def _message_texts(messages_body: Dict[str, Any], role: str) -> List[str]: + out: List[str] = [] + for m in messages_body.get("messages") or []: + if m.get("role") != role: + continue + parts = [c.get("text") or "" for c in (m.get("content") or []) if isinstance(c, dict)] + out.append("".join(parts)) + return out + + +def _restore_check(messages_body: Dict[str, Any], user_prompt: str, assistant_text: str) -> None: + users = _message_texts(messages_body, "user") + assistants = _message_texts(messages_body, "assistant") + if not any(user_prompt in u for u in users): + raise RowFailure(f"restored history lacks the user prompt {user_prompt[:40]!r}; users={[u[:40] for u in users]}") + want = assistant_text.strip() + if want and not any(want[:60] in a for a in assistants): + raise RowFailure(f"restored assistant text differs from the streamed text; streamed={want[:60]!r} restored={[a[:60] for a in assistants]}") + + +# ============================================================ +# Fixtures generated in-script +# ============================================================ + +PDF_TOKEN = "SMOKE-PDF-TOKEN-7731" + + +def minimal_pdf(text: str) -> bytes: + """A valid single-page PDF whose only content is ``text`` (Helvetica 24pt).""" + stream = f"BT /F1 24 Tf 72 700 Td ({text}) Tj ET".encode("latin-1") + objs = [ + b"<< /Type /Catalog /Pages 2 0 R >>", + b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>", + b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>", + b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>", + b"<< /Length " + str(len(stream)).encode() + b" >>\nstream\n" + stream + b"\nendstream", + ] + out = bytearray(b"%PDF-1.4\n") + offsets = [] + for i, body in enumerate(objs, start=1): + offsets.append(len(out)) + out += f"{i} 0 obj\n".encode() + body + b"\nendobj\n" + xref = len(out) + out += f"xref\n0 {len(objs) + 1}\n0000000000 65535 f \n".encode() + for off in offsets: + out += f"{off:010d} 00000 n \n".encode() + out += f"trailer\n<< /Size {len(objs) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode() + return bytes(out) + + +CSV_ROWS = 7 +CSV_TEXT = "id,city,temp_c\n" + "\n".join(f"{i},City{i},{10 + i}" for i in range(1, CSV_ROWS + 1)) + "\n" + + +# ============================================================ +# The matrix +# ============================================================ + + +@dataclass +class Context: + transport: Transport + args: argparse.Namespace + created_sessions: List[str] = field(default_factory=list) + observed: Dict[str, int] = field(default_factory=dict) + first_token_ms: List[float] = field(default_factory=list) + shared_session: str = "" + shared_turns: List[Tuple[str, str]] = field(default_factory=list) # (prompt, final_text) + + def new_session(self, prefix: str = SESSION_PREFIX) -> str: + sid = f"{prefix}{uuid.uuid4().hex[:16]}" + if prefix == SESSION_PREFIX: + self.created_sessions.append(sid) + return sid + + def base_payload(self, session_id: str, message: str, **extra: Any) -> Dict[str, Any]: + payload: Dict[str, Any] = {"session_id": session_id, "message": message, "enabled_tools": []} + if self.args.model_id: + payload["model_id"] = self.args.model_id + payload.update(extra) + return payload + + def note(self, turn: TurnResult) -> None: + for f in turn.frames: + if f.name in OBSERVED_ONLY: + self.observed[f.name] = self.observed.get(f.name, 0) + 1 + elif f.name == "agent_status": + key = f"agent_status:{f.data.get('phase', '?')}" + self.observed[key] = self.observed.get(key, 0) + 1 + if turn.first_token_ms is not None: + self.first_token_ms.append(turn.first_token_ms) + + +async def row_plain_first_turn(ctx: Context) -> RowResult: + sid = ctx.new_session() + ctx.shared_session = sid + prompt = "Reply with exactly the single word PONG and nothing else." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + text = turn.final_text + if "PONG" not in text.upper(): + warnings.append(f"model did not answer PONG (got {text[:40]!r}); the contract checks still passed") + ctx.shared_turns.append((prompt, text)) + title = turn.first("session_title") + data = { + "frames": len(turn.frames), + "firstTokenMs": round(turn.first_token_ms or 0), + "sessionTitle": (title.data.get("title") if title else None), + "agentStatusPhases": sorted({f.data.get("phase", "") for f in turn.all("agent_status")}), + } + return RowResult("plain_first_turn", "PASS", f"{len(turn.frames)} frames, first token {data['firstTokenMs']}ms", warnings, data) + + +async def row_second_turn_cache(ctx: Context) -> RowResult: + if not ctx.shared_session: + raise RowSkip("first turn did not run") + sid = ctx.shared_session + prompt = "Reply with exactly the single word PING and nothing else." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + ctx.shared_turns.append((prompt, turn.final_text)) + prepared = next((f for f in turn.all("agent_status") if f.data.get("phase") == "prepared"), None) + build_ms = int(prepared.data.get("durationMs") or 0) if prepared else 0 + if build_ms > 200: + warnings.append(f"second turn rebuilt the agent in {build_ms}ms (cache miss on a warm session?)") + + # Cache stability (I2/I3): give the write path a moment to land the C# rows. + calls_body: Optional[Dict[str, Any]] = None + code = 0 + for _ in range(6): + code, calls_body = await ctx.transport.get_calls(sid) + if calls_body and len(calls_body.get("calls") or []) >= 2: + break + await asyncio.sleep(1.0) + if code in (401, 403): + return RowResult( + "second_turn_cache", + "PASS", + "frame contract ok; cache rows unreadable (admin endpoint -> %d, run as system_admin to read them)" % code, + warnings, + ) + if not calls_body: + return RowResult("second_turn_cache", "PASS", f"frame contract ok; cost anatomy unavailable (HTTP {code})", warnings) + calls = calls_body.get("calls") or [] + if len(calls) < 2: + raise RowFailure(f"expected >=2 call rows for two turns, found {len(calls)}") + rows = sorted(calls, key=lambda c: c.get("timestamp") or "") + prev, last = rows[-2], rows[-1] + status = last.get("cacheStatus") + if status == "partial_miss": + raise RowFailure("second turn is a partial_miss: the history behind the tools+system prefix was re-written") + fp_prev = prev.get("prefixFingerprints") or {} + fp_last = last.get("prefixFingerprints") or {} + for key in ("toolConfigHash", "systemPromptHash"): + if fp_prev.get(key) and fp_last.get(key) and fp_prev[key] != fp_last[key]: + raise RowFailure(f"{key} changed between consecutive turns of one session ({fp_prev[key][:10]} -> {fp_last[key][:10]})") + if status not in ("hit", "uncached", None): + warnings.append(f"second-turn cacheStatus={status!r} (short prompts sit under Bedrock's minimum cacheable prefix, so 'uncached' is normal)") + data = { + "cacheStatus": status, + "cacheRead": last.get("cacheReadTokens"), + "cacheWrite": last.get("cacheWriteTokens"), + "calls": len(calls), + "buildMs": build_ms, + } + return RowResult("second_turn_cache", "PASS", f"cacheStatus={status}, hashes stable across {len(calls)} calls", warnings, data) + + +async def row_restore(ctx: Context) -> RowResult: + if not ctx.shared_turns: + raise RowSkip("no turns to restore") + code, body = await ctx.transport.get_messages(ctx.shared_session) + if code != 200 or body is None: + raise RowFailure(f"GET /messages -> {code}") + for prompt, text in ctx.shared_turns: + _restore_check(body, prompt, text) + n = len(body.get("messages") or []) + if n < 2 * len(ctx.shared_turns): + raise RowFailure(f"restored {n} messages for {len(ctx.shared_turns)} turns") + return RowResult("restore_after_reload", "PASS", f"{n} messages restored with matching text", data={"messages": n}) + + +async def row_tool_turn(ctx: Context) -> RowResult: + tool = ctx.args.tool + sid = ctx.new_session() + prompt = f"Use the {tool} tool to compute 1234*5678. Reply with only the number the tool returned." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, enabled_tools=[tool])) + ctx.note(turn) + warnings = assert_frame_contract(turn) + uses = turn.all("tool_use") + if not uses: + raise RowFailure(f"no tool_use frame — is {tool!r} granted to this user (RBAC) and enabled? final text: {turn.final_text[:80]!r}") + if not turn.all("tool_result"): + raise RowFailure("tool_use without a tool_result frame") + phases = [f.data.get("phase") for f in turn.all("agent_status")] + if turn.all("agent_status") and not {"tool_start", "tool_end"} <= set(phases): + warnings.append(f"agent_status present but tool_start/tool_end missing: {sorted(set(phases))}") + if "7006652" not in turn.final_text.replace(",", ""): + warnings.append(f"answer did not contain 7006652: {turn.final_text[:60]!r}") + names = sorted({(f.data.get("tool_use") or f.data).get("name", "") for f in uses} - {""}) + return RowResult("tool_turn", "PASS", f"tools used: {names}; {len(turn.all('tool_result'))} result(s)", warnings, {"tools": names}) + + +ASK_PROMPT = ( + "Do not answer this message directly. Your first and only action is to call the ask_user_question " + "tool once, with a single question: header 'Color', question 'Which color do you prefer?', options " + "Red and Blue, multiSelect false. Only after my answer arrives, reply with exactly 'You picked " + ".' and nothing else." +) + + +async def row_ask_user_question(ctx: Context) -> RowResult: + # Whether the model calls the tool is its decision, so a single attempt is a + # coin toss on a weaker model. Three fresh sessions before calling it a failure. + warnings: List[str] = [] + turn: Optional[TurnResult] = None + q: Optional[Frame] = None + sid = "" + for attempt in range(1, 4): + sid = ctx.new_session() + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, ASK_PROMPT, enabled_tools=["ask_user_question"])) + ctx.note(turn) + warnings += assert_frame_contract(turn, allow_interrupt=True) + q = turn.first("user_question_required") + if q is not None: + if attempt > 1: + warnings.append(f"model only called ask_user_question on attempt {attempt}") + break + assert turn is not None + if q is None: + raise RowFailure( + "no user_question_required frame in 3 attempts — is ASK_USER_QUESTION_ENABLED on and the tool " + f"granted, or does this model ignore the directive? last answer: {turn.final_text[:60]!r}" + ) + names = turn.names + if names.index("user_question_required") < len(names) - 1 - names[::-1].index("message_stop"): + raise RowFailure("user_question_required arrived before the final message_stop") + interrupt_id = q.data.get("interruptId") + questions = q.data.get("questions") or [] + if not interrupt_id or not questions: + raise RowFailure(f"user_question_required lacks interruptId/questions: {q.data}") + header = questions[0].get("header") or "Color" + options = [o.get("label") for o in (questions[0].get("options") or [])] + pick = next((o for o in options if o and o.lower().startswith("red")), options[0] if options else "Red") + + resume = await ctx.transport.stream_turn( + { + "session_id": sid, + "message": "", + "enabled_tools": ["ask_user_question"], + **({"model_id": ctx.args.model_id} if ctx.args.model_id else {}), + "interrupt_responses": [{"interruptId": interrupt_id, "response": {"answers": {header: {"selected": [pick], "text": None}}}}], + } + ) + ctx.note(resume) + warnings += assert_frame_contract(resume, resume=True) + if resume.all("user_question_required"): + raise RowFailure("the resumed turn re-raised user_question_required (a null/invalid answer re-asks forever)") + if pick.lower() not in resume.final_text.lower(): + warnings.append(f"resumed answer did not echo {pick!r}: {resume.final_text[:60]!r}") + code, body = await ctx.transport.get_messages(sid) + if code == 200 and body is not None and body.get("pendingInterrupts"): + raise RowFailure("pendingInterrupts still non-empty after a successful resume") + return RowResult( + "ask_user_question_resume", + "PASS", + f"paused with {len(questions)} question(s), resumed with {pick!r}, answer: {resume.final_text[:40]!r}", + warnings, + {"options": options}, + ) + + +async def row_tool_approval(ctx: Context) -> RowResult: + tool = ctx.args.approval_tool + if not tool: + raise RowSkip("pass --approval-tool (e.g. an MCP tool flagged needsApproval) to run") + sid = ctx.new_session() + prompt = ctx.args.approval_prompt or f"Call the {tool} tool now and tell me what it returned." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, enabled_tools=[tool])) + ctx.note(turn) + warnings = assert_frame_contract(turn, allow_interrupt=True) + ev = turn.first("tool_approval_required") + if ev is None: + raise RowFailure(f"no tool_approval_required frame; frames={sorted(set(turn.names))}") + interrupt_id = ev.data.get("interruptId") + resume = await ctx.transport.stream_turn( + { + "session_id": sid, + "message": "", + "enabled_tools": [tool], + **({"model_id": ctx.args.model_id} if ctx.args.model_id else {}), + "interrupt_responses": [{"interruptId": interrupt_id, "response": "approved"}], + } + ) + ctx.note(resume) + warnings += assert_frame_contract(resume, resume=True) + if not resume.all("tool_result"): + raise RowFailure("approved resume produced no tool_result") + return RowResult("tool_approval_resume", "PASS", f"approved {ev.data.get('toolName')} and the turn finished", warnings) + + +async def row_duplicate_and_stop(ctx: Context) -> List[RowResult]: + if not ctx.transport.via_app_api: + raise RowSkip("single-flight 409 and Stop live on app-api; not reachable in runtime-direct mode") + sid = ctx.new_session() + long_prompt = "Count from 1 to 150, one number per line, with no other text." + started = asyncio.Event() + + async def on_frame(frame: Frame) -> None: + if frame.name == "message_start": + started.set() + + task = asyncio.create_task(ctx.transport.stream_turn(ctx.base_payload(sid, long_prompt), on_frame=on_frame, abort_after_deltas=12)) + results: List[RowResult] = [] + try: + await asyncio.wait_for(started.wait(), timeout=120) + except asyncio.TimeoutError: + task.cancel() + raise RowFailure("first stream never reached message_start") + + dup = await ctx.transport.stream_turn(ctx.base_payload(sid, "Say hi."), timeout_s=60) + if dup.status_code != 409: + results.append( + RowResult("duplicate_send_409", "FAIL", f"second send while streaming -> HTTP {dup.status_code} (expected 409); {dup.error or ''}"[:200]) + ) + else: + results.append(RowResult("duplicate_send_409", "PASS", f"409 while streaming: {(dup.error or '')[:80]}")) + + first = await task + ctx.note(first) + if not first.aborted: + results.append(RowResult("stop_mid_answer", "FAIL", "the long turn finished before the client could abort it; lengthen the prompt")) + return results + code = await ctx.transport.interrupt(sid, "user_stopped") + if code not in (204, 200): + results.append(RowResult("stop_mid_answer", "FAIL", f"POST /interrupt user_stopped -> {code}")) + return results + + # The lease is released when the server notices the drop (heartbeat ~10s). + deadline = time.monotonic() + 45 + attempts = 0 + nxt: Optional[TurnResult] = None + while time.monotonic() < deadline: + attempts += 1 + nxt = await ctx.transport.stream_turn(ctx.base_payload(sid, "Reply with exactly the single word RESUMED.")) + if nxt.status_code != 409: + break + await asyncio.sleep(2.0) + assert nxt is not None + if nxt.status_code == 409: + results.append(RowResult("stop_mid_answer", "FAIL", f"still 409 {attempts} tries / 45s after Stop — the lease leaked")) + return results + ctx.note(nxt) + warnings = assert_frame_contract(nxt) + code, body = await ctx.transport.get_messages(sid) + detail = f"resend accepted after {attempts} try(ies)" + if code == 200 and body is not None: + assistants = _message_texts(body, "assistant") + partial = any(("1" in a and "150" not in a) for a in assistants[:-1]) if len(assistants) > 1 else False + detail += ( + "; partial answer persisted" if partial else "; partial answer not visible (a short turn can finish server-side and clear the marker)" + ) + results.append(RowResult("stop_mid_answer", "PASS", detail, warnings, {"resendAttempts": attempts})) + return results + + +async def row_preview_session(ctx: Context) -> RowResult: + sid = ctx.new_session(PREVIEW_PREFIX) + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, "Reply with exactly the single word PREVIEW.")) + ctx.note(turn) + warnings = assert_frame_contract(turn) + code, body = await ctx.transport.get_messages(sid) + if code == 200 and body and (body.get("messages") or []): + raise RowFailure(f"preview session persisted {len(body['messages'])} messages") + listed = await ctx.transport.list_session_ids() + if listed is not None and sid in listed: + raise RowFailure("preview session appeared in the sidebar list") + return RowResult("preview_session", "PASS", f"turn completed; GET /messages -> {code}; not listed", warnings) + + +async def row_steer_observed(ctx: Context) -> RowResult: + if not ctx.transport.via_app_api: + raise RowSkip("steer endpoint lives on app-api") + tool = ctx.args.tool + sid = ctx.new_session() + prompt = f"Use the {tool} tool three separate times, one call per step, to compute 11*11, then 12*12, then 13*13. Then list the three results." + steer_code: Dict[str, int] = {} + fired = asyncio.Event() + + async def on_frame(frame: Frame) -> None: + if frame.name == "tool_use" and not fired.is_set(): + fired.set() + steer_code["code"] = await ctx.transport.steer(sid, "Also compute 14*14 and include it.") + + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, enabled_tools=[tool]), on_frame=on_frame) + ctx.note(turn) + warnings = assert_frame_contract(turn) + code = steer_code.get("code") + if code == 404: + raise RowSkip("POST /steer -> 404 (MID_TURN_STEERING_ENABLED is off)") + if code is None: + raise RowSkip("the model made no tool call, so there was no boundary to steer at") + if code not in (200, 201, 202): + raise RowFailure(f"POST /steer -> {code}") + applied = bool(turn.all("steering_applied")) + detail = ( + "steering_applied landed at a tool boundary" + if applied + else "steer queued but lost the race with the turn's end (the SPA sends it as the next turn)" + ) + return RowResult("steer_mid_turn", "OBSERVED", detail, warnings, {"applied": applied, "steerHttp": code}) + + +async def row_attach_pdf(ctx: Context) -> RowResult: + if not ctx.args.with_attachments: + raise RowSkip("pass --with-attachments") + sid = ctx.new_session() + prompt = "What token is written in the attached PDF? Reply with just the token." + files = [{"filename": "smoke.pdf", "content_type": "application/pdf", "bytes": base64.b64encode(minimal_pdf(PDF_TOKEN)).decode()}] + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, files=files, enabled_tools=None)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + if PDF_TOKEN not in turn.final_text.upper(): + warnings.append(f"model did not read back the token: {turn.final_text[:60]!r}") + code, body = await ctx.transport.get_messages(sid) + if code != 200 or body is None: + raise RowFailure(f"GET /messages -> {code}") + users = body.get("messages") or [] + has_doc = any( + c.get("type") == "document" or c.get("document") for m in users if m.get("role") == "user" for c in (m.get("content") or []) + ) or any("smoke.pdf" in u for u in _message_texts(body, "user")) + if not has_doc: + raise RowFailure("restored user message carries neither a document block nor the attachment marker") + return RowResult("attach_pdf", "PASS", f"inline PDF accepted, answer streamed, attachment restored; tools={_tools_used(turn)}", warnings) + + +async def row_attach_csv(ctx: Context) -> RowResult: + if not ctx.args.with_attachments: + raise RowSkip("pass --with-attachments") + sid = ctx.new_session() + prompt = ( + "Use your spreadsheet tools on the attached file smoke.csv (ignore anything you remember about " + "other files). How many data rows does it have, excluding the header? Reply with just the number." + ) + files = [{"filename": "smoke.csv", "content_type": "text/csv", "bytes": base64.b64encode(CSV_TEXT.encode()).decode()}] + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, files=files, enabled_tools=None)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + if str(CSV_ROWS) not in turn.final_text: + warnings.append(f"answer did not say {CSV_ROWS}: {turn.final_text[:60]!r}") + code, body = await ctx.transport.get_messages(sid) + if code != 200 or body is None: + raise RowFailure(f"GET /messages -> {code}") + marker = any("[Attached files:" in u or "smoke.csv" in u for u in _message_texts(body, "user")) + if not marker: + raise RowFailure("restored user message lacks the [Attached files: …] marker for the diverted CSV") + # The guidance note tells the model the file is reachable through the + # spreadsheet tools; hold it to that. A diverted file the tools cannot + # see is the inline-attachment hole this row exists to catch. + tools = _tools_used(turn) + if not {"list_spreadsheets", "analyze_spreadsheet"} & set(tools): + raise RowFailure(f"model did not use the spreadsheet tools on the diverted CSV; tools={tools}") + seen = any("smoke.csv" in json.dumps(f.data) for f in turn.all("tool_result")) + if not seen: + raise RowFailure("spreadsheet tools ran but no tool_result mentions smoke.csv — the diverted file is not visible to them") + return RowResult("attach_csv", "PASS", f"CSV diverted, visible to the tools, marker persisted; tools={tools}", warnings) + + +async def row_skill_invoke(ctx: Context) -> RowResult: + skill = ctx.args.skill_id + if not skill: + raise RowSkip("pass --skill-id for a skill granted to this user") + sid = ctx.new_session() + prompt = ctx.args.skill_prompt or "Follow the invoked skill and reply briefly." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, enabled_skills=[skill], invoked_skills=[skill], enabled_tools=None)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + names = {(f.data.get("tool_use") or f.data).get("name", "") for f in turn.all("tool_use")} + if "skills" not in names: + raise RowFailure(f"the skills tool was not called for an invoked skill; tools called: {sorted(names)}") + return RowResult("skill_invoke", "PASS", "skills tool activated the invoked skill", warnings) + + +async def row_kb_agent(ctx: Context) -> RowResult: + agent = ctx.args.agent_id + if not agent: + raise RowSkip("pass --agent-id for an agent with a knowledge base") + sid = ctx.new_session() + prompt = ctx.args.agent_prompt or "Using your knowledge base, summarize one document in two sentences." + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, prompt, rag_assistant_id=agent, enabled_tools=None)) + ctx.note(turn) + warnings = assert_frame_contract(turn) + cites = turn.all("citation") + if cites: + names = turn.names + if names.index("citation") > names.index("message_start"): + raise RowFailure("citation frames arrived after message_start") + else: + warnings.append("no citation frames (empty KB, show_citations off, or no retrieval hit)") + return RowResult("kb_agent_first_turn", "PASS", f"{len(cites)} citation frame(s) before message_start", warnings) + + +async def row_quota_exceeded(ctx: Context) -> RowResult: + if not ctx.args.expect_quota_exceeded: + raise RowSkip("pass --expect-quota-exceeded when running as a user on an exhausted test tier") + sid = ctx.new_session() + turn = await ctx.transport.stream_turn(ctx.base_payload(sid, "Reply with exactly the single word PONG.")) + ctx.note(turn) + warnings = assert_frame_contract(turn) + stop = turn.all("message_stop") + reason = stop[-1].data.get("stopReason") if stop else None + if reason != "quota_exceeded" and not turn.all("quota_exceeded"): + raise RowFailure(f"expected a quota_exceeded refusal, got stopReason={reason!r}") + code, calls = await ctx.transport.get_calls(sid) + if calls and (calls.get("calls") or []): + raise RowFailure("a refused turn still produced a model call row") + return RowResult("quota_exceeded", "PASS", "conversational refusal, no model call", warnings) + + +ROWS: List[Tuple[str, Callable[[Context], Awaitable[Any]]]] = [ + ("plain_first_turn", row_plain_first_turn), + ("second_turn_cache", row_second_turn_cache), + ("restore_after_reload", row_restore), + ("tool_turn", row_tool_turn), + ("ask_user_question_resume", row_ask_user_question), + ("tool_approval_resume", row_tool_approval), + ("duplicate_send_409+stop_mid_answer", row_duplicate_and_stop), + ("preview_session", row_preview_session), + ("steer_mid_turn", row_steer_observed), + ("attach_pdf", row_attach_pdf), + ("attach_csv", row_attach_csv), + ("skill_invoke", row_skill_invoke), + ("kb_agent_first_turn", row_kb_agent), + ("quota_exceeded", row_quota_exceeded), +] + + +# ============================================================ +# Runner +# ============================================================ + + +def _looks_like_prod(base_url: Optional[str], prefix: Optional[str]) -> bool: + if base_url: + host = urlsplit(base_url if "://" in base_url else f"https://{base_url}").hostname or "" + if host.lower() in PROD_HOSTS: + return True + if prefix and any(marker in prefix.lower() for marker in PROD_PREFIX_MARKERS): + return True + return False + + +def resolve_runtime_env(prefix: str, region: str) -> None: + """Export the env the harness + shared repositories expect, from SSM + conventions. + + One definition of how a deployed prefix maps onto env vars, extending the + spike driver's resolver with the tables the in-process readers need. + """ + import boto3 + + ssm = boto3.client("ssm", region_name=region) + sts = boto3.client("sts", region_name=region) + account = sts.get_caller_identity()["Account"] + + def param(name: str, default: Optional[str] = None) -> str: + try: + return ssm.get_parameter(Name=f"/{prefix}/{name}")["Parameter"]["Value"] + except Exception: # noqa: BLE001 + if default is None: + raise + return default + + runtime_id = param("inference-api/runtime-id") + runtime_arn = f"arn:aws:bedrock-agentcore:{region}:{account}:runtime/{runtime_id}" + env = { + "AWS_REGION": region, + "INFERENCE_API_URL": f"https://bedrock-agentcore.{region}.amazonaws.com/runtimes/{runtime_arn}", + "BFF_SESSIONS_TABLE_NAME": f"{prefix}-bff-sessions", + "COGNITO_BFF_APP_CLIENT_ID": param("auth/cognito/bff-app-client-id"), + "COGNITO_BFF_APP_CLIENT_SECRET_ARN": f"{prefix}-cognito-bff-app-client-secret", + "DYNAMODB_SESSIONS_METADATA_TABLE_NAME": param("cost-tracking/sessions-metadata-table-name", f"{prefix}-sessions-metadata"), + "DYNAMODB_COST_SUMMARY_TABLE_NAME": param("cost-tracking/user-cost-summary-table-name", f"{prefix}-user-cost-summary"), + "DYNAMODB_SYSTEM_ROLLUP_TABLE_NAME": param("cost-tracking/system-cost-rollup-table-name", f"{prefix}-system-cost-rollup"), + "DYNAMODB_ASSISTANTS_TABLE_NAME": param("rag/assistants-table-name", f"{prefix}-assistants"), + "DYNAMODB_USER_SETTINGS_TABLE_NAME": param("settings/user-settings-table-name", f"{prefix}-user-settings"), + "AGENTCORE_MEMORY_ID": param("inference-api/memory-id", ""), + } + for k, v in env.items(): + os.environ.setdefault(k, v) + + +async def build_transport(args: argparse.Namespace) -> Transport: + if args.auth == "headless-grant": + if not args.user_id or not args.prefix: + raise SystemExit("--auth headless-grant needs --user-id and --prefix") + resolve_runtime_env(args.prefix, args.region) + return RuntimeTransport(args.user_id, args.prefix, args.region, args.invocations_url) + if not args.base_url: + raise SystemExit("--base-url is required unless --auth headless-grant") + t = AppApiTransport(args.base_url, label=args.base_url) + if args.auth == "cognito": + user = os.environ.get("SMOKE_USERNAME") + pw = os.environ.get("SMOKE_PASSWORD") + if not user or not pw: + raise SystemExit("--auth cognito reads SMOKE_USERNAME / SMOKE_PASSWORD from the environment") + await t.login_cognito(user, pw, args.cognito_domain or "") + else: + await t.ensure_anonymous_ok() + return t + + +def _print_report(results: List[RowResult], ctx: Context, label: str) -> None: + width = max(len(r.row) for r in results) + 2 + print() + print(f"smoke_turns against {label}") + print("=" * (width + 70)) + for r in results: + print(f"{r.row.ljust(width)} {r.status.ljust(8)} {r.detail}") + for w in r.warnings: + print(f"{''.ljust(width)} WARN {w}") + print("-" * (width + 70)) + counts = {s: sum(1 for r in results if r.status == s) for s in ("PASS", "FAIL", "SKIP", "OBSERVED")} + print(" ".join(f"{k} {v}" for k, v in counts.items())) + if ctx.first_token_ms: + ms = sorted(ctx.first_token_ms) + print(f"first token (client clock): min {ms[0]:.0f}ms median {ms[len(ms) // 2]:.0f}ms max {ms[-1]:.0f}ms over {len(ms)} turns") + if ctx.observed: + print("observed side-channel frames: " + ", ".join(f"{k}={v}" for k, v in sorted(ctx.observed.items()))) + print() + + +async def main_async(args: argparse.Namespace) -> int: + if _looks_like_prod(args.base_url, args.prefix) and not args.allow_prod: + print("refusing: target looks like production (pass --allow-prod if you really mean it)", file=sys.stderr) + return 2 + try: + transport = await build_transport(args) + except (SystemExit, RuntimeError) as exc: + print(f"setup: {exc}", file=sys.stderr) + return 2 + + if args.delete_sessions: + codes = {sid: await transport.delete_session(sid) for sid in args.delete_sessions.split(",") if sid} + await transport.close() + for sid, code in codes.items(): + print(f"{sid} -> {code}") + return 0 if all(c in (200, 204) for c in codes.values()) else 1 + + ctx = Context(transport=transport, args=args) + results: List[RowResult] = [] + only = set(args.only.split(",")) if args.only else None + try: + for name, fn in ROWS: + if only and not any(part in only for part in name.split("+")): + continue + try: + out = await fn(ctx) + results.extend(out if isinstance(out, list) else [out]) + except RowSkip as skip: + results.append(RowResult(name, "SKIP", str(skip))) + except RowFailure as failure: + results.append(RowResult(name, "FAIL", str(failure))) + if args.fail_fast: + break + except Exception as exc: # noqa: BLE001 — a crashed row is a failed row, not a crashed run + logger.exception("row %s crashed", name) + results.append(RowResult(name, "FAIL", f"{type(exc).__name__}: {exc}")) + if args.fail_fast: + break + finally: + if ctx.created_sessions and not args.keep: + deleted = 0 + for sid in ctx.created_sessions: + if await transport.delete_session(sid) in (200, 204): + deleted += 1 + logger.info("deleted %d/%d smoke sessions", deleted, len(ctx.created_sessions)) + if deleted < len(ctx.created_sessions): + logger.info("not deleted: %s", ", ".join(ctx.created_sessions)) + elif ctx.created_sessions: + logger.info("kept sessions: %s", ", ".join(ctx.created_sessions)) + await transport.close() + + _print_report(results, ctx, transport.label) + if args.json: + with open(args.json, "w", encoding="utf-8") as fh: + json.dump( + { + "target": transport.label, + "rows": [r.__dict__ for r in results], + "observed": ctx.observed, + "firstTokenMs": ctx.first_token_ms, + "sessions": ctx.created_sessions, + }, + fh, + indent=2, + default=str, + ) + return 1 if any(r.status == "FAIL" for r in results) else 0 + + +def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--base-url", help="app-api origin, e.g. http://127.0.0.1:8000 or https://host/api") + p.add_argument("--auth", choices=["none", "cognito", "headless-grant"], default="none") + p.add_argument("--cognito-domain", help="Hosted UI origin (for --auth cognito; informational)") + p.add_argument("--user-id", help="Cognito sub owning the headless grant (for --auth headless-grant)") + p.add_argument("--prefix", help="CDK project prefix, e.g. dev-boisestateai-v2 (for --auth headless-grant)") + p.add_argument("--region", default=os.environ.get("AWS_REGION", "us-west-2")) + p.add_argument( + "--invocations-url", + help="headless-grant mode only: send turns to this inference-api base instead of the Runtime " + "(e.g. http://127.0.0.1:8001 to test a local branch with a real dev bearer)", + ) + p.add_argument("--model-id", help="model for every turn; default = the user's default model") + p.add_argument("--tool", default="calculator", help="a granted, deterministic tool for the tool rows") + p.add_argument("--approval-tool", help="tool id flagged needsApproval (enables tool_approval_resume)") + p.add_argument("--approval-prompt", help="prompt that makes the model call --approval-tool") + p.add_argument("--skill-id", help="granted skill id (enables skill_invoke)") + p.add_argument("--skill-prompt") + p.add_argument("--agent-id", help="assistant/agent id with a knowledge base (enables kb_agent_first_turn)") + p.add_argument("--agent-prompt") + p.add_argument("--with-attachments", action="store_true", help="run the PDF and CSV rows") + p.add_argument("--expect-quota-exceeded", action="store_true", help="the user is on an exhausted tier") + p.add_argument("--only", help="comma-separated row names to run") + p.add_argument("--fail-fast", action="store_true") + p.add_argument("--keep", action="store_true", help="do not delete the sessions this run created") + p.add_argument("--delete-sessions", help="comma-separated session ids to delete, then exit (no turns)") + p.add_argument("--json", help="write the report to this path") + p.add_argument("--allow-prod", action="store_true") + return p.parse_args(argv) + + +if __name__ == "__main__": + sys.exit(asyncio.run(main_async(parse_args()))) diff --git a/backend/src/agents/main_agent/base_agent.py b/backend/src/agents/main_agent/base_agent.py index 5e59d1abf..05a211d51 100644 --- a/backend/src/agents/main_agent/base_agent.py +++ b/backend/src/agents/main_agent/base_agent.py @@ -33,7 +33,7 @@ from agents.main_agent.streaming import StreamCoordinator from apis.shared.tools.scoped_ids import base_tool_id -from apis.shared.observability.build_stages import mark_stage +from apis.shared.observability.build_stages import mark_stage, record_detail logger = logging.getLogger(__name__) @@ -504,11 +504,17 @@ def _build_filtered_tools(self) -> List: Returns: list: Combined list of local tools + MCP clients + + Each phase closes a ``tools.*`` build sub-stage (P1a in + docs/specs/turn-path-ttft.md); ``groups`` sums them back into + ``agent_build.tools``. Every mark is taken on this thread, after the + executor hop returns — contextvars do not cross into the pool. """ filter_result = self.tool_filter.filter_tools_extended(self.enabled_tools) local_tools = filter_result.local_tools gateway_tool_ids = filter_result.gateway_tool_ids external_mcp_tool_ids = filter_result.external_mcp_tool_ids + mark_stage("tools.filter") # Get gateway client and add to tools if available if gateway_tool_ids: @@ -524,6 +530,7 @@ def _build_filtered_tools(self) -> List: ) if gateway_client: local_tools = self.gateway_integration.add_to_tool_list(local_tools) + mark_stage("tools.gateway") # Load external MCP tools if external_mcp_tool_ids: @@ -540,6 +547,9 @@ def _build_filtered_tools(self) -> List: workload_access_token = BedrockAgentCoreContext.get_workload_access_token() external_integration = get_external_mcp_integration() + # Filled inside the executor's loop — a plain list crosses the + # thread boundary where a contextvar recorder would not. + server_timings: List[dict] = [] async def _load_with_context(): if oauth2_callback_url: @@ -550,6 +560,7 @@ async def _load_with_context(): external_mcp_tool_ids, user_id=self.user_id, auth_token=self.auth_token, + timings=server_timings, ) # Probe with ``get_running_loop`` rather than ``get_event_loop``: @@ -577,11 +588,17 @@ async def _load_with_context(): local_tools.append(client) logger.info(f"Added {len(external_clients)} external MCP clients to tools") + # Which server owned `tools.mcp`. The hop's thread and event-loop + # startup are in the stage but in no server's numbers; the gap + # between the two is that overhead. + record_detail("mcpServers", server_timings) + mark_stage("tools.mcp") # Append context-bound tools (e.g., spreadsheet analysis) created per-request if self.extra_tools: local_tools.extend(self.extra_tools) logger.info(f"Added {len(self.extra_tools)} extra context-bound tools") + mark_stage("tools.extra") return local_tools diff --git a/backend/src/agents/main_agent/chat_agent.py b/backend/src/agents/main_agent/chat_agent.py index 8b3c13a77..f2622cb0d 100644 --- a/backend/src/agents/main_agent/chat_agent.py +++ b/backend/src/agents/main_agent/chat_agent.py @@ -7,7 +7,7 @@ import logging import os -from typing import Any, AsyncGenerator, Dict, List, Optional +from typing import Any, AsyncGenerator, Callable, Dict, List, Optional from agents.main_agent.base_agent import BaseAgent from agents.main_agent.core import AgentFactory @@ -53,10 +53,9 @@ def __init__( def _create_agent(self) -> None: """Create Strands Agent with filtered tools, hooks, and skills plugin.""" try: + # Closes its own `tools.*` sub-stages, external MCP pre-flight + # included; `groups` sums them into `agent_build.tools`. tools = self._build_filtered_tools() - # External MCP pre-flight lives in here — the spec's standing - # (and unverified) hypothesis for the cold build. - mark_stage("tools") hooks = self._create_hooks() mark_stage("hooks") @@ -164,6 +163,8 @@ async def stream_async( turn_project_id: Optional[str] = None, turn_lease: Any = None, turn_started_at: Optional[float] = None, + poll_side_frame: Optional[Callable[[], Optional[str]]] = None, + turn_clock: Any = None, ) -> AsyncGenerator[str, None]: """ Stream agent responses. @@ -198,6 +199,12 @@ async def stream_async( off the agent for the same reason as `turn_agent_id`: the agent instance is cached across turns, so per-turn state must never live on it (#741/#751). + poll_side_frame: Non-blocking check for a frame produced outside the + agent stream (the first turn's `session_title`), polled by the + coordinator's live status merge. Per turn, like `turn_lease`. + turn_clock: The turn's latency clock, continued by the coordinator + to the first model output (`turn_first_token`). Per turn, like + `turn_lease`; None disables it. Yields: str: SSE formatted events @@ -234,5 +241,7 @@ async def stream_async( turn_project_id=turn_project_id, turn_lease=turn_lease, turn_started_at=turn_started_at, + poll_side_frame=poll_side_frame, + turn_clock=turn_clock, ): yield event diff --git a/backend/src/agents/main_agent/core/model_config.py b/backend/src/agents/main_agent/core/model_config.py index 01c9815e2..2855bfa81 100644 --- a/backend/src/agents/main_agent/core/model_config.py +++ b/backend/src/agents/main_agent/core/model_config.py @@ -415,8 +415,25 @@ def bedrock_cache_points_supported(self) -> bool: ) def to_bedrock_config(self) -> Dict[str, Any]: - """Convert to BedrockModel kwargs, translating canonical inference params.""" + """Convert to BedrockModel kwargs, translating canonical inference params. + + With ``agent_build_shared_session_enabled`` (default on) the model is + built on the process-wide boto3 session instead of the fresh + ``boto3.Session()`` Strands would otherwise construct (and re-parse + the bedrock-runtime model on). Nothing here reaches the prompt. + """ config: Dict[str, Any] = {"model_id": self.model_id} + + from apis.shared.feature_flags import agent_build_shared_session_enabled + + if agent_build_shared_session_enabled(): + from apis.shared.aws_clients import shared_boto_session + + # Never alongside `region_name`: BedrockModel.__init__ raises when + # both are given (strands-agents 1.55.0). This config sets no + # region either way; the session resolves it from the + # environment exactly as Strands' own fresh session would. + config["boto_session"] = shared_boto_session() _apply_canonical_params( config, self.inference_params, _BEDROCK_PARAM_MAP, "bedrock", self.model_id ) diff --git a/backend/src/agents/main_agent/integrations/external_mcp_client.py b/backend/src/agents/main_agent/integrations/external_mcp_client.py index bdce9ff21..91a8e7608 100644 --- a/backend/src/agents/main_agent/integrations/external_mcp_client.py +++ b/backend/src/agents/main_agent/integrations/external_mcp_client.py @@ -13,9 +13,11 @@ gates execution. """ +import asyncio import logging import re -from typing import Any, Callable, Iterator, Optional, List, Set +import time +from typing import Any, Callable, Iterator, Optional, List, Set, Tuple from urllib.parse import urlparse from mcp.client.streamable_http import streamablehttp_client @@ -39,6 +41,7 @@ CompositeAuth, create_oauth_bearer_auth, ) +from apis.shared.feature_flags import mcp_parallel_preflight_enabled from apis.shared.timestamps import to_iso logger = logging.getLogger(__name__) @@ -139,6 +142,11 @@ def detect_aws_service_from_url(url: str) -> Optional[str]: ) +def _elapsed_ms(started: float) -> int: + """Milliseconds since a ``perf_counter`` reading.""" + return int((time.perf_counter() - started) * 1000) + + def _iter_exception_chain(exc: BaseException) -> Iterator[BaseException]: """Yield `exc` and everything reachable through its cause/context chain and any ExceptionGroup members, each exactly once. @@ -391,6 +399,7 @@ async def _recover_oauth_preflight( user_id: Optional[str], provider_id: Optional[str], exc: Exception, + off_loop: bool = False, ) -> bool: """Second chance for an OAuth-gated tool whose pre-flight failed. @@ -463,7 +472,7 @@ async def _recover_oauth_preflight( if resolved["token"]: oauth_token_cache.set(user_id, provider_id, resolved["token"]) try: - await client.load_tools() + await self._preflight(client, off_loop=off_loop) except Exception as retry_exc: logger.warning( f"Skipping external MCP tool {tool_id}: pre-flight still failed " @@ -505,11 +514,238 @@ def approval_names_for_client(self, client: Any) -> set[str]: """Return the set of tool names on `client` flagged needs_approval.""" return self._approval_names_for_client_id.get(id(client), set()) + async def _preflight(self, client: MCPClient, *, off_loop: bool = False) -> None: + """Start ``client`` and list its tools, off the calling event loop. + + Strands' ``load_tools`` is a coroutine in name only: it calls the + synchronous ``start()``, which blocks on the server's ``initialize`` + handshake, and then a blocking ``tools/list``. Awaited inline it holds + the loop for the whole round trip, so concurrent pre-flights would + still run one after another. On a worker thread (with its own loop, + and the caller's contextvars) they overlap. The client's session lives + on its own background thread either way, so where ``start()`` is + called from does not matter once it returns. + + ``off_loop`` is set only when several servers load together; a lone + server gains nothing from the hop and keeps the inline call. + """ + if not off_loop: + await client.load_tools() + return + await asyncio.to_thread(asyncio.run, client.load_tools()) + + async def _load_server( + self, + tool_id: str, + allowed_tool_names: Optional[Set[str]], + repository: Any, + user_id: Optional[str], + auth_token: Optional[str], + off_loop: bool = False, + ) -> Tuple[Optional[MCPClient], dict]: + """Build, pre-flight and cache one external MCP server's client. + + Returns the client (None when the server is skipped or dropped) and + its ``timings`` entry. Touches only this server's cache key, so + several can run at once. + """ + started = time.perf_counter() + timing: dict = {"id": tool_id, "outcome": "skipped"} + try: + tool = await repository.get_tool(tool_id) + timing["catalogMs"] = _elapsed_ms(started) + if not tool: + return None, timing + + if tool.protocol != "mcp_external": + return None, timing + + if not tool.mcp_config: + logger.warning(f"Tool {tool_id} has protocol=mcp_external but no mcp_config") + return None, timing + + forward_auth = bool(getattr(tool, "forward_auth_token", False)) + requires_oauth = bool(tool.requires_oauth_provider) + exchange_audience = getattr(tool, "token_exchange_audience", None) + # An exchanged token is per-user just like the other two modes, + # so it must partition the client cache the same way — otherwise + # one user's Directory token would be reused for another. + requires_user_auth = forward_auth or requires_oauth or bool(exchange_audience) + + cache_key = self._get_cache_key(tool_id, user_id, requires_user_auth) + # A different selected-tool subset is a different client, so fold + # the filter into the cache key (the integration is shared across + # agents — two users may select different subsets of one server). + if allowed_tool_names is not None: + cache_key += "|allow:" + ",".join(sorted(allowed_tool_names)) + tool_version = ( + to_iso(tool.updated_at) if tool.updated_at else "" + ) + + if ( + cache_key in self.clients + and self._client_versions.get(cache_key) == tool_version + ): + timing["outcome"] = "cached" + return self.clients[cache_key], timing + + # Stale entry — admin edited this tool since the client + # was built. Drop it so the block below creates a fresh + # client with the current config. + if cache_key in self.clients: + stale = self.clients.pop(cache_key) + self._client_versions.pop(cache_key, None) + self._provider_for_client_id.pop(id(stale), None) + self._approval_names_for_client_id.pop(id(stale), None) + + static_token: Optional[str] = None + token_provider: Optional[Callable[[], Optional[str]]] = None + provider_id: Optional[str] = None + + if exchange_audience: + # RFC 8693: trade the user's Cognito token for one the + # downstream API already trusts. Checked before forward_auth + # because forwarding the raw Cognito token to such an API + # would send a credential it cannot validate — a silent + # 401 rather than an obvious misconfiguration. + if not auth_token: + logger.warning( + f"Tool {tool_id} needs a token exchange but no auth_token " + "was provided; skipping" + ) + return None, timing + if not user_id: + logger.warning( + f"Tool {tool_id} needs a token exchange but no user_id " + "was provided; skipping (tokens must be cached per user)" + ) + return None, timing + + exchanger = get_token_exchange_client() + if not exchanger.configured: + logger.warning( + f"Tool {tool_id} declares token_exchange_audience but the " + "runtime has no exchange configuration; skipping" + ) + return None, timing + + # Resolved lazily per request, like the OAuth path: exchanged + # tokens are short-lived (600s in dev) and a client may + # outlive one, so binding a value here would go stale + # mid-conversation. TokenExchangeClient handles caching. + async def _exchange( + t=auth_token, a=exchange_audience, u=user_id, tid=tool_id + ) -> Optional[str]: + try: + return await exchanger.exchange( + subject_token=t, audience=a, user_id=u + ) + except TokenExchangeError as exc: + # Fail closed: no token means the request goes + # unauthenticated and the downstream API refuses it, + # which is the correct outcome and is visible here. + logger.warning(f"Token exchange failed for {tid}: {exc}") + return None + + token_provider = _exchange + logger.info( + f"Using RFC 8693 token exchange for tool {tool_id} " + f"(audience={exchange_audience})" + ) + + elif forward_auth: + if not auth_token: + logger.warning( + f"Tool {tool_id} has forward_auth_token=true but no auth_token provided" + ) + else: + static_token = auth_token + logger.info(f"Using OIDC token forwarding for tool {tool_id}") + + elif requires_oauth: + if not user_id: + logger.warning( + f"Tool {tool_id} requires OAuth provider '{tool.requires_oauth_provider}' " + "but no user_id provided" + ) + return None, timing + + provider_id = tool.requires_oauth_provider + # Bind user_id and provider_id at closure time so the + # provider stays valid for the client's lifetime. + token_provider = ( + lambda u=user_id, p=provider_id: oauth_token_cache.get(u, p) + ) + + client = create_external_mcp_client( + config=tool.mcp_config, + tool_definition=tool, + oauth_token=static_token, + token_provider=token_provider, + allowed_tool_names=allowed_tool_names, + ) + + if client: + # What the context breakdown calls this server's tools. + # Display-only: never let it cost the server its tools. + try: + client.context_label = getattr(tool, "display_name", None) or tool_id + except Exception: # noqa: BLE001 + pass + # Pre-flight the MCP session so a single unreachable + # server (e.g. a connector that isn't running locally) + # drops out of the registry instead of failing the + # whole turn when Strands later calls load_tools(). + # On success this also primes the client's tool cache, + # so Strands' subsequent load_tools() is a no-op. + preflight_started = time.perf_counter() + try: + await self._preflight(client, off_loop=off_loop) + timing["outcome"] = "loaded" + except Exception as exc: + recovered = await self._recover_oauth_preflight( + tool_id=tool_id, + client=client, + user_id=user_id, + provider_id=provider_id, + exc=exc, + off_loop=off_loop, + ) + timing["outcome"] = "recovered" if recovered else "dropped" + if not recovered: + return None, timing + finally: + timing["preflightMs"] = _elapsed_ms(preflight_started) + + self.clients[cache_key] = client + self._client_versions[cache_key] = tool_version + if provider_id: + self._provider_for_client_id[id(client)] = provider_id + approval_names = tool.mcp_config.approval_required_names() + if approval_names: + self._approval_names_for_client_id[id(client)] = approval_names + auth_label = ( + " (with OIDC forwarding)" if forward_auth and static_token + else " (OAuth)" if provider_id + else "" + ) + logger.info(f"✅ Loaded external MCP tool: {tool_id}{auth_label}") + return client, timing + return None, timing + + except Exception as e: + timing["outcome"] = "error" + logger.error(f"Error loading external MCP tool {tool_id}: {e}") + return None, timing + finally: + timing["totalMs"] = _elapsed_ms(started) + async def load_external_tools( self, enabled_tool_ids: List[str], user_id: Optional[str] = None, auth_token: Optional[str] = None, + timings: Optional[List[dict]] = None, ) -> List[MCPClient]: """ Load external MCP clients for enabled tools. @@ -524,6 +760,11 @@ async def load_external_tools( enabled_tool_ids: List of enabled tool IDs user_id: User ID (required for OAuth-gated and OIDC-forwarded tools) auth_token: Raw OIDC token for forwarding + timings: When given, one entry per server is appended — + ``{id, outcome, catalogMs, preflightMs, totalMs}`` — so the + caller can say which server owned the load (turn-path P1a). + ``outcome`` is ``cached``, ``loaded``, ``recovered``, + ``dropped`` (pre-flight failed), ``skipped`` or ``error``. Returns: List of MCPClient instances to add to the agent's tools @@ -538,184 +779,33 @@ async def load_external_tools( # server (filter is None). name_filters = collect_tool_name_filters(enabled_tool_ids) - for tool_id, allowed_tool_names in name_filters.items(): - try: - tool = await repository.get_tool(tool_id) - if not tool: - continue - - if tool.protocol != "mcp_external": - continue - - if not tool.mcp_config: - logger.warning(f"Tool {tool_id} has protocol=mcp_external but no mcp_config") - continue - - forward_auth = bool(getattr(tool, "forward_auth_token", False)) - requires_oauth = bool(tool.requires_oauth_provider) - exchange_audience = getattr(tool, "token_exchange_audience", None) - # An exchanged token is per-user just like the other two modes, - # so it must partition the client cache the same way — otherwise - # one user's Directory token would be reused for another. - requires_user_auth = forward_auth or requires_oauth or bool(exchange_audience) - - cache_key = self._get_cache_key(tool_id, user_id, requires_user_auth) - # A different selected-tool subset is a different client, so fold - # the filter into the cache key (the integration is shared across - # agents — two users may select different subsets of one server). - if allowed_tool_names is not None: - cache_key += "|allow:" + ",".join(sorted(allowed_tool_names)) - tool_version = ( - to_iso(tool.updated_at) if tool.updated_at else "" - ) - - if ( - cache_key in self.clients - and self._client_versions.get(cache_key) == tool_version - ): - clients.append(self.clients[cache_key]) - continue - - # Stale entry — admin edited this tool since the client - # was built. Drop it so the block below creates a fresh - # client with the current config. - if cache_key in self.clients: - stale = self.clients.pop(cache_key) - self._client_versions.pop(cache_key, None) - self._provider_for_client_id.pop(id(stale), None) - self._approval_names_for_client_id.pop(id(stale), None) - - static_token: Optional[str] = None - token_provider: Optional[Callable[[], Optional[str]]] = None - provider_id: Optional[str] = None - - if exchange_audience: - # RFC 8693: trade the user's Cognito token for one the - # downstream API already trusts. Checked before forward_auth - # because forwarding the raw Cognito token to such an API - # would send a credential it cannot validate — a silent - # 401 rather than an obvious misconfiguration. - if not auth_token: - logger.warning( - f"Tool {tool_id} needs a token exchange but no auth_token " - "was provided; skipping" - ) - continue - if not user_id: - logger.warning( - f"Tool {tool_id} needs a token exchange but no user_id " - "was provided; skipping (tokens must be cached per user)" - ) - continue - - exchanger = get_token_exchange_client() - if not exchanger.configured: - logger.warning( - f"Tool {tool_id} declares token_exchange_audience but the " - "runtime has no exchange configuration; skipping" - ) - continue - - # Resolved lazily per request, like the OAuth path: exchanged - # tokens are short-lived (600s in dev) and a client may - # outlive one, so binding a value here would go stale - # mid-conversation. TokenExchangeClient handles caching. - async def _exchange( - t=auth_token, a=exchange_audience, u=user_id, tid=tool_id - ) -> Optional[str]: - try: - return await exchanger.exchange( - subject_token=t, audience=a, user_id=u - ) - except TokenExchangeError as exc: - # Fail closed: no token means the request goes - # unauthenticated and the downstream API refuses it, - # which is the correct outcome and is visible here. - logger.warning(f"Token exchange failed for {tid}: {exc}") - return None - - token_provider = _exchange - logger.info( - f"Using RFC 8693 token exchange for tool {tool_id} " - f"(audience={exchange_audience})" - ) - - elif forward_auth: - if not auth_token: - logger.warning( - f"Tool {tool_id} has forward_auth_token=true but no auth_token provided" - ) - else: - static_token = auth_token - logger.info(f"Using OIDC token forwarding for tool {tool_id}") - - elif requires_oauth: - if not user_id: - logger.warning( - f"Tool {tool_id} requires OAuth provider '{tool.requires_oauth_provider}' " - "but no user_id provided" - ) - continue - - provider_id = tool.requires_oauth_provider - # Bind user_id and provider_id at closure time so the - # provider stays valid for the client's lifetime. - token_provider = ( - lambda u=user_id, p=provider_id: oauth_token_cache.get(u, p) + servers = list(name_filters.items()) + if len(servers) > 1 and mcp_parallel_preflight_enabled(): + # Each server's pre-flight is a network handshake that blocks its + # thread (see `_preflight`), so N servers used to cost the SUM of + # their handshakes — 9.5s + 12s on a cold KB agent on dev + # (turn-path spec §5 P3). Run them together; `gather` returns in + # input order, so the client list — and with it the tool order in + # the cacheable `toolConfig` prefix — is the same as the serial loop. + results = await asyncio.gather( + *( + self._load_server( + tool_id, names, repository, user_id, auth_token, off_loop=True ) - - client = create_external_mcp_client( - config=tool.mcp_config, - tool_definition=tool, - oauth_token=static_token, - token_provider=token_provider, - allowed_tool_names=allowed_tool_names, + for tool_id, names in servers ) - - if client: - # What the context breakdown calls this server's tools. - # Display-only: never let it cost the server its tools. - try: - client.context_label = getattr(tool, "display_name", None) or tool_id - except Exception: # noqa: BLE001 - pass - # Pre-flight the MCP session so a single unreachable - # server (e.g. a connector that isn't running locally) - # drops out of the registry instead of failing the - # whole turn when Strands later calls load_tools(). - # On success this also primes the client's tool cache, - # so Strands' subsequent load_tools() is a no-op. - try: - await client.load_tools() - except Exception as exc: - recovered = await self._recover_oauth_preflight( - tool_id=tool_id, - client=client, - user_id=user_id, - provider_id=provider_id, - exc=exc, - ) - if not recovered: - continue - - self.clients[cache_key] = client - self._client_versions[cache_key] = tool_version - if provider_id: - self._provider_for_client_id[id(client)] = provider_id - approval_names = tool.mcp_config.approval_required_names() - if approval_names: - self._approval_names_for_client_id[id(client)] = approval_names - clients.append(client) - auth_label = ( - " (with OIDC forwarding)" if forward_auth and static_token - else " (OAuth)" if provider_id - else "" - ) - logger.info(f"✅ Loaded external MCP tool: {tool_id}{auth_label}") - - except Exception as e: - logger.error(f"Error loading external MCP tool {tool_id}: {e}") - continue + ) + else: + results = [ + await self._load_server(tool_id, names, repository, user_id, auth_token) + for tool_id, names in servers + ] + + for client, timing in results: + if timings is not None: + timings.append(timing) + if client is not None: + clients.append(client) return clients diff --git a/backend/src/agents/main_agent/session/compaction_summary.py b/backend/src/agents/main_agent/session/compaction_summary.py index 23f668a88..12a8f6e01 100644 --- a/backend/src/agents/main_agent/session/compaction_summary.py +++ b/backend/src/agents/main_agent/session/compaction_summary.py @@ -58,7 +58,9 @@ import logging import os from dataclasses import dataclass -from typing import List, Optional, Sequence, Tuple +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from apis.shared.aws_clients import get_client from .compaction_policy import CHARS_PER_TOKEN @@ -84,6 +86,12 @@ # cap (half the budget) is 4k, so this is what binds; it is the figure the # harness screen ran with. _EXTRACTION_MAX_OUTPUT_TOKENS = 3_000 +# A generation is used only on a positive completion signal. ``max_tokens`` +# is the one exception, because it is salvaged (see the module docstring). +# Anything else (a guardrail stop, a content-filter refusal, an unknown +# reason) could be the refusal text itself, and a summary persists into the +# cacheable history until the next cut, so it falls back instead. +_ACCEPTED_STOP_REASONS = frozenset({"end_turn", "max_tokens"}) _COMPRESSION_SYSTEM_PROMPT = """You maintain the running summary of a long conversation between a user and an AI assistant. You are given the existing summary notes (oldest first). Rewrite them into ONE compact summary the assistant can continue the conversation from. @@ -206,6 +214,17 @@ def _salvage(text: str, budget_tokens: int) -> Optional[str]: return _keep_head_lines(head, budget_tokens) if head else None +def _converse(region: Optional[str], **kwargs: Any) -> Dict[str, Any]: + """Run in a worker thread, so a first-use client build stays off the event loop. + + The client is process-cached: one per call paid botocore's service-model + load (~250ms the first time in a process) and, every time, a fresh + connection pool — a new TCP+TLS handshake per compaction call. + """ + region = region or os.environ.get("AWS_REGION", "us-west-2") + return get_client("bedrock-runtime", region).converse(**kwargs) + + async def _compress( records: Sequence[str], budget_tokens: int, @@ -218,17 +237,12 @@ async def _compress( if not text.strip(): return None, False try: - import boto3 - except ImportError: # pragma: no cover - dev without boto3 - return None, False - try: - region = region or os.environ.get("AWS_REGION", "us-west-2") - client = boto3.client("bedrock-runtime", region_name=region) # ~0.75 words/token; aim well under the budget so the chars/4 check # below passes with margin. word_budget = max(150, int(budget_tokens * 0.55)) response = await asyncio.to_thread( - client.converse, + _converse, + region, modelId=model_id, system=[{"text": _COMPRESSION_SYSTEM_PROMPT.replace("{word_budget}", f"{word_budget:,}")}], messages=[{"role": "user", "content": [{"text": "Summary notes, oldest first:\n\n" + text}]}], @@ -240,8 +254,14 @@ async def _compress( "maxTokens": min(_max_output_tokens(model_id), max(256, int(budget_tokens))), }, ) + stop_reason = response.get("stopReason") + if stop_reason not in _ACCEPTED_STOP_REASONS: + logger.info( + "compaction_summary_model_refused: stopReason=%s; falling back to truncation", stop_reason, + ) + return None, False out = response["output"]["message"]["content"][0]["text"].strip() - if response.get("stopReason") == "max_tokens": + if stop_reason == "max_tokens": salvaged = _salvage(out, budget_tokens) logger.info( "compaction_summary_model_truncated: generation hit the token ceiling; kept %d of %d chars", @@ -292,21 +312,22 @@ async def extract_with_model( if not text.strip(): return None try: - import boto3 - except ImportError: # pragma: no cover - dev without boto3 - return None - try: - region = region or os.environ.get("AWS_REGION", "us-west-2") - client = boto3.client("bedrock-runtime", region_name=region) response = await asyncio.to_thread( - client.converse, + _converse, + region, modelId=model_id, system=[{"text": _EXTRACTION_SYSTEM_PROMPT}], messages=[{"role": "user", "content": [{"text": "Summary notes, oldest first:\n\n" + text}]}], inferenceConfig={"temperature": 0.0, "maxTokens": max(256, int(max_tokens))}, ) + stop_reason = response.get("stopReason") + if stop_reason not in _ACCEPTED_STOP_REASONS: + logger.info( + "compaction_summary_extract_refused: stopReason=%s; falling back to plain compression", stop_reason, + ) + return None out = response["output"]["message"]["content"][0]["text"].strip() - if response.get("stopReason") == "max_tokens": + if stop_reason == "max_tokens": logger.info("compaction_summary_extract_truncated: keeping the complete lines") cut = out.rfind("\n") out = out[:cut].rstrip() if cut > 0 else "" diff --git a/backend/src/agents/main_agent/session/hooks/agent_status.py b/backend/src/agents/main_agent/session/hooks/agent_status.py index 73824625f..9ddd8ca13 100644 --- a/backend/src/agents/main_agent/session/hooks/agent_status.py +++ b/backend/src/agents/main_agent/session/hooks/agent_status.py @@ -59,6 +59,7 @@ import json import logging +import time from typing import Any, Dict, List, Optional from strands.hooks import ( @@ -169,6 +170,11 @@ def __init__(self) -> None: self._cycle = 0 # Monotonic batch counter, used to build a stable batch id. self._batch_seq = 0 + # `perf_counter()` at this turn's FIRST model call — the boundary the + # coordinator's first-token clock needs between pre-model work (the + # user message's append, LTM retrieval) and the model's own time to + # first token. Read by the coordinator within the same turn. + self.first_model_call_at: Optional[float] = None def register_hooks(self, registry: HookRegistry, **kwargs: Any) -> None: registry.add_callback(BeforeInvocationEvent, self._on_turn_start) @@ -204,8 +210,13 @@ def _on_turn_start(self, event: BeforeInvocationEvent) -> None: self._open_batch = [] self._cycle = 0 self._batch_seq = 0 + self.first_model_call_at = None def _on_before_model_call(self, event: BeforeModelCallEvent) -> None: + # A measurement, not narration: stamped whether or not the status + # line is on. One `perf_counter()` per model call. + if self.first_model_call_at is None: + self.first_model_call_at = time.perf_counter() if not self._enabled(): return try: diff --git a/backend/src/agents/main_agent/session/hooks/display_text.py b/backend/src/agents/main_agent/session/hooks/display_text.py index 268aeb905..fe95d0dad 100644 --- a/backend/src/agents/main_agent/session/hooks/display_text.py +++ b/backend/src/agents/main_agent/session/hooks/display_text.py @@ -45,7 +45,7 @@ from __future__ import annotations import logging -from typing import Any, Optional +from typing import Any, Awaitable, Callable, Optional, Union from strands.hooks import HookProvider, HookRegistry, MessageAddedEvent @@ -82,7 +82,7 @@ def arm( *, session_id: str, user_id: str, - message_index: int, + message_index: Union[int, Callable[[], Awaitable[int]]], display_text: Optional[str], ) -> None: """Prime the hook for one turn, or clear it when there's nothing to write. @@ -91,6 +91,11 @@ def arm( prompt was modified before reaching the model. A turn that sends the user's text verbatim (and a resume / continuation, which sends no new user turn at all) passes ``None`` and disarms. + + ``message_index`` may be an async callable — the stream coordinator + passes its ``HistoryCount.resolve`` so the history read behind the + index runs concurrently with the turn and is awaited only here, at + the write. """ self._written = False if not display_text: @@ -130,18 +135,21 @@ async def write_display_text(self, event: MessageAddedEvent) -> None: # the same turn can never re-enter this. self._armed = None + message_id: Any = armed["message_id"] try: from apis.shared.sessions.metadata import store_user_display_text - await store_user_display_text(**armed) + if callable(message_id): + message_id = await message_id() + await store_user_display_text(**{**armed, "message_id": message_id}) self._written = True logger.info( "💾 Stored displayText for user message %s at append time", - armed["message_id"], + message_id, ) except Exception: # noqa: BLE001 - a UI nicety must never break a turn logger.error( "Failed to store displayText for user message %s", - armed["message_id"], + message_id, exc_info=True, ) diff --git a/backend/src/agents/main_agent/session/session_factory.py b/backend/src/agents/main_agent/session/session_factory.py index 6bb12856e..8ef60f265 100644 --- a/backend/src/agents/main_agent/session/session_factory.py +++ b/backend/src/agents/main_agent/session/session_factory.py @@ -1,6 +1,7 @@ """ Session manager factory for creating AgentCore Memory session managers """ +import contextvars import os import logging from typing import Optional, Any, Dict, Tuple @@ -68,24 +69,83 @@ def session_async_persistence_enabled() -> bool: try: from bedrock_agentcore.memory.integrations.strands.config import AgentCoreMemoryConfig, RetrievalConfig from bedrock_agentcore.memory import MemoryClient + from bedrock_agentcore.memory.integrations.strands import session_manager as _sdk_session_manager AGENTCORE_MEMORY_AVAILABLE = True except ImportError: AGENTCORE_MEMORY_AVAILABLE = False -@lru_cache(maxsize=1) -def _discover_strategy_ids(memory_id: str, region: str) -> Tuple[Optional[str], Optional[str], Optional[str]]: +# --------------------------------------------------------------------------- +# One set of AgentCore Memory clients per process +# --------------------------------------------------------------------------- +# +# The SDK's session manager builds its clients from scratch on every +# construction, twice (see ``agent_build_shared_session_enabled``). Everything +# below exists to hand it the process-wide session from +# ``apis.shared.aws_clients`` instead, which warm-up builds at container +# start (``apis/inference_api/warmup.py``). + +# Set by the factory for the duration of one session manager's construction. +# The SDK builds its ``MemoryClient`` with no session and no session id, so this +# is how the per-session decision reaches ``_SharedSessionMemoryClient``. +_use_shared_memory_clients: contextvars.ContextVar[bool] = contextvars.ContextVar( + "use_shared_memory_clients", default=False +) + + +if AGENTCORE_MEMORY_AVAILABLE: + from apis.shared.aws_clients import shared_boto_session + + class _SharedSessionMemoryClient(MemoryClient): + """``MemoryClient`` built from the shared session when the flag is on. + + The SDK session manager constructs ``MemoryClient(region_name=...)`` + with no session, so it cannot be handed one; its clients are then + replaced a few lines later, so they were pure cost. Rebinding the name + in the SDK module is the only seam the pinned bedrock-agentcore offers. + Pinned by ``test_session_factory_shared_clients.py``, which fails if an + SDK upgrade stops going through it. + """ + + def __init__( + self, + region_name: Optional[str] = None, + integration_source: Optional[str] = None, + boto3_session: Any = None, + ) -> None: + if boto3_session is None and _use_shared_memory_clients.get(): + boto3_session = shared_boto_session() + super().__init__( + region_name=region_name, + integration_source=integration_source, + boto3_session=boto3_session, + ) + + _sdk_session_manager.MemoryClient = _SharedSessionMemoryClient + + +@lru_cache(maxsize=2) +def _discover_strategy_ids( + memory_id: str, region: str, *, shared_session: bool = False +) -> Tuple[Optional[str], Optional[str], Optional[str]]: """ Discover the actual strategy IDs from the configured memory strategies. AgentCore Memory stores memories in strategy-specific namespaces: /strategies/{strategyId}/actors/{actorId} - This function queries the memory to find the actual strategy IDs. + This function queries the memory to find the actual strategy IDs. It is + a control-plane read of static configuration, cached for the life of the + process. Args: memory_id: AgentCore Memory ID region: AWS region + shared_session: Build the ``MemoryClient`` on the process-wide session + (``agent_build_shared_session_enabled``) instead of a fresh one. + Part of the cache key on purpose: warm-up primes the shared entry + at container start, and a process with the kill switch set must + still do exactly what it did before. Returns: Tuple of (semantic_strategy_id, preference_strategy_id, summary_strategy_id) @@ -94,7 +154,10 @@ def _discover_strategy_ids(memory_id: str, region: str) -> Tuple[Optional[str], return None, None, None try: - client = MemoryClient(region_name=region) + client = MemoryClient( + region_name=region, + boto3_session=shared_boto_session() if shared_session else None, + ) strategies = client.get_memory_strategies(memory_id=memory_id) semantic_id = None @@ -122,6 +185,19 @@ def _discover_strategy_ids(memory_id: str, region: str) -> Tuple[Optional[str], return None, None, None +def warm_strategy_ids() -> Tuple[Optional[str], Optional[str], Optional[str]]: + """Discover the memory's strategy ids on the shared session, once, at container start. + + Called from ``apis/inference_api/warmup.py`` on the startup daemon thread + so the first turn finds the ids cached and its clients built. Raises when no memory is configured (``load_memory_config``); the + warm-up step logs that and moves on. + """ + if not AGENTCORE_MEMORY_AVAILABLE: + return None, None, None + config = load_memory_config() + return _discover_strategy_ids(config.memory_id, config.region, shared_session=True) + + class SessionFactory: """Factory for creating appropriate session manager based on environment""" @@ -205,8 +281,15 @@ def _create_cloud_session_manager( logger.info(f" • Memory ID: {memory_id}") logger.info(f" • Region: {aws_region}") - # Discover actual strategy IDs from the memory configuration - semantic_id, preference_id, summary_id = _discover_strategy_ids(memory_id, aws_region) + # Discover actual strategy IDs from the memory configuration. With the + # shared session on this is a cache hit: warm-up discovered them at + # container start (`warm_strategy_ids`). + from apis.shared.feature_flags import agent_build_shared_session_enabled + + shared_clients = agent_build_shared_session_enabled() + semantic_id, preference_id, summary_id = _discover_strategy_ids( + memory_id, aws_region, shared_session=shared_clients + ) # Load retrieval thresholds from environment (configurable per deployment) relevance_score = float(os.environ.get(EnvVars.MEMORY_RELEVANCE_SCORE, str(Defaults.MEMORY_RELEVANCE_SCORE))) @@ -282,13 +365,18 @@ def _create_cloud_session_manager( compaction_config.token_threshold = compaction_threshold # Create session manager with compaction built-in - session_manager = TurnBasedSessionManager( - agentcore_memory_config=agentcore_memory_config, - region_name=aws_region, - compaction_config=compaction_config if compaction_config.enabled else None, - user_id=user_id, - summarization_strategy_id=summary_id, - ) + token = _use_shared_memory_clients.set(shared_clients) + try: + session_manager = TurnBasedSessionManager( + agentcore_memory_config=agentcore_memory_config, + region_name=aws_region, + compaction_config=compaction_config if compaction_config.enabled else None, + user_id=user_id, + summarization_strategy_id=summary_id, + boto_session=shared_boto_session() if shared_clients else None, + ) + finally: + _use_shared_memory_clients.reset(token) logger.info("✅ AgentCore Memory initialized") logger.info(" • Storage: AWS-managed DynamoDB") @@ -299,6 +387,7 @@ def _create_cloud_session_manager( else: logger.info(" • Compaction: Disabled") logger.info(" • Persistence: %s", "Async (off the event loop)" if async_persistence else "Sync (blocking)") + logger.info(" • Clients: %s", "Shared (process-wide)" if shared_clients else "Per session manager") return session_manager diff --git a/backend/src/agents/main_agent/session/turn_based_session_manager.py b/backend/src/agents/main_agent/session/turn_based_session_manager.py index 450437542..aa040ef63 100644 --- a/backend/src/agents/main_agent/session/turn_based_session_manager.py +++ b/backend/src/agents/main_agent/session/turn_based_session_manager.py @@ -28,6 +28,7 @@ Based on: https://github.com/aws-samples/sample-strands-agent-with-agentcore """ +import asyncio import copy import json import logging @@ -38,6 +39,7 @@ from typing import Optional, Dict, Any, List, Tuple, TYPE_CHECKING from agents.main_agent.config.constants import Defaults, EnvVars +from apis.shared.observability.build_stages import mark_stage from bedrock_agentcore.memory.integrations.strands.session_manager import AgentCoreMemorySessionManager from bedrock_agentcore.memory.integrations.strands.config import AgentCoreMemoryConfig @@ -47,6 +49,8 @@ from .compaction_summary import bound_summary if TYPE_CHECKING: + from concurrent.futures import ThreadPoolExecutor + from strands.agent.agent import Agent logger = logging.getLogger(__name__) @@ -67,6 +71,26 @@ _RETRIEVAL_CLIENT_LOCK = threading.Lock() +_LTM_PREFETCH_POOL: Optional["ThreadPoolExecutor"] = None +_LTM_PREFETCH_POOL_LOCK = threading.Lock() + + +def _ltm_prefetch_pool() -> "ThreadPoolExecutor": + """The process-wide pool the long-term-memory prefetch runs on. + + Small on purpose: one lookup per user message, and every conversation runs + in its own Runtime process. Created on first use, never per turn. + """ + global _LTM_PREFETCH_POOL + if _LTM_PREFETCH_POOL is None: + with _LTM_PREFETCH_POOL_LOCK: + if _LTM_PREFETCH_POOL is None: + from concurrent.futures import ThreadPoolExecutor + + _LTM_PREFETCH_POOL = ThreadPoolExecutor(max_workers=4, thread_name_prefix="ltm-prefetch") + return _LTM_PREFETCH_POOL + + def memory_retrieval_timeout_seconds() -> float: """Per-attempt timeout for long-term memory retrieval, from the environment.""" raw = os.environ.get(MEMORY_RETRIEVAL_TIMEOUT_ENV, "") @@ -130,6 +154,22 @@ def _approx_tokens(text: Any) -> int: return 0 +def _is_bidi_agent(agent: Any) -> bool: + """True for the voice path's ``BidiAgent``. + + Since strands-agents 1.56 a ``BidiAgent`` drives the *ordinary* session + hooks (``AgentInitializedEvent`` -> ``initialize``, ``MessageAddedEvent`` + -> ``append_message`` / ``retrieve_customer_context``) instead of the + dedicated Bidi callbacks it used before, so every override here now sees + voice agents too. The text-only work in them must not run for voice. + """ + try: + from strands.bidi import BidiAgent + except ImportError: + return False + return isinstance(agent, BidiAgent) + + class TurnBasedSessionManager(AgentCoreMemorySessionManager): """ Session manager with token-based context compaction. @@ -311,6 +351,112 @@ def _replace_retrieval_client(self, stale: Any) -> Any: self._retrieval_client = None return self._get_retrieval_client() + def register_hooks(self, registry: Any, **kwargs: Any) -> None: + """The SDK's wiring, plus a long-term-memory prefetch ahead of it. + + Registered FIRST on purpose: Strands runs a hook's callbacks one at a + time in registration order, so ``_prefetch_customer_context`` runs + before the SDK's persist callback and the lookup's round trip overlaps + the two writes the SDK awaits (docs/specs/turn-path-ttft.md P4a). + + Also persists voice transcripts on ``MessageUpdatedEvent`` — see + ``_persist_bidi_message_update``. + """ + from strands.hooks import MessageAddedEvent, MessageUpdatedEvent + + registry.add_callback(MessageAddedEvent, self._prefetch_customer_context) + super().register_hooks(registry, **kwargs) + registry.add_callback(MessageUpdatedEvent, self._persist_bidi_message_update) + + async def _persist_bidi_message_update(self, event: Any) -> None: + """Persist a voice transcript once the BidiAgent fills it in. + + Since strands-agents 1.57 a ``BidiAgent`` appends each transcript to + history EMPTY when it starts (``MessageAddedEvent``) and replaces it + with the text when it stops — or with the partial text if the response + is cut off — through ``MessageUpdatedEvent``, exactly once per message. + Through 1.55 the message was appended only when final, already + carrying its text. Nothing in the SDK's session wiring (or + bedrock-agentcore's) listens for the update, and ``append_message`` + skips the empty shell, so without this every spoken turn and every + assistant reply was dropped: voice worked live and saved only typed + input. + + Only a ``BidiAgent`` fires this event today; the guard keeps it that + way if text agents start to. The write is offloaded so the boto3 call + never blocks the voice event loop. + """ + if not _is_bidi_agent(event.agent): + return + await asyncio.to_thread(self.append_message, event.message, event.agent) + + def _retrieval_query_for(self, messages: Any) -> Optional[str]: + """The lookup's query for the last message, or None if it gets none. + + One eligibility rule for the prefetch and the inline path: a user + message whose first block is text, with retrieval configured. + """ + if not messages or messages[-1].get("role") != "user": + return None + content = messages[-1].get("content") + if not content or "text" not in content[0]: + return None + if not getattr(self.config, "retrieval_config", None): + return None + return memory_retrieval_query(messages[-1]["content"][0]["text"]) + + def _prefetch_customer_context(self, event: Any) -> None: + """Start the lookup for the message just added, and return at once. + + Reads only the query string; never touches the message. The result is + applied later, by ``retrieve_customer_context``, at the same point in + the callback sequence as without the prefetch — after the SDK has + persisted the message — so persisted bytes cannot change. + """ + self._ltm_prefetch = None + try: + from apis.shared.feature_flags import memory_retrieval_prefetch_enabled + + if not memory_retrieval_prefetch_enabled() or self.cancelled: + return + # retrieve_customer_context never consumes a lookup for the voice + # BidiAgent, so starting one would pay for a retrieval per + # transcript and throw it away. + if _is_bidi_agent(event.agent): + return + messages = event.agent.messages + query = self._retrieval_query_for(messages) + if query is None: + return + future = _ltm_prefetch_pool().submit(self._fetch_customer_context, query) + self._ltm_prefetch = (messages[-1], future, time.perf_counter()) + except Exception: # noqa: BLE001 - the inline path still runs + logger.debug("Memory retrieval prefetch skipped", exc_info=True) + + def _take_prefetched_context(self, message: Any) -> Optional[List[str]]: + """The prefetched items for ``message``, or None to fetch inline. + + Matched by object identity: a prefetch started for any other message + (a turn that died between the two callbacks) is discarded, never + applied to this one. + """ + pending, self._ltm_prefetch = getattr(self, "_ltm_prefetch", None), None + if pending is None or pending[0] is not message: + return None + _, future, started = pending + waited_from = time.perf_counter() + try: + items = future.result(timeout=2 * memory_retrieval_timeout_seconds() + 1) + except Exception as e: # noqa: BLE001 - a miss, not a failed turn + logger.warning("memory retrieval prefetch failed: %s: %s", type(e).__name__, e) + items = [] + logger.info( + "memory retrieval prefetch waitedMs=%d lookupMs=%d", + (time.perf_counter() - waited_from) * 1000, + (time.perf_counter() - started) * 1000, + ) + return items + def retrieve_customer_context(self, event: Any) -> None: """Retrieve long-term memory for the last user message and prepend it. @@ -321,19 +467,37 @@ def retrieve_customer_context(self, event: Any) -> None: differs, see :meth:`_get_retrieval_client`. Registered by the SDK's ``register_hooks`` in both sync and async mode via ``self.retrieve_customer_context``, so the override is picked up. + + Uses the lookup ``_prefetch_customer_context`` started for this same + message when there is one, and fetches inline otherwise. """ - messages = event.agent.messages - if not messages or messages[-1].get("role") != "user": + # Mirrors bedrock-agentcore 1.23.1's own guard: voice transcripts are + # not queries, and LTM context spliced into a live Bidi history would + # be persisted into the voice agent's messages. + if _is_bidi_agent(event.agent): return None - content = messages[-1].get("content") - if not content or "text" not in content[0]: + messages = event.agent.messages + query = self._retrieval_query_for(messages) + if query is None: return None + items = self._take_prefetched_context(messages[-1]) + if items is None: + items = self._fetch_customer_context(query) + if items: + tag = getattr(self.config, "context_tag", "user_context") + messages[-1]["content"].insert(0, {"text": f"<{tag}>{chr(10).join(items)}"}) + logger.info("Retrieved %s customer context items", len(items)) + return None + + def _fetch_customer_context(self, user_query: str) -> List[str]: + """Every namespace's context items for ``user_query``. Never raises. + + The network half of the lookup, safe to run on any thread: it reads + the query and the config and returns text; it never touches a message. + """ retrieval_config = getattr(self.config, "retrieval_config", None) if not retrieval_config: - return None - - user_query = memory_retrieval_query(messages[-1]["content"][0]["text"]) - client = self._get_retrieval_client() + return [] def retrieve_for_namespace(namespace: str, cfg: Any) -> List[str]: from botocore.exceptions import ConnectionClosedError, SSLError @@ -381,10 +545,11 @@ def retrieve_for_namespace(namespace: str, cfg: Any) -> List[str]: items.append(text) return items + all_context: List[str] = [] try: from concurrent.futures import ThreadPoolExecutor, as_completed - all_context: List[str] = [] + client = self._get_retrieval_client() with ThreadPoolExecutor() as executor: futures = { executor.submit(retrieve_for_namespace, ns, cfg): ns @@ -408,16 +573,20 @@ def retrieve_for_namespace(namespace: str, cfg: Any) -> List[str]: logger.warning( "memory retrieval failed namespace=%s: %s: %s", futures[future], type(e).__name__, e ) - - if all_context: - tag = getattr(self.config, "context_tag", "user_context") - event.agent.messages[-1]["content"].insert( - 0, {"text": f"<{tag}>{chr(10).join(all_context)}"} - ) - logger.info("Retrieved %s customer context items", len(all_context)) except Exception as e: # noqa: BLE001 - retrieval must never break a turn logger.error("Failed to retrieve customer context: %s", e) - return None + return all_context + + def read_session(self, session_id: str, **kwargs: Any) -> Any: + """The SDK's session read, preceded by a build-stage mark. + + The SDK constructor builds its boto3 clients and then calls this, so the + mark splits ``agent_build.session_mgr`` into client setup + (``session_mgr_clients``) and the network calls that follow. A no-op + outside a turn's build. + """ + mark_stage("session_mgr_clients") + return super().read_session(session_id, **kwargs) def initialize(self, agent: "Agent", **kwargs: Any) -> None: """ @@ -428,8 +597,22 @@ def initialize(self, agent: "Agent", **kwargs: Any) -> None: 2. Let the SDK restore agent state and load messages from AgentCore Memory 3. Apply compaction (checkpoint + truncation) on the loaded messages """ + # Splits `agent_build.finalize`: everything before this is Strands' + # own Agent construction (tool registration, including MCP + # `load_tools`); everything after is the session restore. + mark_stage("strands_agent") logger.info(f"TurnBasedSessionManager.initialize() called for agent_id={agent.agent_id}") + # A voice BidiAgent reaches here since strands-agents 1.56 (it used to + # take the SDK's separate initialize_bidi_agent path). Restore it the + # same way that path did and stop: everything below is text-chat + # history processing, and the compaction state it reads is keyed on the + # session, so applying the text agent's checkpoint to the voice + # agent's own message list would slice the wrong history. + if _is_bidi_agent(agent): + super().initialize(agent, **kwargs) + return + # Let the SDK handle all session restore logic: # - read/create agent in session repository # - restore agent state, internal state, conversation manager state diff --git a/backend/src/agents/main_agent/streaming/history_count.py b/backend/src/agents/main_agent/streaming/history_count.py new file mode 100644 index 000000000..db479da8f --- /dev/null +++ b/backend/src/agents/main_agent/streaming/history_count.py @@ -0,0 +1,170 @@ +"""This turn's message-index base, counted off the critical path. + +Per-message metadata (cost, usage, ``displayText``, the artifact anchor) is +keyed by a message's position in the session's full history — the same +``enumerate`` index ``get_messages_from_cloud`` re-derives on reload. So every +turn needs the number of messages that existed before it, counted across +every agent that writes to the session (an ``@``-mention's second agent, +voice, synthetic messages), which only a read of the history can give: a +count held on a session manager instance goes stale the moment another +instance writes (the #741/#751 rule in CLAUDE.md). + +That read is a paginated ``ListEvents`` with payloads, and it used to sit on +the head of every turn, ahead of the model call: ~62ms + 2.25ms per stored +event, ~390ms at 100+ events (docs/specs/turn-path-ttft.md §5 P4). Nothing +needs the number before the user's message is appended, and most consumers +need it only after the model answers. So the read starts at the head of the +turn on a worker thread and each consumer awaits it where it is used. + +**The cutoff.** Started concurrently, the read races this turn's own writes — +the user's message is persisted while it runs. Every persisted message carries +the ``created_at`` it was built with (the SDK derives the event timestamp from +it), and this turn's messages are all built after the head of the turn, so +counting only messages created before that instant gives the count the +serial read gave, whether or not this turn's writes have landed yet. +""" + +from __future__ import annotations + +import asyncio +import logging +import threading +import time +from concurrent.futures import Future, ThreadPoolExecutor +from datetime import datetime, timezone +from typing import Any, Callable, Iterable, Optional + +logger = logging.getLogger(__name__) + +_POOL: Optional[ThreadPoolExecutor] = None +_POOL_LOCK = threading.Lock() + + +def _pool() -> ThreadPoolExecutor: + """The process-wide pool the count runs on, built on first use.""" + global _POOL + if _POOL is None: + with _POOL_LOCK: + if _POOL is None: + _POOL = ThreadPoolExecutor(max_workers=4, thread_name_prefix="history-count") + return _POOL + + +def count_created_before(messages: Optional[Iterable[Any]], before: Optional[datetime]) -> int: + """How many of ``messages`` were created strictly before ``before``. + + ``before=None`` counts them all. A message whose ``created_at`` is missing + or unparseable is counted: anything already stored when the read ran and + not stamped by this turn belongs to the history before it. + """ + if not messages: + return 0 + if before is None: + return sum(1 for _ in messages) + count = 0 + for message in messages: + created = getattr(message, "created_at", None) + if not created: + count += 1 + continue + try: + stamp = datetime.fromisoformat(str(created).replace("Z", "+00:00")) + except ValueError: + count += 1 + continue + if stamp.tzinfo is None: + stamp = stamp.replace(tzinfo=timezone.utc) + if stamp < before: + count += 1 + return count + + +class HistoryCount: + """The number of messages in the session before this turn, resolved lazily. + + Build one per turn with :meth:`start` (or :meth:`resolved` for a count + that needs no read) and ``await resolve()`` wherever the number is used. + The first resolve logs ``history count prefetch waitedMs= lookupMs=``; + ``waitedMs=0`` means the read finished before anyone needed it. + """ + + def __init__( + self, + *, + value: Optional[int] = None, + future: Optional["Future[int]"] = None, + fallback: int = 0, + ) -> None: + self._value = value + self._future = future + self._fallback = fallback + self._lookup_ms: Optional[float] = None + + @classmethod + def resolved(cls, value: int) -> "HistoryCount": + return cls(value=value) + + @classmethod + def start( + cls, + count: Callable[[datetime], int], + *, + fallback: int, + now: Optional[datetime] = None, + ) -> "HistoryCount": + """Submit ``count(cutoff)`` to the pool, with the cutoff taken now. + + ``fallback`` is used if the read raises; capture it before this turn + appends anything (a maintained count read later would include this + turn's own messages). + """ + cutoff = now or datetime.now(timezone.utc) + instance = cls(fallback=fallback) + + def _run() -> int: + started = time.perf_counter() + try: + return count(cutoff) + finally: + instance._lookup_ms = (time.perf_counter() - started) * 1000 + + try: + instance._future = _pool().submit(_run) + except RuntimeError: + # The pool is shut down (interpreter exit). Count inline. + try: + instance._value = count(cutoff) + except Exception: # noqa: BLE001 - a count must not break a turn + instance._value = fallback + return instance + + @property + def value(self) -> Optional[int]: + """The count if already resolved, else None. Never waits.""" + return self._value + + async def resolve(self) -> int: + if self._value is not None: + return self._value + future = self._future + if future is None: + self._value = self._fallback + return self._value + waited_from = time.perf_counter() + try: + value = await asyncio.wrap_future(future) + except Exception as e: # noqa: BLE001 - a count must not break a turn + logger.warning( + "history count prefetch failed, using maintained count %d: %s: %s", + self._fallback, type(e).__name__, e, + ) + value = self._fallback + if self._value is None: + self._value = value + logger.info( + "history count prefetch waitedMs=%d lookupMs=%d count=%d", + round((time.perf_counter() - waited_from) * 1000), + round(self._lookup_ms or 0), + value, + ) + return self._value diff --git a/backend/src/agents/main_agent/streaming/stream_coordinator.py b/backend/src/agents/main_agent/streaming/stream_coordinator.py index 06d594ab2..527d44bed 100644 --- a/backend/src/agents/main_agent/streaming/stream_coordinator.py +++ b/backend/src/agents/main_agent/streaming/stream_coordinator.py @@ -9,7 +9,7 @@ import os import time from datetime import datetime, timezone -from typing import Any, AsyncGenerator, Dict, List, Optional, Union +from typing import Any, AsyncGenerator, Callable, Dict, List, Optional, Union from agents.main_agent.config.constants import EnvVars from agents.main_agent.session.hooks.prefix_fingerprint import ( @@ -24,6 +24,7 @@ from apis.shared.feature_flags import ( agent_status_live_drain_enabled, cost_diagnostics_enabled, + history_count_prefetch_enabled, ) from apis.shared.errors import ( ConversationalErrorEvent, @@ -32,8 +33,32 @@ build_conversational_error_event, ) +from .history_count import HistoryCount, count_created_before from .stream_processor import process_agent_stream +# Processed event types that mean the model has started answering: text, a +# tool call's first block or input, or reasoning. The first one of a turn +# closes the first-token clock (`turn_first_token`, docs/specs/turn-path-ttft.md +# P1b) — a tool-first turn's "first token" is its tool call, which is what the +# user sees appear. +_FIRST_MODEL_OUTPUT_TYPES = frozenset( + {"content_block_start", "content_block_delta", "reasoning", "tool_use"} +) + + +def _turn_clock_call(turn_clock: Any, method: str, *args: Any) -> None: + """Call a method on the turn's clock if one was passed. Never raises. + + Duck-typed: the clock is inference-api's ``TurnPrelude``, which this + package must not import (``tests/architecture/test_import_boundaries.py``). + """ + if turn_clock is None: + return + try: + getattr(turn_clock, method)(*args) + except Exception: # noqa: BLE001 - a measurement must not break a turn + logger.debug("Turn clock %s skipped", method, exc_info=True) + logger = logging.getLogger(__name__) # How long the status merge waits on the agent stream before looking at the @@ -256,6 +281,8 @@ async def stream_response( turn_project_id: Optional[str] = None, turn_lease: Any = None, turn_started_at: Optional[float] = None, + poll_side_frame: Optional[Callable[[], Optional[str]]] = None, + turn_clock: Any = None, ) -> AsyncGenerator[str, None]: """ Stream agent responses with proper lifecycle management @@ -286,10 +313,26 @@ async def stream_response( stamped *unconditionally*, including to None, for the same reason ``reset_cancellation_state`` exists: a lease left behind by a previous turn on a cached agent would be read against a row that no longer names us. + poll_side_frame: Non-blocking check for one formatted SSE frame produced + outside the agent stream (the first turn's ``session_title``). Polled + on every pass of the live status merge, so the frame goes out within + one poll interval of being ready instead of waiting for the agent + stream's next event. Must be idempotent: the caller keeps checking it + between events too, which is its only route while the merge is off. + turn_clock: The turn's ``TurnPrelude`` (duck-typed), continued here + past the agent build: ``head_of_turn.*`` sub-stages, then + ``pre_model`` (closed at the status hook's first + ``BeforeModelCallEvent`` stamp) and ``model`` at the first model + output, then ``emit_first_token`` once that output has been + yielded. Per turn, like ``turn_lease``; None disables it. Yields: str: SSE formatted events """ + # Everything since the agent was ready: the route's generator wiring, + # the `prepared` frame, quota warnings and the prompt build. + _turn_clock_call(turn_clock, "mark", "head_of_turn.handoff") + # Set environment variables for browser session isolation os.environ[EnvVars.SESSION_ID] = session_id os.environ[EnvVars.USER_ID] = user_id @@ -341,6 +384,8 @@ async def stream_response( # interrupt state makes Strands reject this turn's prompt outright. # See ``reset_stale_interrupt_state``. reset_stale_interrupt_state(agent, prompt) + # Compaction state re-read (#751), document offload, the resets above. + _turn_clock_call(turn_clock, "mark", "head_of_turn.compaction") # Track timing for latency metrics stream_start_time = time.time() @@ -409,12 +454,13 @@ async def stream_response( # See the CancelledError/GeneratorExit handler below. assistant_text_acc: List[str] = [] - # OPTIMIZATION: Capture initial message count BEFORE streaming starts - # This allows us to calculate message indices without post-stream AgentCore Memory queries - # The TurnBasedSessionManager.message_count is initialized from AgentCore Memory at session start - # and represents the number of messages that existed BEFORE this stream - initial_message_count = self._get_initial_message_count(session_manager) - logger.info(f"📊 Initial message count before streaming: {initial_message_count}") + # The number of messages stored before this turn: the base every + # per-message index below is computed from. Started here, read on a + # worker, and awaited only where an index is used, so the ListEvents + # behind it (~2ms per stored event, turn-path spec F5) no longer sits + # in front of the model call. See `history_count.py`. + history_count = self._start_history_count(session_manager) + _turn_clock_call(turn_clock, "mark", "head_of_turn.history_count") # Arm the displayText write for this turn. The hook stores the user's # original message on `MessageAddedEvent` — i.e. before the model @@ -428,7 +474,7 @@ async def stream_response( main_agent_wrapper, session_id=session_id, user_id=user_id, - message_index=initial_message_count, + message_index=history_count.resolve, display_text=original_message, ) @@ -456,6 +502,13 @@ async def stream_response( logger.warning("MCP Apps broker subscribe failed: %s", e) app_event_queue = None + # First-token clock state: the first model output is marked on + # arrival, and the line is emitted on the NEXT pass (or in `finally`), + # after that output has been yielded — never in front of it. + first_model_output_seen = False + first_token_emit_pending = False + + _turn_clock_call(turn_clock, "mark", "head_of_turn.rest") try: # Get raw agent stream agent_stream = agent.stream_async(prompt) @@ -472,10 +525,15 @@ async def stream_response( ) if agent_status_live_drain_enabled(): processed_stream = self._merge_agent_status( - processed_stream, main_agent_wrapper, session_id + processed_stream, main_agent_wrapper, session_id, + poll_side_frame=poll_side_frame, ) async for event in processed_stream: + if first_token_emit_pending: + first_token_emit_pending = False + _turn_clock_call(turn_clock, "emit_first_token") + # A status transition the merge picked up mid-silence. It is # already a formatted SSE frame and describes nothing the rest # of this body reasons about (no message index, no metadata, no @@ -484,6 +542,15 @@ async def stream_response( yield event.sse continue + if ( + turn_clock is not None + and not first_model_output_seen + and event.get("type") in _FIRST_MODEL_OUTPUT_TYPES + ): + first_model_output_seen = True + self._mark_first_model_output(turn_clock, main_agent_wrapper) + first_token_emit_pending = True + # Cooperative stop. A user Stop arms a cancel on the session's # single-flight lease; the inference-api heartbeat observes it # and flips ``session_manager.cancelled``. Because a client @@ -952,6 +1019,7 @@ async def stream_response( # post-loop block uses for per-message metadata # (assistant_message_ids[-1]), which the messages # endpoint re-derives as `idx` on reload. + initial_message_count = await history_count.resolve() produced_by_message_index = ( initial_message_count + 2 * current_assistant_message_index @@ -1283,6 +1351,7 @@ async def stream_response( # # This eliminates the need for post-stream AgentCore Memory queries! num_assistant_messages = current_assistant_message_index + 1 if current_assistant_message_index >= 0 else 0 + initial_message_count = await history_count.resolve() # Calculate assistant message absolute indices using the turn structure pattern # Assistant messages are at odd positions: initial_count + 1, initial_count + 3, ... @@ -1521,7 +1590,7 @@ async def stream_response( partial_text="".join(assistant_text_acc), main_agent_wrapper=main_agent_wrapper, accumulated_metadata=accumulated_metadata, - initial_message_count=initial_message_count, + history_count=history_count, current_assistant_message_index=current_assistant_message_index, stream_start_time=stream_start_time, first_token_time=first_token_time, @@ -1557,7 +1626,7 @@ async def stream_response( partial_text="".join(assistant_text_acc), main_agent_wrapper=main_agent_wrapper, accumulated_metadata=accumulated_metadata, - initial_message_count=initial_message_count, + history_count=history_count, current_assistant_message_index=current_assistant_message_index, stream_start_time=stream_start_time, first_token_time=first_token_time, @@ -1614,6 +1683,11 @@ async def stream_response( except Exception as persist_error: logger.error(f"Failed to persist stream error to session: {persist_error}") finally: + # The first model output was the stream's last event (or the turn + # ended right after it): emit now rather than never. + if first_token_emit_pending: + _turn_clock_call(turn_clock, "emit_first_token") + # MCP Apps PR #5: always release the broker subscription — # covers normal completion, the in-loop error `return`, and # the except path, so a dropped stream never leaks a queue. @@ -1664,6 +1738,7 @@ async def _persist_interruption( accumulated_metadata: Optional[Dict[str, Any]] = None, initial_message_count: int = 0, current_assistant_message_index: int = -1, + history_count: Optional[HistoryCount] = None, stream_start_time: Optional[float] = None, first_token_time: Optional[float] = None, reason: str = "connection_lost", @@ -1704,8 +1779,15 @@ async def _persist_interruption( Whenever nothing is persisted, only the marker is set. The write itself also passes ``last_persisted_role`` to ``persist_synthetic_messages`` so the centralized alternation guard is the single enforcement point. + + ``history_count``, when passed, supersedes ``initial_message_count`` + and is resolved inside the shielded task, so the teardown cannot + cancel the wait for it. """ async def _do() -> None: + base_index = initial_message_count + if history_count is not None: + base_index = await history_count.resolve() text = partial_text.strip() last_role = None try: @@ -1783,9 +1865,9 @@ async def _do() -> None: if projected: metadata_for_message = {**metadata_for_message, "usage": projected} message_id = ( - initial_message_count + 2 * current_assistant_message_index + 1 + base_index + 2 * current_assistant_message_index + 1 if current_assistant_message_index >= 0 - else initial_message_count + 1 + else base_index + 1 ) await self._store_message_metadata( session_id=session_id, @@ -2686,6 +2768,7 @@ async def _merge_agent_status( events: AsyncGenerator[Dict[str, Any], None], main_agent_wrapper: Any, session_id: str, + poll_side_frame: Optional[Callable[[], Optional[str]]] = None, ) -> AsyncGenerator[Any, None]: """Yield the agent's events, interleaved with status transitions as they happen. @@ -2758,6 +2841,9 @@ async def _merge_agent_status( main_agent_wrapper, session_id ): yield _StatusFrame(sse) + side_frame = self._poll_side_frame(poll_side_frame) + if side_frame: + yield _StatusFrame(side_frame) if not done: continue @@ -2789,6 +2875,40 @@ async def _merge_agent_status( exc_info=True, ) + @staticmethod + def _poll_side_frame( + poll_side_frame: Optional[Callable[[], Optional[str]]], + ) -> Optional[str]: + """Best-effort: a failing poll must never break the stream it rides on.""" + if poll_side_frame is None: + return None + try: + return poll_side_frame() + except Exception: # noqa: BLE001 - side-channel frames are never load-bearing + logger.warning("Side-frame poll failed", exc_info=True) + return None + + @staticmethod + def _mark_first_model_output(turn_clock: Any, main_agent_wrapper: Any) -> None: + """Close ``pre_model`` and ``model`` on the turn's clock. + + ``pre_model`` ends at the status hook's ``perf_counter`` stamp of this + turn's first ``BeforeModelCallEvent``: the user message's append, LTM + retrieval and the ``BeforeInvocation`` hooks sit before it, and + ``ContextAttributionHook`` (registered ahead of the status hook). The + hooks registered after it (census, ledger, fingerprint) and the + model's own time to first token are ``model``. Without a stamp — a + wrapper with no status hook — the two cannot be told apart and the + whole gap is reported as ``pre_model_and_model``. + """ + hook = getattr(main_agent_wrapper, "agent_status_hook", None) + stamp = getattr(hook, "first_model_call_at", None) + if stamp is None: + _turn_clock_call(turn_clock, "mark", "pre_model_and_model") + return + _turn_clock_call(turn_clock, "mark_at", "pre_model", stamp) + _turn_clock_call(turn_clock, "mark", "model") + def _drain_agent_status_events( self, main_agent_wrapper: Any, session_id: str ) -> List[str]: @@ -3115,21 +3235,53 @@ def _get_initial_message_count(self, session_manager: Any) -> int: Returns: int: Number of messages that existed before this stream started (0 if unknown) """ + try: + count = self._count_stored_messages(session_manager, before=None) + except Exception as e: + logger.warning(f"Failed to get global message count: {e}") + count = None + if count is not None: + logger.info(f"Using global list_messages count: {count}") + return count + return self._maintained_message_count(session_manager) + + def _start_history_count(self, session_manager: Any) -> HistoryCount: + """This turn's message-index base, read off the critical path. + + Starts the same global ``list_messages`` count on a worker thread, + restricted to messages created before now, and returns a handle each + consumer awaits where it uses the index (see ``history_count.py``). + With ``HISTORY_COUNT_PREFETCH_ENABLED=false`` the count is read here, + inline, as before. + """ + if not history_count_prefetch_enabled(): + return HistoryCount.resolved(self._get_initial_message_count(session_manager)) + # Captured before this turn appends anything: read on the worker it + # could already include this turn's user message. + fallback = self._maintained_message_count(session_manager) + if self._resolve_session_id(session_manager) is None or self._resolve_list_messages(session_manager) is None: + return HistoryCount.resolved(fallback) + + def _count(before: datetime) -> int: + count = self._count_stored_messages(session_manager, before=before) + return fallback if count is None else count + + return HistoryCount.start(_count, fallback=fallback) + + def _count_stored_messages(self, session_manager: Any, *, before: Optional[datetime]) -> Optional[int]: + """The session's stored messages across every agent, created before + ``before`` (all of them when None). None if the manager can't list.""" # Prefer list_messages() for global count — it returns ALL messages # regardless of agent_id, matching how get_messages_from_cloud() retrieves them. session_id = self._resolve_session_id(session_manager) - if session_id: - lister = self._resolve_list_messages(session_manager) - if lister: - try: - messages = lister(session_id, "default") - count = len(messages) if messages else 0 - logger.info(f"Using global list_messages count: {count}") - return count - except Exception as e: - logger.warning(f"Failed to get global message count: {e}") + lister = self._resolve_list_messages(session_manager) if session_id else None + if not lister: + return None + return count_created_before(lister(session_id, "default"), before) - # Fallback to agent-specific message_count (may undercount in mixed sessions) + @staticmethod + def _maintained_message_count(session_manager: Any) -> int: + """The session manager's own running count (may undercount in mixed sessions).""" if hasattr(session_manager, "message_count"): count = session_manager.message_count logger.debug(f"Fallback to TurnBasedSessionManager.message_count: {count}") diff --git a/backend/src/agents/main_agent/voice_agent.py b/backend/src/agents/main_agent/voice_agent.py index ca44b0aba..83de7223a 100644 --- a/backend/src/agents/main_agent/voice_agent.py +++ b/backend/src/agents/main_agent/voice_agent.py @@ -12,14 +12,16 @@ """ import asyncio +import base64 import logging import os import sys import types -from typing import Any, AsyncGenerator, List, Optional +from typing import Any, AsyncGenerator, Dict, List, Optional from agents.main_agent.base_agent import BaseAgent from agents.main_agent.config.constants import EnvVars, Defaults +from apis.shared.voice_catalog import NovaSonicVoice, get_voice, resolve_voice_id logger = logging.getLogger(__name__) @@ -32,12 +34,85 @@ # not crash, it would silently turn voice off everywhere with one INFO line. # Import the name explicitly rather than leaning on the package's lazy # ``__getattr__``, so a future rename fails loudly here too. +# +# 1.57.2 graduated the API from ``strands.experimental.bidi`` to +# ``strands.bidi``. The experimental path survives as a shim that resolves to +# the same module objects (so ``isinstance`` holds across both) but warns on +# import; import the stable path. try: - from strands.experimental.bidi import BidiAgent - from strands.experimental.bidi.models.bedrock import BedrockNovaSonicModel + from strands.bidi import BidiAgent + from strands.bidi.models.bedrock import BedrockNovaSonicModel + from strands.bidi.types.events import BidiUsageEvent, ModalityUsage BIDI_AVAILABLE = True except ImportError: BIDI_AVAILABLE = False + + +def usage_modality_details(usage_event: dict) -> list: + """Split a Nova Sonic ``usageEvent`` into Strands ``ModalityUsage`` rows. + + Nova reports four buckets under ``details.total`` — speech and text, in and + out — and bills them on two rate cards ten-fold apart. Strands 1.55's + provider keeps only the grand totals, which is what made every voice + session price as if it were all speech (or, with no catalog row, $0). + + Returns ``[]`` when the event carries no ``details`` so the caller can + tell "no split" from "a split of zeros". The rows are plain dicts (the + ``ModalityUsage`` TypedDict shape) so this stays importable without the + bidi extra installed. + """ + total = (usage_event.get("details") or {}).get("total") or {} + if not total: + return [] + inp = total.get("input") or {} + out = total.get("output") or {} + return [ + { + "modality": "audio", + "input_tokens": int(inp.get("speechTokens") or 0), + "output_tokens": int(out.get("speechTokens") or 0), + }, + { + "modality": "text", + "input_tokens": int(inp.get("textTokens") or 0), + "output_tokens": int(out.get("textTokens") or 0), + }, + ] + + +if BIDI_AVAILABLE: + + class NovaSonicModelWithUsageDetails(BedrockNovaSonicModel): + """``BedrockNovaSonicModel`` that keeps the speech/text usage split. + + Only ``usageEvent`` is intercepted; every other event goes to the + provider untouched. The split rides the ``modality_details`` slot the + SDK already reserves on ``BidiUsageEvent`` (and serialises through + ``as_dict``), so nothing downstream has to learn a new event shape. + + 1.57 changed this hook's contract: it now takes the provider's + per-response state and returns a *list* of events. The provider calls + it positionally, so an override with the old one-argument signature + raises on every Nova event — which is why the signature is asserted + against the provider's in the tests rather than trusted. + """ + + def _convert_nova_event(self, nova_event: dict, response_state: Any) -> list: + usage = nova_event.get("usageEvent") + if not usage: + return super()._convert_nova_event(nova_event, response_state) + details = usage_modality_details(usage) + total_input = usage.get("totalInputTokens", 0) + total_output = usage.get("totalOutputTokens", 0) + return [ + BidiUsageEvent( + input_tokens=total_input, + output_tokens=total_output, + total_tokens=usage.get("totalTokens", total_input + total_output), + modality_details=[ModalityUsage(**row) for row in details] or None, + ) + ] +else: logger.info("BidiAgent not available — install strands-agents[bidi] for voice support") @@ -48,6 +123,139 @@ sys.modules["pyaudio"] = _fake_pyaudio +class VoiceWireAdapter: + """Translate Strands >=1.57 Bidi output events into our voice WebSocket contract. + + strands-agents 1.57 renamed and re-shaped every Bidi output event (1.57.0 + #4444 "adopt production component names", 1.57.1 #4604 "remove response + stop reasons"). The SPA (``voice-chat.service.ts``) and our own turn and + usage accounting still speak the 1.55 wire names, so the translation lives + here, at the one seam we own, rather than in two packages at once. + + What changed, and how each is mapped: + + * ``bidi_audio_delta`` -> ``bidi_audio_stream`` (same fields). + * ``bidi_transcript_delta`` -> ``bidi_transcript_stream``. Nova used to + stream an assistant transcript twice (SPECULATIVE, then FINAL) and the + SPA commits text only when ``is_final`` is set. 1.57 streams it once, + from the SPECULATIVE stage, and drops FINAL entirely, so every delta is + now the only pass and is forwarded with ``is_final=True``. + * ``bidi_response_start`` now fires at the *user's* first content, before + any user speech has been transcribed. The SPA flushes the user's + transcript into a message when it sees a response start, so forwarding + that event as-is would file each user message one turn late. The legacy + ``bidi_response_start`` is therefore emitted when the *assistant* begins + (its transcript or its audio), which is when 1.55 emitted it too. + * ``bidi_barge_in`` -> ``bidi_interruption``. + * ``bidi_response_stop`` -> ``bidi_response_complete``; ``stop_reason`` is + no longer on the event, so it is derived from whether a barge-in was + seen in this response. + * ``bidi_connection_stop`` -> ``bidi_connection_close``. + * Start/stop bracketing events with no 1.55 equivalent + (``bidi_transcript_start``/``_stop``, ``bidi_audio_start``/``_stop``) + are dropped. So are the completed-block events 1.57.2 added + (``bidi_transcript_block`` and its text/reasoning siblings): each repeats + text the deltas already carried. Everything else passes through unchanged. + """ + + _DROPPED = frozenset( + { + "bidi_transcript_start", + "bidi_transcript_stop", + "bidi_audio_start", + "bidi_audio_stop", + "bidi_transcript_block", + "bidi_text_block", + "bidi_reasoning_block", + } + ) + + def __init__(self) -> None: + self._response_id: Optional[str] = None + self._assistant_started = False + self._interrupted = False + + def translate(self, event: Dict[str, Any]) -> List[Dict[str, Any]]: + """Return the wire events for one Strands Bidi event (zero or more).""" + event_type = event.get("type", "") + + if event_type in self._DROPPED: + if event_type == "bidi_transcript_start" and event.get("role") == "assistant": + return self._start_assistant() + if event_type == "bidi_audio_start": + return self._start_assistant() + return [] + + if event_type == "bidi_response_start": + self._response_id = event.get("response_id") + self._assistant_started = False + self._interrupted = False + return [] + + if event_type == "bidi_transcript_delta": + delta = event.get("delta", "") + return [ + { + "type": "bidi_transcript_stream", + "role": event.get("role"), + "delta": {"text": delta}, + "text": delta, + "is_final": True, + "current_transcript": None, + } + ] + + if event_type == "bidi_audio_delta": + return [ + { + "type": "bidi_audio_stream", + "audio": event.get("audio"), + "format": event.get("format"), + "sample_rate": event.get("sample_rate"), + "channels": event.get("channels"), + } + ] + + if event_type == "bidi_barge_in": + self._interrupted = True + return [{"type": "bidi_interruption", "reason": event.get("reason")}] + + if event_type == "bidi_response_stop": + # A response that ended before the assistant produced anything + # (tool-only, or barged in immediately) still owes the SPA a start, + # or the user's words stay buffered and the SPA re-files the + # previous turn's text as this turn's transcript entry. + events = self._start_assistant() + events.append( + { + "type": "bidi_response_complete", + "response_id": event.get("response_id") or self._response_id, + "stop_reason": "interrupted" if self._interrupted else "complete", + } + ) + self._response_id = None + self._assistant_started = False + self._interrupted = False + return events + + if event_type == "bidi_connection_stop": + return [ + { + "type": "bidi_connection_close", + "connection_id": event.get("connection_id"), + "reason": event.get("reason"), + } + ] + + return [event] + + def _start_assistant(self) -> List[Dict[str, Any]]: + if self._assistant_started: + return [] + self._assistant_started = True + return [{"type": "bidi_response_start", "response_id": self._response_id}] + + class VoiceAgent(BaseAgent): """ Bidirectional voice agent using AWS Nova Sonic 2. @@ -62,8 +270,8 @@ class VoiceAgent(BaseAgent): agent = VoiceAgent(session_id="sess-123", enabled_tools=[...]) await agent.start() await agent.send_audio(audio_base64, sample_rate=16000) - async for event in agent.stream_async(""): - # BidiOutputEvent, BidiAudioStreamEvent, etc. + async for event in agent.receive_events(): + # wire-contract dicts: bidi_audio_stream, bidi_transcript_stream, etc. """ def __init__(self, voice: Optional[str] = None, **kwargs): @@ -71,12 +279,14 @@ def __init__(self, voice: Optional[str] = None, **kwargs): Initialize voice agent. Args: - voice: Voice name override ("matthew", "tiffany", "amy"). - Defaults to NOVA_SONIC_VOICE env var or "tiffany". + voice: Nova 2 Sonic voice id (see ``apis.shared.voice_catalog``). + Unknown or blank falls back to the NOVA_SONIC_VOICE env var, + then "tiffany" — never an id Bedrock would refuse. **kwargs: All BaseAgent constructor args """ - self._voice = voice or os.environ.get( - EnvVars.NOVA_SONIC_VOICE, Defaults.NOVA_SONIC_VOICE + self._voice = resolve_voice_id( + voice, + os.environ.get(EnvVars.NOVA_SONIC_VOICE, Defaults.NOVA_SONIC_VOICE), ) self._bidi_agent: Any = None # Nova Sonic bidi_usage events report CUMULATIVE token counts (not deltas). @@ -85,6 +295,7 @@ def __init__(self, voice: Optional[str] = None, **kwargs): self._per_turn_usage: List[dict] = [] # Snapshot of usage per completed turn self._turn_count: int = 0 # Completed turns (bidi_response_complete) self._response_start_count: int = 0 # Started turns (bidi_response_start) + self._wire = VoiceWireAdapter() super().__init__(**kwargs) def _create_agent(self) -> None: @@ -103,19 +314,16 @@ def _create_agent(self) -> None: EnvVars.NOVA_SONIC_MODEL_ID, Defaults.NOVA_SONIC_MODEL_ID ) - # 1.55.0 flattened the provider's constructor: the audio settings - # moved from provider_config["audio"] to the `audio` kwarg (an - # AudioConfig TypedDict with these same five keys), and the region - # moved from client_config["region"] to `region`. Both are - # keyword-only now. - model = BedrockNovaSonicModel( + # 1.57 split the provider's audio config: the voice is its own + # `voice` kwarg, and `audio` carries per-direction sample rates + # only (channels and format are fixed at mono PCM). Unknown keys + # are rejected, so the 1.55 five-key dict no longer fits. + model = NovaSonicModelWithUsageDetails( model_id=model_id, + voice=self._voice, audio={ - "voice": self._voice, - "input_rate": Defaults.NOVA_SONIC_INPUT_RATE, - "output_rate": Defaults.NOVA_SONIC_OUTPUT_RATE, - "channels": 1, - "format": "pcm", + "input": {"sample_rate": Defaults.NOVA_SONIC_INPUT_RATE}, + "output": {"sample_rate": Defaults.NOVA_SONIC_OUTPUT_RATE}, }, region=os.environ.get(EnvVars.AWS_REGION, Defaults.AWS_REGION), ) @@ -126,13 +334,20 @@ def _create_agent(self) -> None: # Load text history for voice-text continuity initial_messages = self._load_text_history() - # Create BidiAgent with separate agent_id + # Create BidiAgent with separate agent_id. + # + # 1.57.2 removed BidiAgent's `session_manager` kwarg (#4698). All + # it did was register the manager as a hook provider, and BidiAgent + # still fires the events the manager listens on + # (AgentInitializedEvent, MessageAddedEvent, BidiAgentStopEvent), + # so passing it as a hook keeps voice transcripts persisting. + # Dropping it would not error — voice would just stop saving. self._bidi_agent = BidiAgent( model=model, tools=tools, system_prompt=voice_prompt, agent_id=Defaults.VOICE_AGENT_ID, - session_manager=self.session_manager, + hooks=[self.session_manager] if self.session_manager else None, messages=initial_messages, ) @@ -161,7 +376,26 @@ def _build_voice_system_prompt(self) -> str: "- Use natural speech patterns\n" "- Confirm understanding before taking actions\n" ) - return base + voice_addendum + return base + voice_addendum + self._voice_presentation_line() + + def _voice_presentation_line(self) -> str: + """One line telling the model which voice it speaks with. + + Amazon's Nova 2 Sonic prompt guidance: languages that conjugate the + speaker's own gender (Hindi, Portuguese, French, Italian, Spanish) need + the system prompt to say which form matches the voice, or "I am tired" + comes out as the wrong one. English never conjugates it, so the line is + inert for the default voice and costs a few tokens — but it is part of + the cached prefix, so it is written once per voice, not per turn. + """ + voice: Optional[NovaSonicVoice] = get_voice(getattr(self, "_voice", None)) + if voice is None: + return "" + form = "feminine" if voice.gender == "feminine" else "masculine" + return ( + f"- You speak with the voice \"{voice.name}\", which sounds {form}; " + f"in languages that mark the speaker's gender, use the {form} form for yourself\n" + ) def _load_text_history(self) -> list: """ @@ -213,12 +447,17 @@ async def send_audio(self, audio_base64: str, sample_rate: int = 16000) -> None: if not self._bidi_agent: raise RuntimeError("Voice agent not started") + # 1.57 takes raw bytes in an `audio_delta` block; the sample rate is + # fixed by the model's audio config rather than sent per chunk. + # The client's value is not logged: it arrives unvalidated off the + # WebSocket, so it could carry a forged log line. + if sample_rate != Defaults.NOVA_SONIC_INPUT_RATE: + logger.debug( + "Voice audio sample rate differs from the configured input rate %s; using the configured rate", + Defaults.NOVA_SONIC_INPUT_RATE, + ) await self._bidi_agent.send({ - "type": "bidi_audio_input", - "audio": audio_base64, - "format": "pcm", - "sample_rate": sample_rate, - "channels": 1, + "audio_delta": {"format": "pcm", "source": {"bytes": base64.b64decode(audio_base64)}}, }) async def send_text(self, text: str) -> None: @@ -231,11 +470,7 @@ async def send_text(self, text: str) -> None: if not self._bidi_agent: raise RuntimeError("Voice agent not started") - await self._bidi_agent.send({ - "type": "bidi_text_input", - "text": text, - "role": "user", - }) + await self._bidi_agent.send({"text": text}) @property def accumulated_usage(self) -> dict: @@ -262,6 +497,11 @@ def voice_model_id(self) -> str: """Nova Sonic model ID used by this agent.""" return os.environ.get(EnvVars.NOVA_SONIC_MODEL_ID, Defaults.NOVA_SONIC_MODEL_ID) + @property + def voice_id(self) -> str: + """The Nova 2 Sonic voice this agent speaks with (always a catalog id).""" + return self._voice + async def receive_events(self) -> AsyncGenerator[dict, None]: """ Receive and transform events from BidiAgent for WebSocket transmission. @@ -280,73 +520,91 @@ async def receive_events(self) -> AsyncGenerator[dict, None]: if not self._bidi_agent: raise RuntimeError("Voice agent not started") - # Lazy-loaded pricing for real-time cost calculation - pricing_dict: Optional[dict] = None - pricing_loaded = False + # Lazy-loaded pricing for real-time cost calculation, fetched once + # and cached for the session lifetime. + pricing_state: dict = {"loaded": False, "pricing": None} async for event in self._bidi_agent.receive(): if hasattr(event, "as_dict"): - event_dict = event.as_dict() + raw_dict = event.as_dict() + else: + raw_dict = {"type": "unknown", "data": str(event)} + + for event_dict in self._wire.translate(raw_dict): + yield await self._account_and_enrich(event_dict, pricing_state) + + async def _account_and_enrich(self, event_dict: dict, pricing_state: dict) -> dict: + """Apply turn and usage accounting to one wire event and return it.""" + event_type = event_dict.get("type", "") + + # Log non-audio event types for debugging (skip audio to avoid noise) + if event_type not in ("bidi_audio_stream",): + if any(k in event_dict for k in ("usage", "inputTokens", "outputTokens", "totalTokens")): + logger.info(f"Voice event: type={event_type}, keys={list(event_dict.keys())}") else: - event_dict = {"type": "unknown", "data": str(event)} - - event_type = event_dict.get("type", "") - - # Log non-audio event types for debugging (skip audio to avoid noise) - if event_type not in ("bidi_audio_stream",): - if any(k in event_dict for k in ("usage", "inputTokens", "outputTokens", "totalTokens")): - logger.info(f"Voice event: type={event_type}, keys={list(event_dict.keys())}") - else: - logger.info(f"Voice event: type={event_type}") - - # Track started turns (bidi_response_start) - if event_type == "bidi_response_start": - self._response_start_count += 1 - logger.info(f"Voice response started (count={self._response_start_count})") - - # Update cumulative token usage from bidi_usage events. - # Nova Sonic reports CUMULATIVE totals in each event (not deltas), - # so we replace rather than sum. - if event_type == "bidi_usage": - usage = event_dict.get("usage", event_dict) - for key in ("inputTokens", "outputTokens", "totalTokens"): - self._accumulated_usage[key] = usage.get(key, self._accumulated_usage[key]) - logger.info(f"Voice bidi_usage snapshot: {usage}") - logger.info(f"Voice usage current: {self._accumulated_usage}") - - # Calculate real-time cost and enrich the event for the client. - # Pricing is fetched once and cached for the session lifetime. - if not pricing_loaded: - pricing_loaded = True - try: - from apis.shared.costs.pricing_config import get_model_pricing - pricing_dict = await get_model_pricing(self.voice_model_id) - except Exception as e: - logger.debug(f"Voice pricing unavailable: {e}") - - if pricing_dict and self._accumulated_usage.get("totalTokens", 0) > 0: - try: - from apis.shared.costs.calculator import CostCalculator - total_cost, breakdown = CostCalculator.calculate_message_cost( - self._accumulated_usage, pricing_dict - ) - event_dict["cost"] = { - "total": total_cost, - "inputCost": breakdown.input_cost, - "outputCost": breakdown.output_cost, - "cacheReadCost": breakdown.cache_read_cost, - "cacheWriteCost": breakdown.cache_write_cost, - } - except Exception as e: - logger.debug(f"Voice cost calculation error: {e}") - - # Track completed assistant turns and snapshot cumulative usage at turn boundary - if event_type == "bidi_response_complete": - self._per_turn_usage.append(self._accumulated_usage.copy()) - self._turn_count += 1 - logger.info(f"Voice turn {self._turn_count} complete, cumulative usage: {self._accumulated_usage}") - - yield event_dict + logger.info(f"Voice event: type={event_type}") + + # Track started turns (bidi_response_start) + if event_type == "bidi_response_start": + self._response_start_count += 1 + logger.info(f"Voice response started (count={self._response_start_count})") + + if event_type == "bidi_usage": + usage = event_dict.get("usage", event_dict) + self._apply_usage_snapshot(usage) + logger.info(f"Voice bidi_usage snapshot: {usage}") + logger.info(f"Voice usage current: {self._accumulated_usage}") + + # Calculate real-time cost and enrich the event for the client. + if not pricing_state["loaded"]: + pricing_state["loaded"] = True + try: + from apis.shared.costs.pricing_config import get_model_pricing + pricing_state["pricing"] = await get_model_pricing(self.voice_model_id) + except Exception as e: + logger.debug(f"Voice pricing unavailable: {e}") + + pricing_dict = pricing_state["pricing"] + if pricing_dict and self._accumulated_usage.get("totalTokens", 0) > 0: + try: + from apis.shared.costs.calculator import CostCalculator + total_cost, breakdown = CostCalculator.calculate_voice_cost( + self._accumulated_usage, pricing_dict + ) + event_dict["cost"] = { + "total": total_cost, + "inputCost": breakdown.input_cost, + "outputCost": breakdown.output_cost, + "cacheReadCost": breakdown.cache_read_cost, + "cacheWriteCost": breakdown.cache_write_cost, + } + except Exception as e: + logger.debug(f"Voice cost calculation error: {e}") + + # Track completed assistant turns and snapshot cumulative usage at turn boundary + if event_type == "bidi_response_complete": + self._per_turn_usage.append(self._accumulated_usage.copy()) + self._turn_count += 1 + logger.info(f"Voice turn {self._turn_count} complete, cumulative usage: {self._accumulated_usage}") + + return event_dict + + def _apply_usage_snapshot(self, usage: dict) -> None: + """Fold one ``bidi_usage`` payload into ``_accumulated_usage``. + + Nova Sonic reports CUMULATIVE totals in each event (not deltas), so we + replace rather than sum — the speech/text split too, when the provider + kept it (see NovaSonicModelWithUsageDetails). Shared by the live stream + and the post-stop drain so the two can never price a session differently. + """ + for key in ("inputTokens", "outputTokens", "totalTokens"): + self._accumulated_usage[key] = usage.get(key, self._accumulated_usage[key]) + for row in usage.get("modality_details") or []: + bucket = {"audio": "speech", "text": "text"}.get(row.get("modality")) + if not bucket: + continue + self._accumulated_usage[f"{bucket}InputTokens"] = int(row.get("input_tokens") or 0) + self._accumulated_usage[f"{bucket}OutputTokens"] = int(row.get("output_tokens") or 0) async def stream_async( self, @@ -386,20 +644,18 @@ async def drain_remaining_events(self, timeout: float = 3.0) -> None: try: async with asyncio.timeout(timeout): async for event in self._bidi_agent.receive(): - if hasattr(event, "as_dict"): - event_dict = event.as_dict() - else: + if not hasattr(event, "as_dict"): continue - event_type = event_dict.get("type", "") - if event_type == "bidi_usage": - usage = event_dict.get("usage", event_dict) - for key in ("inputTokens", "outputTokens", "totalTokens"): - self._accumulated_usage[key] = usage.get(key, self._accumulated_usage[key]) - logger.info(f"Drained bidi_usage: {usage}, current: {self._accumulated_usage}") - elif event_type == "bidi_response_complete": - self._per_turn_usage.append(self._accumulated_usage.copy()) - self._turn_count += 1 - logger.info(f"Drained turn {self._turn_count} complete") + for event_dict in self._wire.translate(event.as_dict()): + event_type = event_dict.get("type", "") + if event_type == "bidi_usage": + usage = event_dict.get("usage", event_dict) + self._apply_usage_snapshot(usage) + logger.info(f"Drained bidi_usage: {usage}, current: {self._accumulated_usage}") + elif event_type == "bidi_response_complete": + self._per_turn_usage.append(self._accumulated_usage.copy()) + self._turn_count += 1 + logger.info(f"Drained turn {self._turn_count} complete") except (TimeoutError, asyncio.CancelledError, StopAsyncIteration): pass except Exception as e: diff --git a/backend/src/apis/app_api/documents/services/document_service.py b/backend/src/apis/app_api/documents/services/document_service.py index 7f6113f09..f04cc2bad 100644 --- a/backend/src/apis/app_api/documents/services/document_service.py +++ b/backend/src/apis/app_api/documents/services/document_service.py @@ -98,8 +98,10 @@ async def release_reservation_if_managed(document: Document) -> None: released by whichever terminal path the document reaches first — a client-reported upload failure, this stale sweep, or the ingestion consumer's own failure path — and the guard stops two of them double-crediting the - allowance. A no-op for legacy knowledge bases (uncapped) and for zero-size - rows (nothing was reserved). + allowance. A no-op for legacy knowledge bases (uncapped) and for rows that + reserved nothing: zero-size rows, and imported, crawled or synced documents, + whose ``sizeBytes`` is real but was never reserved + (``byte_cap.reserved_at_request``). ``app_kb_id == assistant_id`` this phase. boto3 is called synchronously here, matching the rest of this module. @@ -107,7 +109,7 @@ async def release_reservation_if_managed(document: Document) -> None: from apis.shared.kb_backend import byte_cap from apis.shared.kb_backend.records import ENGINE_MANAGED, get_kb_record, resolve_engine - size_bytes = int(document.size_bytes or 0) + size_bytes = byte_cap.reserved_at_request(document.size_bytes, document.source_adapter_key) if size_bytes <= 0: return assistant_id = document.assistant_id @@ -151,10 +153,25 @@ async def settle_bytes_on_delete(document: Document, previous_status: Optional[s Refund runs first. A row can hold both markers only because a consumer settled it, and then it has nothing left to release. + + Independently, a document deleted while a KB sync's changed source was being + re-ingested holds the growth reserved for that version + (``byte_cap.claim_reingest``). The consumer's completion is refused on a + ``deleting`` row, so this is the only path left to return it. """ from apis.shared.kb_backend import byte_cap assistant_id = document.assistant_id + try: + reingest_reserved = byte_cap.release_reingest_once(assistant_id, document.document_id) + if reingest_reserved: + byte_cap.release(assistant_id, assistant_id, reingest_reserved) + except Exception as e: # noqa: BLE001 - a bookkeeping failure must not break the delete + logger.error( + f"Failed to release re-ingest reservation for document {document.document_id}: {e}", + exc_info=True, + ) + try: refunded = byte_cap.refund_once(assistant_id, document.document_id) if refunded: diff --git a/backend/src/apis/app_api/kb_migration/ingestion_consumer.py b/backend/src/apis/app_api/kb_migration/ingestion_consumer.py index 384a21268..650888432 100644 --- a/backend/src/apis/app_api/kb_migration/ingestion_consumer.py +++ b/backend/src/apis/app_api/kb_migration/ingestion_consumer.py @@ -35,6 +35,17 @@ and ``retrievableAt`` separately so the gap stays measurable instead of becoming folklore. +A changed source is not a redelivery +------------------------------------ +A KB sync (Drive file or web re-crawl) that finds its source changed overwrites the +document's S3 object in place, so its ``ObjectCreated`` event arrives for a document +that is already ``complete`` with its bytes settled — exactly what a redelivery looks +like. The sync marks the difference: it writes ``stagedContentHash`` before the +overwrite, and :func:`staged_version_to_reingest` compares that with the +``ingestedContentHash`` the last re-ingest recorded. A version still owed is +re-ingested and its size difference settled against the byte cap +(:func:`_reingest_changed_document`); anything else keeps the settled early exit. + Import boundary --------------- Raw DynamoDB table access rather than importing ``apis.shared.assistants``, whose @@ -413,6 +424,7 @@ def wait_until_indexed( timeout_seconds: Optional[float] = None, interval_seconds: Optional[float] = None, sleep: Any = time.sleep, + not_before: Optional[str] = None, ) -> Tuple[str, Optional[str]]: """Poll Bedrock's document status until it settles, or give up. @@ -420,6 +432,10 @@ def wait_until_indexed( outcome, not an error: the caller leaves the document non-terminal and lets redelivery come back to it, by which time indexing has usually finished. + ``not_before`` is for a document Bedrock already holds a version of: a status + last updated at or before it describes that previous version, not the one just + submitted, and is waited out like an in-flight one (:func:`_is_stale`). + Why a *bounded* in-invocation wait rather than pure redelivery: most documents index in a few seconds, and making every one of them wait for an EventBridge retry would add a minute of latency to the common case for the sake of the rare @@ -438,12 +454,30 @@ def wait_until_indexed( deadline = time.monotonic() + timeout_seconds status, updated_at = document_status(backend, kb_ref, document_id) - while _still_working(status, document_id) and time.monotonic() < deadline: + while ( + _still_working(status, document_id) or _is_stale(updated_at, not_before) + ) and time.monotonic() < deadline: sleep(interval_seconds) status, updated_at = document_status(backend, kb_ref, document_id) return status, updated_at +def _is_stale(updated_at: Optional[str], not_before: Optional[str]) -> bool: + """Whether a Bedrock status predates ``not_before`` and so is not about it. + + A status with no timestamp cannot be placed and is taken at face value — the + same answer the consumer gave before this check existed. + """ + if not updated_at or not not_before: + return False + from apis.shared.timestamps import from_iso + + try: + return from_iso(updated_at) <= from_iso(not_before) + except (TypeError, ValueError): + return False + + def _still_working(status: str, document_id: str) -> bool: """Whether to keep waiting on ``status``. @@ -581,18 +615,18 @@ def _deleted_reason(doc_row: Optional[Dict[str, Any]]) -> Optional[str]: def _declared_bytes(doc_row: Optional[Dict[str, Any]]) -> int: - """The size the client declared at request time, or 0 if the row has none. + """The bytes this document reserved at request time, or 0 if it reserved none. - 0 is the correct default for a document that reserved nothing at request time - (an imported file, whose row is created with ``sizeBytes=0`` and whose true - size is only known here) — the reconcile then reserves the whole real size. + 0 for anything but an interactive upload (``byte_cap.reserved_at_request``): + an imported, crawled or synced document carries a real ``sizeBytes`` by the + time it lands, but reserved nothing, so the reconcile reserves the whole real + size and a failure has nothing to release. """ if not doc_row: return 0 - try: - return int(doc_row.get("sizeBytes") or 0) - except (TypeError, ValueError): - return 0 + from apis.shared.kb_backend import byte_cap + + return byte_cap.reserved_at_request(doc_row.get("sizeBytes"), doc_row.get("sourceAdapterKey")) def _delete_s3_object(bucket: str, key: str) -> None: @@ -706,6 +740,321 @@ def _commit_settled(assistant_id: str, document_id: str, real: int, reserved: in byte_cap.release(assistant_id, assistant_id, reserved - real) +def staged_version_to_reingest(doc_row: Optional[Dict[str, Any]]) -> Optional[str]: + """The source version a KB sync staged over this document and it has yet to + ingest, or ``None``. + + A KB sync that finds its source changed writes ``stagedContentHash`` BEFORE it + overwrites the S3 object (``kb_sync.records.update_document_sync_fields``), + and a completed re-ingest stamps the same hash as ``ingestedContentHash``. A + difference between the two is therefore a version still owed to the + knowledge base, and an event that finds them equal is a redelivery. Nothing + else writes the staged hash, so a document never synced has none and keeps the + settled early exit exactly as before. + + Only a terminal document qualifies. One still being ingested for the first + time is handled by the ordinary path, which reads the object as it is now. + ``failed`` qualifies too: new bytes deserve a fresh attempt, and a document + that failed on the byte cap or in Bedrock is otherwise stuck for good. + """ + if not doc_row or doc_row.get("status") not in (STATUS_COMPLETE, STATUS_FAILED): + return None + staged = doc_row.get("stagedContentHash") + if not staged or staged == doc_row.get("ingestedContentHash"): + return None + return str(staged) + + +def _reingest_changed_document( + bucket: str, + key: str, + assistant_id: str, + document_id: str, + filename: str, + doc_row: Dict[str, Any], + record: Optional[Dict[str, Any]], + staged: str, +) -> Dict[str, Any]: + """Re-ingest a settled document whose source a KB sync replaced in place. + + The first ingestion's path cannot do this. Its byte accounting is claimed once + per document (``settle_once``), which is what makes a redelivery harmless, and + Bedrock already reports the document ``INDEXED`` — so that path would neither + submit the new bytes nor count them. Here each staged version is handled once + instead, keyed on its hash (``byte_cap.claim_reingest``): + + 1. **Reserve the growth, then claim.** The S3 size is measured now and only + the difference from ``committedBytes`` is reserved, against the cap, before + anything is submitted. A file that grew past the cap is refused while the + previous version is still indexed and still served + (:func:`_record_reingest_over_cap`), rather than replacing it and then + failing the document. A redelivery finds the claim and reserves nothing. + 2. **Submit once.** ``reingestSubmittedAt`` records the submission, so a + redelivery waits on the ingestion already running instead of restarting it. + 3. **Wait for the NEW version.** Bedrock already holds a document under this + id, so a status it last updated before the submission is the old version's + and is waited out (``not_before``). + 4. **Complete.** One conditional write re-stamps ``committedBytes``, records + the version as ingested and drops the claim; the ledger then moves by the + difference (``byte_cap.settle_reingest``). + + The status stays ``complete`` throughout, so the document keeps answering from + its previous version until the new one has replaced it. + """ + import asyncio + + from apis.shared.kb_backend import byte_cap + from apis.shared.kb_backend.managed_backend import ManagedKbBackend + from apis.shared.kb_backend.protocol import DocumentSource + + summary: Dict[str, Any] = {"routed": "managed", "document_id": document_id} + + if doc_row.get("reingestHash") != staged: + try: + _claim_reingest(assistant_id, document_id, bucket, key, doc_row, record, staged) + except byte_cap.ByteCapExceeded: + _record_reingest_over_cap(assistant_id, document_id, staged) + return {**summary, "ingested": False, "note": "byte-cap-exceeded"} + # Re-read: the claim may have been won by a concurrent delivery of this + # version, or lost to a delete or a newer version. + doc_row = _get_doc_row(assistant_id, document_id) or {} + if doc_row.get("reingestHash") != staged or doc_row.get("status") == STATUS_DELETING: + logger.info( + f"document {document_id} was deleted or re-staged before its changed " + f"source could be re-ingested; leaving it to that change" + ) + return {**summary, "ingested": False, "note": "reingest-superseded"} + + backend = ManagedKbBackend(bucket=bucket) + submitted_at = doc_row.get("reingestSubmittedAt") + if submitted_at: + logger.info( + f"document {document_id}'s changed source was already submitted at " + f"{submitted_at}; waiting on that ingestion rather than restarting it" + ) + else: + submitted_at = _now_iso() + source = DocumentSource(document_id=document_id, filename=filename, s3_key=key) + # A submit that raises leaves the claim and its reservation in place for + # redelivery. The document is not failed: its previous version is intact. + asyncio.run(backend.ingest(assistant_id, source)) + _mark_reingest_submitted(assistant_id, document_id, staged, submitted_at) + + status, bedrock_updated_at = wait_until_indexed( + backend, assistant_id, document_id, not_before=submitted_at + ) + stale = _is_stale(bedrock_updated_at, submitted_at) + + if status in DOC_STATUSES_FAILED and not stale: + logger.error(f"re-ingestion of document {document_id} became {status}") + released = byte_cap.release_reingest_once(assistant_id, document_id, staged) + if released: + byte_cap.release(assistant_id, assistant_id, released) + set_document_terminal( + assistant_id, document_id, STATUS_FAILED, + error=f"the knowledge base reports this document as {status}", + ) + return {**summary, "ingested": True, "status": status} + + if stale or status not in (DOC_STATUS_INDEXED, *DOC_STATUSES_PARTIAL): + raise IngestionRoutingError( + f"document {document_id}'s changed source is {status} after waiting; " + f"leaving it for redelivery to confirm indexing" + ) + + indexed_at = bedrock_updated_at or _now_iso() + retrievable_at = wait_until_retrievable(backend, assistant_id, document_id) + if retrievable_at is None: + raise IngestionRoutingError( + f"document {document_id}'s changed source is INDEXED but was not " + f"retrievable within the poll window; leaving it for redelivery" + ) + + if not _complete_reingest( + assistant_id, document_id, staged, doc_row, indexed_at, retrievable_at + ): + return {**summary, "ingested": True, "note": "reingest-superseded"} + return { + **summary, + "ingested": True, + "note": "reingested", + "indexedAt": indexed_at, + "retrievableAt": retrievable_at, + } + + +def _claim_reingest( + assistant_id: str, + document_id: str, + bucket: str, + key: str, + doc_row: Dict[str, Any], + record: Optional[Dict[str, Any]], + staged: str, +) -> None: + """Reserve a staged version's growth and claim its re-ingest (step 1). + + Raises ``ByteCapExceeded`` with nothing reserved or claimed. Reserve comes + before the claim so a crash between the two leaks a reservation — the safe + direction — rather than leaving a claim that commits bytes nobody reserved. + """ + from apis.shared.kb_backend import byte_cap + + # A claim for an older version never completed; this change supersedes it. + # Return its reservation first, so it cannot count against this one's cap. + stale_claim = doc_row.get("reingestHash") + if stale_claim: + stale = byte_cap.release_reingest_once(assistant_id, document_id, str(stale_claim)) + if stale: + byte_cap.release(assistant_id, assistant_id, stale) + + real = byte_cap.object_size_bytes(bucket, key) + growth = max(real - int(doc_row.get("committedBytes") or 0), 0) + if growth: + elevated = bool((record or {}).get("elevatedByteCap")) + byte_cap.reserve(assistant_id, assistant_id, growth, byte_cap.effective_cap(elevated)) + + claimed, superseded = byte_cap.claim_reingest(assistant_id, document_id, staged, real, growth) + if not claimed: + # A concurrent delivery of this version claimed it first, or the row left + # a re-ingestable state. Either way this reservation is not needed. + if growth: + byte_cap.release(assistant_id, assistant_id, growth) + return + if superseded: + byte_cap.release(assistant_id, assistant_id, superseded) + + +def _mark_reingest_submitted( + assistant_id: str, document_id: str, staged: str, submitted_at: str +) -> None: + """Record that this version was submitted (step 2). Best-effort: without it a + redelivery submits again, which restarts indexing but loses nothing.""" + from botocore.exceptions import ClientError + + try: + _table().update_item( + Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + UpdateExpression="SET reingestSubmittedAt = :t", + ConditionExpression="reingestHash = :h", + ExpressionAttributeValues={":t": submitted_at, ":h": staged}, + ) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") != "ConditionalCheckFailedException": + raise + + +def _complete_reingest( + assistant_id: str, + document_id: str, + staged: str, + doc_row: Dict[str, Any], + indexed_at: str, + retrievable_at: str, +) -> bool: + """Record the re-ingested version and move the ledger by its size (step 4). + + The ``DOC#`` write comes first and carries everything — the new + ``committedBytes``, the ingested hash, the status, and the claim's removal — + conditioned on the claim still being this version's and the row not being + deleted. That is the re-ingest's :func:`byte_cap.record_commit`: refused means + a delete or a newer version owns the claim's reservation now, and this path + must not touch the ledger. ``byteCapSettled`` is set too, for a ``failed`` row + that never settled, so a later delete refunds rather than releases. + """ + from botocore.exceptions import ClientError + + from apis.shared.kb_backend import byte_cap + from apis.shared.kb_backend.records import RECORD_EXISTS + + new_bytes = int(doc_row.get("reingestBytes") or 0) + reserved = int(doc_row.get("reingestReservedBytes") or 0) + try: + response = _table().update_item( + Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + UpdateExpression=( + "SET #status = :complete, updatedAt = :now, indexedAt = :indexed, " + "retrievableAt = :retrievable, committedBytes = :bytes, " + "byteCapSettled = :true, ingestedContentHash = :h " + f"REMOVE {', '.join(byte_cap.REINGEST_CLAIM_ATTRIBUTES)}, ingestionError" + ), + ConditionExpression=( + f"{RECORD_EXISTS} AND #status <> :deleting " + f"AND reingestHash = :h AND attribute_not_exists(byteCapRefunded)" + ), + ExpressionAttributeNames={"#status": "status"}, + ExpressionAttributeValues={ + ":complete": STATUS_COMPLETE, + ":deleting": STATUS_DELETING, + ":now": _now_iso(), + ":indexed": indexed_at, + ":retrievable": retrievable_at, + ":bytes": new_bytes, + ":true": True, + ":h": staged, + }, + ReturnValues="ALL_OLD", + ) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") == "ConditionalCheckFailedException": + logger.info( + f"document {document_id} was deleted or re-staged while its changed " + f"source was indexing; not recording the re-ingest" + ) + return False + raise + + previous = int((response.get("Attributes") or {}).get("committedBytes") or 0) + byte_cap.settle_reingest(assistant_id, assistant_id, previous, new_bytes, reserved) + logger.info( + f"document {document_id} re-ingested from its changed source: " + f"{previous} -> {new_bytes} bytes" + ) + return True + + +def _record_reingest_over_cap(assistant_id: str, document_id: str, staged: str) -> None: + """Refuse a changed source that would take the owner over the byte cap. + + Nothing was submitted, so the knowledge base keeps serving the previous + version and the document stays as it was. The sync's change-detection gates + (``sourceEtag``, ``contentHash``) are cleared so the next sync run stages the + source again — the only thing that would retry it once the owner has freed + space. Conditioned on the version, so a newer change is left alone. + """ + from botocore.exceptions import ClientError + + from apis.shared.kb_backend.records import RECORD_EXISTS + + logger.warning( + f"document {document_id}'s changed source would exceed the byte cap; keeping " + f"its previous version and retrying on the next sync" + ) + try: + _table().update_item( + Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + UpdateExpression=( + "SET ingestionError = :err, updatedAt = :now REMOVE sourceEtag, contentHash" + ), + ConditionExpression=( + f"{RECORD_EXISTS} AND #status <> :deleting AND stagedContentHash = :h" + ), + ExpressionAttributeNames={"#status": "status"}, + ExpressionAttributeValues={ + ":err": ( + "the updated source exceeds the knowledge base's storage limit; " + "delete unused documents or request an elevated storage tier" + ), + ":now": _now_iso(), + ":deleting": STATUS_DELETING, + ":h": staged, + }, + ) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") != "ConditionalCheckFailedException": + raise + + def handle_object(bucket: str, key: str) -> Dict[str, Any]: """Route one uploaded object. Returns a summary for logging and tests.""" from apis.shared.kb_backend.records import BORN_MANAGED, ENGINE_MANAGED @@ -772,13 +1121,22 @@ def handle_object(bucket: str, key: str) -> Dict[str, Any]: from apis.shared.kb_backend.managed_backend import ManagedKbBackend from apis.shared.kb_backend.protocol import DocumentSource - # The DOC# row carries the size declared and reserved at request time - # (sizeBytes) and the byteCapSettled marker. Read it once. If a PRIOR delivery + # The DOC# row carries what was reserved at request time (`_declared_bytes`) + # and the byteCapSettled marker. Read it once. If a PRIOR delivery # already drove this document terminal AND settled its bytes, this is a # redelivery and re-running the byte accounting would double-count — a second # commit drives reservedBytes negative, a second release over-credits the cap. # Return without touching anything (Requirement 12.4/12.5). doc_row = _get_doc_row(assistant_id, document_id) + staged = staged_version_to_reingest(doc_row) + if staged: + # ...unless a KB sync has since overwritten the object with a changed + # source. Then this event is the new version's, not a redelivery, and the + # settled early exit below would leave the knowledge base serving the old + # content for good. + return _reingest_changed_document( + bucket, key, assistant_id, document_id, filename, doc_row, record, staged + ) if ( doc_row and doc_row.get("byteCapSettled") diff --git a/backend/src/apis/app_api/kb_migration/requirements.txt b/backend/src/apis/app_api/kb_migration/requirements.txt index 0f1a5cf32..51944aed6 100644 --- a/backend/src/apis/app_api/kb_migration/requirements.txt +++ b/backend/src/apis/app_api/kb_migration/requirements.txt @@ -17,8 +17,8 @@ # members, `embeddingDataType` enum is ['FLOAT32','BINARY'], and all four # document operations — Ingest/Get/List/DeleteKnowledgeBaseDocuments — exist. # No `AWS_DATA_PATH` side-load is required, and none must be relied on. -boto3==1.43.68 -botocore==1.43.68 +boto3==1.43.103 +botocore==1.43.103 # Nothing else. The four handlers' whole import closure is 16 first-party # modules plus these two, which is the point of `kb_backend` being its own diff --git a/backend/src/apis/app_api/kb_sync/records.py b/backend/src/apis/app_api/kb_sync/records.py index 4faff18ae..946ad8019 100644 --- a/backend/src/apis/app_api/kb_sync/records.py +++ b/backend/src/apis/app_api/kb_sync/records.py @@ -81,11 +81,19 @@ def update_document_sync_fields( previous_chunk_count: Optional[int] = None, last_synced_at: Optional[str] = None, sync_policy_id: Optional[str] = None, + staged_content_hash: Optional[str] = None, ) -> None: """Targeted update of the sync-bookkeeping fields on a document record. Only sets the fields passed — safe alongside the ingestion pipeline's own targeted UpdateExpressions (which never touch these attributes). + + ``staged_content_hash`` is for a sync that is about to overwrite the + document's S3 object with changed bytes, and must be written BEFORE the + overwrite. A managed knowledge base's ingestion consumer reads it to tell + the overwrite's event from a redelivery of the original upload's + (``kb_migration.ingestion_consumer.staged_version_to_reingest``); without + it the changed bytes are never re-ingested. The legacy pipeline ignores it. """ set_parts = [] values: Dict[str, Any] = {} @@ -104,6 +112,9 @@ def update_document_sync_fields( if sync_policy_id is not None: set_parts.append("syncPolicyId = :spid") values[":spid"] = sync_policy_id + if staged_content_hash is not None: + set_parts.append("stagedContentHash = :staged") + values[":staged"] = staged_content_hash if not set_parts: return @@ -114,6 +125,69 @@ def update_document_sync_fields( ) +def rollback_document_sync_fields( + assistant_id: str, + document_id: str, + *, + written: Dict[str, str], + previous: Dict[str, Optional[str]], +) -> bool: + """Undo a pre-stage sync write whose S3 stage then failed. + + ``written`` maps each attribute to the value the sync just wrote; + ``previous`` maps it to the value it held before (``None`` = absent). + The change-detection gates (``sourceEtag``, ``contentHash``) have to be + written before staging only because they travel with the fields that + must precede the ObjectCreated event, so a stage that never happened + must not leave them advanced — the next run would match them and never + stage the change. ``stagedContentHash`` goes back too: left behind, it + names a version the object does not hold. + + Conditioned on every attribute still holding what was written, so a + newer write (another run, the consumer clearing the gates) is left + alone. Returns False when that condition refused the rollback. + """ + from botocore.exceptions import ClientError + + set_parts = [] + remove_parts = [] + conditions = [] + names: Dict[str, str] = {} + values: Dict[str, Any] = {} + for i, (attribute, value) in enumerate(written.items()): + names[f"#a{i}"] = attribute + values[f":w{i}"] = value + conditions.append(f"#a{i} = :w{i}") + prior = previous.get(attribute) + if prior is None: + remove_parts.append(f"#a{i}") + else: + set_parts.append(f"#a{i} = :p{i}") + values[f":p{i}"] = prior + if not conditions: + return True + + expression = [] + if set_parts: + expression.append("SET " + ", ".join(set_parts)) + if remove_parts: + expression.append("REMOVE " + ", ".join(remove_parts)) + try: + _table().update_item( + Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + UpdateExpression=" ".join(expression), + ConditionExpression=" AND ".join(conditions), + ExpressionAttributeNames=names, + ExpressionAttributeValues=values, + ) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") != "ConditionalCheckFailedException": + raise + logger.info(f"Document {document_id}'s sync fields changed since the failed stage; not rolling back") + return False + return True + + def clear_document_sync_policy_id(assistant_id: str, document_id: str) -> None: """Remove the SyncPolicy back-pointer when its policy is deleted.""" _table().update_item( diff --git a/backend/src/apis/app_api/kb_sync/requirements.txt b/backend/src/apis/app_api/kb_sync/requirements.txt index bf99f470b..e8bb3daa5 100644 --- a/backend/src/apis/app_api/kb_sync/requirements.txt +++ b/backend/src/apis/app_api/kb_sync/requirements.txt @@ -1,9 +1,9 @@ # kb-sync Lambda image dependencies (backend/Dockerfile.kb-sync). # Exact pins per repo policy; versions match backend/uv.lock. -boto3==1.43.68 +boto3==1.43.103 pydantic==2.12.5 httpx==0.28.1 -bedrock-agentcore==1.21.0 +bedrock-agentcore==1.24.0 # Web re-crawl path (crawler markdown extraction) beautifulsoup4==4.13.5 trafilatura==2.0.0 diff --git a/backend/src/apis/app_api/kb_sync/worker.py b/backend/src/apis/app_api/kb_sync/worker.py index 81428e852..3d13c8089 100644 --- a/backend/src/apis/app_api/kb_sync/worker.py +++ b/backend/src/apis/app_api/kb_sync/worker.py @@ -208,17 +208,40 @@ async def _sync_drive_file(policy: SyncPolicy) -> Dict[str, Any]: return await _finish(policy, "unchanged") # Changed: stash the old chunk count for the ingestion tail-delete - # (shrinkage cleanup) BEFORE staging, then overwrite the S3 object. + # (shrinkage cleanup) and mark the version being staged — what a managed + # KB's consumer uses to tell this overwrite from a redelivery — BEFORE + # staging, then overwrite the S3 object. previous_chunk_count = int(document.get("chunkCount") or 0) + synced_at = _now_timestamp() records.update_document_sync_fields( assistant_id, policy.source_ref, source_etag=new_etag, content_hash=content_hash, previous_chunk_count=previous_chunk_count, - last_synced_at=_now_timestamp(), + last_synced_at=synced_at, + staged_content_hash=content_hash, ) - _stage_to_s3(document["s3Key"], downloaded.content, downloaded.content_type) + try: + _stage_to_s3(document["s3Key"], downloaded.content, downloaded.content_type) + except Exception: + # The gates advanced above but the bytes never reached S3; left + # as they are, the next run would match them and never stage + # this change. previousChunkCount is harmless to leave — the next + # changed run rewrites it before its own stage. + written = { + "sourceEtag": new_etag, + "contentHash": content_hash, + "stagedContentHash": content_hash, + "lastSyncedAt": synced_at, + } + records.rollback_document_sync_fields( + assistant_id, + policy.source_ref, + written=written, + previous={attribute: document.get(attribute) for attribute in written}, + ) + raise logger.info( f"Sync policy {policy.policy_id}: staged {len(downloaded.content)} changed bytes for " f"document {policy.source_ref} (prev chunks: {previous_chunk_count})" @@ -295,11 +318,15 @@ async def _sync_web_crawl(policy: SyncPolicy) -> Dict[str, Any]: web_docs[url] = item now = _now_timestamp() + # document_id -> (values written before staging, values they replaced), + # so a page whose stage then fails can be rolled back. + pre_stage_writes: Dict[str, Any] = {} async def on_result(url: str, document_id: str, outcome: str, etag, content_hash) -> None: if outcome == "changed": # BEFORE the S3 overwrite: stash the previous chunk count for - # the ingestion tail-delete, alongside the new gate values. + # the ingestion tail-delete and the staged version for a managed + # KB's re-ingest, alongside the new gate values. records.update_document_sync_fields( assistant_id, document_id, @@ -307,6 +334,14 @@ async def on_result(url: str, document_id: str, outcome: str, etag, content_hash content_hash=content_hash, previous_chunk_count=int(web_docs[url].get("chunkCount") or 0), last_synced_at=now, + staged_content_hash=content_hash, + ) + written = {"contentHash": content_hash, "stagedContentHash": content_hash, "lastSyncedAt": now} + if etag is not None: + written["sourceEtag"] = etag + pre_stage_writes[document_id] = ( + written, + {attribute: web_docs[url].get(attribute) for attribute in written}, ) elif outcome == "unchanged": records.update_document_sync_fields( @@ -318,6 +353,13 @@ async def on_result(url: str, document_id: str, outcome: str, etag, content_hash records.update_document_sync_fields( assistant_id, document_id, content_hash=content_hash, last_synced_at=now ) + pre_stage_writes[document_id] = ({"contentHash": content_hash, "lastSyncedAt": now}, {}) + elif outcome == "stage_failed" and document_id in pre_stage_writes: + # The page's gates advanced above but its bytes never reached S3; + # roll them back so the next re-crawl stages it instead of + # hash-matching a version the knowledge base never received. + written, previous = pre_stage_writes.pop(document_id) + records.rollback_document_sync_fields(assistant_id, document_id, written=written, previous=previous) refresh = crawler.RefreshState( docs={ @@ -397,9 +439,13 @@ async def on_result(url: str, document_id: str, outcome: str, etag, content_hash logger.info( f"Sync policy {policy.policy_id}: re-crawl done — {refresh.changed} changed, " - f"{refresh.created} new, {refresh.unchanged} unchanged, {deleted} deleted" + f"{refresh.created} new, {refresh.unchanged} unchanged, {deleted} deleted, " + f"{refresh.stage_failed} failed to stage" ) - result = "changed" if (refresh.changed or refresh.created or deleted) else "unchanged" + # A page that failed to stage was counted as changed/created when it was + # emitted, but nothing reached the knowledge base. + staged = refresh.changed + refresh.created - refresh.stage_failed + result = "changed" if (staged or deleted) else "unchanged" return await _finish(policy, result) diff --git a/backend/src/apis/app_api/models/routes.py b/backend/src/apis/app_api/models/routes.py index f451d37a3..4f6867a0a 100644 --- a/backend/src/apis/app_api/models/routes.py +++ b/backend/src/apis/app_api/models/routes.py @@ -9,6 +9,7 @@ from apis.app_api.admin.models import ManagedModelsListResponse from apis.shared.auth import User, get_current_user_from_session +from apis.shared.models.modalities import is_speech_model from apis.shared.models.managed_models import list_all_managed_models from apis.app_api.admin.services.model_icons import ModelIconError, read_model_icon from apis.app_api.admin.services.model_access import ( @@ -62,8 +63,10 @@ async def list_models_for_user( ) try: - # Get all models, then filter by access - all_models = await list_all_managed_models() + # Get all models, then filter by access. Speech-to-speech rows (Nova 2 + # Sonic) are catalogued so voice sessions price, but they answer no + # chat API — offering one in the picker would fail every turn. + all_models = [m for m in await list_all_managed_models() if not is_speech_model(m)] # Filter models based on hybrid AppRole + JWT role access accessible_models = await model_access_service.filter_accessible_models( diff --git a/backend/src/apis/app_api/user_settings/routes.py b/backend/src/apis/app_api/user_settings/routes.py index 15ee4b024..17f478d4b 100644 --- a/backend/src/apis/app_api/user_settings/routes.py +++ b/backend/src/apis/app_api/user_settings/routes.py @@ -41,6 +41,10 @@ async def update_settings( update_data = body.model_dump(by_alias=True, exclude_unset=True) if "personalInstructions" in update_data: update_data["personalInstructions"] = (update_data["personalInstructions"] or "").strip() or None + if "voiceId" in update_data: + # '' is the wire value for "back to the default"; the repository + # removes a None attribute rather than storing an empty string. + update_data["voiceId"] = update_data["voiceId"] or None # Validate defaultModelId if provided and not null if "defaultModelId" in update_data and update_data["defaultModelId"] is not None: diff --git a/backend/src/apis/app_api/web_sources/crawler.py b/backend/src/apis/app_api/web_sources/crawler.py index 5b284f065..537d40ecb 100644 --- a/backend/src/apis/app_api/web_sources/crawler.py +++ b/backend/src/apis/app_api/web_sources/crawler.py @@ -152,7 +152,10 @@ class RefreshState: "unchanged" — 304 or identical content hash; nothing re-staged "changed" — existing doc, new bytes; emitted BEFORE staging so the worker can stash the previous chunk count first - "created" — page new to this crawl; emitted after staging + "created" — page new to this crawl; also emitted before staging + "stage_failed" — the S3 stage after a "changed"/"created" raised; the + worker rolls back the gate values it wrote, or the next + refresh would hash-match bytes that never reached S3 `seen_urls` collects every URL that survived the robots gate — the worker diffs it against `docs` for miss counting. Fetch failures ARE seen (a flaky page is not a missing page); robots-disallowed pages are @@ -167,6 +170,7 @@ class RefreshState: changed: int = 0 unchanged: int = 0 created: int = 0 + stage_failed: int = 0 async def _emit( self, @@ -182,6 +186,8 @@ async def _emit( self.unchanged += 1 elif outcome == "created": self.created += 1 + elif outcome == "stage_failed": + self.stage_failed += 1 if self.on_result is not None: await self.on_result(url, document_id, outcome, etag, content_hash) @@ -648,12 +654,36 @@ async def worker(url: str, depth: int) -> None: url, len(markdown.encode("utf-8")), ) - s3_key = await _put_markdown( - assistant_id=assistant_id, - document_id=document_id, - markdown=markdown, - filename=filename, - ) + try: + s3_key = await _put_markdown( + assistant_id=assistant_id, + document_id=document_id, + markdown=markdown, + filename=filename, + ) + except Exception as stage_err: + logger.warning("Staging %s to S3 failed: %s", url, stage_err) + if refresh is not None: + await refresh._emit(url, document_id, "stage_failed", etag, content_hash) + if existing is None: + await update_document_status( + assistant_id=assistant_id, + document_id=document_id, + status="failed", + error_message="The page could not be stored.", + error_details=str(stage_err)[:500], + ) + await increment_counters( + assistant_id=assistant_id, + crawl_id=crawl_id, + failed_delta=1, + ) + # The fetch itself succeeded, so its links are good. + # Not walking them would leave every page below this + # one unseen, and a refresh counts unseen pages as + # misses toward deletion. + await enqueue_links(html, url, depth) + return await update_document_import_metadata( assistant_id=assistant_id, document_id=document_id, diff --git a/backend/src/apis/inference_api/chat/routes.py b/backend/src/apis/inference_api/chat/routes.py index 4727deef2..52c9320c5 100644 --- a/backend/src/apis/inference_api/chat/routes.py +++ b/backend/src/apis/inference_api/chat/routes.py @@ -8,10 +8,12 @@ """ import asyncio +import time import json import logging from collections import OrderedDict -from typing import TYPE_CHECKING, AsyncGenerator, Optional, Tuple, Union +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, AsyncGenerator, Optional, Tuple, Union from fastapi import APIRouter, Depends, HTTPException, status from fastapi.responses import JSONResponse, StreamingResponse @@ -27,15 +29,19 @@ ) from apis.inference_api.runtime_health import ping_payload from apis.shared.feature_flags import ( + agent_build_shared_session_enabled, agent_preparing_phase_enabled, + kb_search_ahead_enabled, agents_enabled, attachment_turn_guard_enabled, + inline_attachment_persist_enabled, memory_spaces_enabled, mid_turn_steering_enabled, skills_enabled, ) from apis.shared.files.document_read import is_document_class from apis.shared.files.file_resolver import get_file_resolver +from apis.shared.files.inline_persist import get_inline_attachment_persister from apis.shared.files.models import ( INLINE_ATTACHMENTS_MAX_TOTAL_BYTES, MAX_FILES_PER_MESSAGE, @@ -90,7 +96,7 @@ from .app_tool_dispatch import AppToolCallError, dispatch_app_tool_call from .agent_binding_policy import binds_conversation from .models import FileContent, InvocationRequest -from .service import generate_conversation_title, get_agent +from .service import generate_conversation_title, get_agent, process_build_count from .turn_timing import TurnPrelude from .system_prompt_resolver import ( append_active_prompt, @@ -224,13 +230,14 @@ async def _session_has_messages(*, session_id: str, user_id: str) -> bool: has no history, because history produced under other instructions, tools and skills is exactly what a binding would misrepresent. - Deliberately `limit=1` — the question is existence, not count, and this runs on - the invocation path. + An existence check, priced like one (`session_has_messages`: one filtered + `ListEvents`, no payloads). It used to be `get_messages(limit=1)`, which read the + whole history and its metadata to return one message — this runs on the + invocation path, before the stream opens. """ - from apis.shared.sessions.messages import get_messages + from apis.shared.sessions.messages import session_has_messages - response = await get_messages(session_id=session_id, user_id=user_id, limit=1) - return bool(response.messages) + return await session_has_messages(session_id, user_id) def is_preview_session(session_id: str) -> bool: @@ -1529,6 +1536,7 @@ def _build_attachment_guidance( dropped_over_count_names: list[str] | None = None, dropped_over_count_total: int = 0, max_files: int = 0, + unpersisted: list | None = None, ) -> str: """Return a short markdown addendum describing how attachments will be handled, to append to the user's message so the agent (and the user) @@ -1540,6 +1548,10 @@ def _build_attachment_guidance( follow-up message). They get separate sentences because the remedy differs. ``dropped_over_count_*`` describe files beyond the per-message count cap: names where known (direct ``files``), a count otherwise. + ``unpersisted`` are inline spreadsheets/decks that could not be stored as + session files (``PersistFailure`` entries); they are not diverted and the + tools cannot see them, so the note must say so instead of the usual + "available through the tool" line. """ parts: list[str] = [] @@ -1604,6 +1616,14 @@ def _build_attachment_guidance( f"single message. Send them in a follow-up message._" ) + if unpersisted: + described = "; ".join(f"`{x.file.filename}` ({x.reason})" for x in unpersisted) + parts.append( + f"_Attached file(s) {described} could not be stored for this " + f"conversation and were not attached. Upload them through the " + f"Files panel and re-send your message._" + ) + if dropped_over_count_total > 0: limit = f"{max_files} file" + ("s" if max_files != 1 else "") if dropped_over_count_names: @@ -1986,283 +2006,219 @@ def _build_skill_invocation_note(skill_slugs: list[str]) -> str: ) -@router.post("/invocations") -async def invocations(request: InvocationRequest, current_user: User = Depends(get_current_user_trusted)): + +# ============================================================ +# Turn phases +# ============================================================ +# +# `invocations` runs a turn as the sequence docs/specs/turn-path-ttft.md §2B +# lays out. Each function below is one contiguous stretch of that sequence, +# named for what it settles and for the `turn_prelude` mark that times it. A +# phase takes what it reads and returns what it produces; none closes over the +# handler's locals. That is what lets the handler read as the map, and lets a +# change to one phase be reasoned about — and tested — without the rest. + + +def _sse_headers(session_id: str) -> dict: + """Response headers for every SSE stream this route returns. + + ``X-Accel-Buffering: no`` defeats proxy buffering so events after + ``message_stop`` (``oauth_required`` and friends) reach the browser at once. """ - AgentCore Runtime standard invocation endpoint (required) + return {"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": session_id} + - Supports user-specific tool filtering and SSE streaming. - Creates/caches agent instance per session + tool configuration. - Uses the authenticated user's ID from the JWT token. +def _refuse_turn( + input_data: InvocationRequest, + user_id: str, + *, + message: str, + stop_reason: str, + metadata_event: Union[QuotaExceededEvent, ConversationalErrorEvent, None], +) -> StreamingResponse: + """End the turn before any model call, as one short assistant message. - Quota enforcement (when enabled via ENABLE_QUOTA_ENFORCEMENT=true): - - Checks user quota before processing - - Streams quota_exceeded as assistant message if quota exceeded (better UX) - - Injects quota_warning event into stream if approaching limit + Errors stream as assistant messages (CLAUDE.md): a quota block, a retired + model, a blocked Agent binding or an archived project each become a + persisted assistant turn carrying its metadata event, rather than an HTTP + status the SPA would render as a generic failure. """ - input_data = request - user_id = current_user.user_id - auth_token = current_user.raw_token + return StreamingResponse( + stream_conversational_message( + message=message, + stop_reason=stop_reason, + metadata_event=metadata_event, + session_id=input_data.session_id, + user_id=user_id, + user_input=input_data.message, + ), + media_type="text/event-stream", + headers=_sse_headers(input_data.session_id), + ) - # Where the pre-stream time goes. Everything between here and the - # `StreamingResponse` return happens with NO channel open to the client — - # measured at 3.75s on a warm turn — so this is the only way to see which - # stage owns it. Pure timing: nothing reaches the model. - # See `turn_timing.py` and docs/specs/agent-state-feedback.md. - prelude = TurnPrelude() - # Whether this turn's agent is built inside the stream (PR-3). Recorded on - # the `turn_prelude` line so the two shapes stay distinguishable in the - # logs once the flag has been on for a while. - deferred_build = False - # Refuse a turn against a session id another user already owns. - # - # Session ids travel in shareable URLs (`/s/{sessionId}`). Opening someone - # else's link 404s on the metadata read, but the SPA then treats the - # session as new and lets the user send — which used to fork the id: a - # SECOND metadata row under the requester, on the same session, invisible - # to both parties. In prod on 2026-08-31 that also left the original - # owner's session resolving non-deterministically between the two rows. - # - # Not a confidentiality fix — conversation content is keyed by actor id in - # AgentCore Memory, so the second user only ever saw an empty thread. This - # stops the id from being forked at all. 404 rather than 403 so the - # response says nothing about whether the session exists, matching what - # `GET /sessions/{id}/metadata` already returns for the same case. - # - # ONE read of the session's META row, shared by everything in the preamble - # that used to fetch it again (PR-2, docs/specs/turn-latency-preamble.md). - # Measured on dev: eight separate reads of this item cost ~445ms of a - # ~455ms stage, because a GSI query from an AgentCore Runtime container is - # ~53ms rather than the ~12ms an in-region figure would suggest. - # - # Deliberately explicit rather than a per-request memo inside - # `_get_session_by_gsi`: CLAUDE.md's "never cache session state" rule has - # been paid for twice (#741, #751), and a snapshot callers opt into cannot - # leak into one that needs a fresh read. - session_meta = await load_session_meta(input_data.session_id, user_id) - if session_meta.owned_by_other: - logger.warning( - "Rejected invocation for session %s — owned by a different user", - _sanitize_log(input_data.session_id), - ) - raise HTTPException(status_code=404, detail="Session not found") - # First of the preamble's five sub-stages (docs/specs/turn-latency-preamble.md). - # The coarse `preamble` number survives as `groups.preamble` in the emitted - # line, so the four-turn baseline in the agent-state-feedback spec stays - # comparable across this split. - prelude.mark("preamble.ownership") - # Resume requests reuse the cached agent and its paused interrupt state; - # they bypass quota, file resolution, and RAG augmentation because those - # already ran on the original turn that got paused. - is_resume = bool(input_data.interrupt_responses) - # Resolve the effective agent type: the client's explicit choice, else the - # compiled-in default ("chat"). Used for the skill resolution below and the - # non-resume get_agent calls (resume reuses the snapshot's type). An Agent - # that binds skills is coerced to "skill" later by the agent-binding - # resolver. - effective_agent_type = input_data.agent_type or DEFAULT_AGENT_TYPE - # Skills feature deferred for this environment: neutralize the legacy - # "skill" agent type, which is a ChatAgent alias since v2 PR-2. Voice and - # other agent types pass through untouched. - if not skills_enabled() and effective_agent_type == "skill": - effective_agent_type = "chat" - # Resolve the user's *effective* skills once for the whole request: the - # accessible set (catalog ∪ own), narrowed by the client's per-turn - # enabled_skills selection. Threaded into every get_agent call below so they - # share one skills_hash cache key (otherwise the app-tool-call / resume paths - # would miss the main turn's cached agent). - # - # Skills v2: this is no longer gated on agent_type == "skill". Skills are a - # plain-chat capability now — the picker in model settings sends - # enabled_skills on an ordinary turn and ChatAgent mounts the AgentSkills - # plugin. The opt-in default (D6) is what keeps this cheap: an absent or - # empty selection short-circuits to [] without touching RBAC or the skill - # table, so every turn that doesn't ask for skills costs exactly what it did - # before. An Agent's skill bindings override this further down. - effective_skill_ids = None - if skills_enabled() and input_data.enabled_skills: - effective_skill_ids = _apply_enabled_skills_filter( - await _resolve_accessible_skill_ids(current_user), - input_data.enabled_skills, +def _forbidden_turn(input_data: InvocationRequest, user_id: str, message: str) -> StreamingResponse: + """A refusal the user cannot recover by retrying (``recoverable=False``).""" + return _refuse_turn( + input_data, + user_id, + message=message, + stop_reason="error", + metadata_event=ConversationalErrorEvent( + code=ErrorCode.FORBIDDEN, message=message, recoverable=False + ), + ) + + +async def _agent_for_app_dispatch( + input_data: InvocationRequest, + current_user: User, + user_id: str, + auth_token: Optional[str], + *, + agent_type: str, + accessible_skill_ids: Optional[list], +): + """The conversation's agent for an MCP App dispatch (map stage B3). + + An App's ``tools/call`` and ``ui/update-model-context`` run no model turn, + but they need the same ``Agent`` the real turns use: the MCP client session, + its auth, and ``agent.state``, where pushed context lives. So the model, its + settings and the effective tool list are resolved exactly as the main turn + resolves them, and ``get_agent`` reads the same cache slot with + ``cache_write=False`` — this path builds no injected tools, so an agent it + built must never seed a slot the real turns would then hit. + """ + request_inference_params = dict(input_data.inference_params or {}) + dispatch_model_id, dispatch_provider = input_data.model_id, input_data.provider + if not dispatch_model_id: + dispatch_model_id, dispatch_provider = await _resolve_fallback_model( + user_id, current_user, dispatch_provider ) - # Near-zero on a turn that selects no skills — the opt-in default (D6) - # short-circuits before touching RBAC or the skill table. A non-trivial - # number here means the RBAC cache missed or the owner-index query is slow. - prelude.mark("preamble.skills") - # A "Continue" after a max_tokens truncation. Like resume, it bypasses - # quota / RAG / file resolution and does NOT clear the turn state; unlike - # resume there is no interrupt to validate — the agent is rebuilt from the - # resent params and re-entered with an empty prompt (assistant-prefill). - is_continuation = bool(input_data.continue_truncated) - # Marketplace D11: the Agent was `@`-mentioned in the composer, so it runs - # this turn only — it does not bind the conversation. Only meaningful - # alongside `rag_assistant_id`; on its own it does nothing. - is_agent_mention = bool(input_data.agent_mention) and bool(input_data.rag_assistant_id) - logger.info( - "Invocation request received (resume=%s, continue_truncated=%s, agent_mention=%s)" - % (is_resume, is_continuation, is_agent_mention) + caching_enabled, inference_params, mantle_api_mode, mantle_region, registry_provider = await _resolve_model_settings( + model_id=dispatch_model_id, + explicit_caching_enabled=input_data.caching_enabled, + request_inference_params=request_inference_params, + ) + return await get_agent( + session_id=input_data.session_id, + user_id=user_id, + auth_token=auth_token, + # Same auto-enable seam as the main turn, so a spreadsheet + # session's dispatch reads the slot the real turns fill — and + # the same always-on union, or the dispatch would compute a + # different effective list and miss into its own agent-cache + # slot on every App call. + enabled_tools=await _apply_admin_always_on_tools( + await _apply_attachment_tool_autoenable( + input_data.enabled_tools, current_user, input_data.session_id, user_id + ), + current_user, + ), + model_id=dispatch_model_id, + system_prompt=await _plain_turn_prompt(input_data, user_id), + caching_enabled=caching_enabled, + provider=dispatch_provider or registry_provider, + inference_params=inference_params, + mantle_api_mode=mantle_api_mode, + mantle_region=mantle_region, + agent_type=agent_type, + is_resume=False, + accessible_skill_ids=accessible_skill_ids, + # This path builds no injected tools, but shares a cache slot + # with the real turns that do. Read the slot; never seed it. + cache_write=False, + assistant_id=input_data.rag_assistant_id, ) - logger.info("Message received") - # Model retirement (docs/specs/model-retirement.md §7). Resolved before anything - # builds an agent from ``input_data.model_id``, so the App tool-call / context / - # continuation paths land in the same agent-cache slot as the turn itself. A - # redirect swaps the provider as well: the request's described the retired - # model, and a successor on another transport misroutes with it. A denial is - # streamed at the access check below, once there is a turn to answer. - retired_model_denial: Optional[str] = None - requested_model = await resolve_effective_model(input_data.model_id) - if requested_model is not None: - if requested_model.denied: - retired_model_denial = retired_model_message(requested_model.retired) - elif requested_model.redirected: - input_data.model_id = requested_model.model_id - input_data.provider = requested_model.provider - # App-initiated tools/call (MCP Apps PR #5). Like resume/continuation it - # bypasses quota / RAG / file resolution / title — there is no model - # turn. We rebuild the conversation agent (so the MCP client session + - # auth are wired exactly as for a model-driven call), dispatch the one - # named tool, publish synthesized tool_use/tool_result into the thread - # via the per-session broker, and return the CallToolResult as JSON for - # app-api to relay back to the iframe. Inert behind the host flag (the - # UIToolCatalog is empty, so dispatch rejects every call as not - # app-visible). - if input_data.app_tool_call is not None: - atc = input_data.app_tool_call - try: - request_inference_params = dict(input_data.inference_params or {}) - dispatch_model_id, dispatch_provider = input_data.model_id, input_data.provider - if not dispatch_model_id: - dispatch_model_id, dispatch_provider = await _resolve_fallback_model( - user_id, current_user, dispatch_provider - ) - caching_enabled, inference_params, mantle_api_mode, mantle_region, registry_provider = await _resolve_model_settings( - model_id=dispatch_model_id, - explicit_caching_enabled=input_data.caching_enabled, - request_inference_params=request_inference_params, - ) - agent = await get_agent( - session_id=input_data.session_id, - user_id=user_id, - auth_token=auth_token, - # Same auto-enable seam as the main turn, so a spreadsheet - # session's dispatch reads the slot the real turns fill — and - # the same always-on union, or the dispatch would compute a - # different effective list and miss into its own agent-cache - # slot on every App call. - enabled_tools=await _apply_admin_always_on_tools( - await _apply_attachment_tool_autoenable( - input_data.enabled_tools, current_user, input_data.session_id, user_id - ), - current_user, - ), - model_id=dispatch_model_id, - system_prompt=await _plain_turn_prompt(input_data, user_id), - caching_enabled=caching_enabled, - provider=dispatch_provider or registry_provider, - inference_params=inference_params, - mantle_api_mode=mantle_api_mode, - mantle_region=mantle_region, - agent_type=effective_agent_type, - is_resume=False, - accessible_skill_ids=effective_skill_ids, - # This path builds no injected tools, but shares a cache slot - # with the real turns that do. Read the slot; never seed it. - cache_write=False, - assistant_id=input_data.rag_assistant_id, - ) - payload = await dispatch_app_tool_call( - agent, - session_id=input_data.session_id, - user_id=user_id, - tool_use_id=atc.tool_use_id, - tool_name=atc.tool_name, - arguments=atc.arguments, - ) - return JSONResponse(payload) - except AppToolCallError as e: - # 200 + envelope, not `status_code=e.code`: AgentCore Runtime - # rewrites any non-2xx to a generic 424 and discards the - # message, so a deliberate 409 ("connect the account") reached - # the SPA as "check your CloudWatch logs". app-api restores the - # real status. See `mcp_apps.error_envelope`. - return app_tool_error_response(e.message, e.code) - except HTTPException: - raise - except Exception: - logger.error("app tools/call invocation failed", exc_info=True) - return JSONResponse({"error": "Internal error"}, status_code=500) +@dataclass +class TurnAttachments: + """What this turn's attachments resolved to (map stage B4, ``preamble.files``). - # App-pushed model context (MCP Apps PR #6, `ui/update-model-context`). - # Like app_tool_call it bypasses quota / RAG / file resolution / title - # and runs NO model turn — we rebuild the conversation agent (so the - # same cached `agent.state` is reused) and stash the payload under - # `mcp_apps.context[resource_uri]`. The next real user turn merges and - # clears it. Inert behind the host flag (no live App ever calls this). - if input_data.app_context_update is not None: - acu = input_data.app_context_update - try: - request_inference_params = dict(input_data.inference_params or {}) - dispatch_model_id, dispatch_provider = input_data.model_id, input_data.provider - if not dispatch_model_id: - dispatch_model_id, dispatch_provider = await _resolve_fallback_model( - user_id, current_user, dispatch_provider - ) - caching_enabled, inference_params, mantle_api_mode, mantle_region, registry_provider = await _resolve_model_settings( - model_id=dispatch_model_id, - explicit_caching_enabled=input_data.caching_enabled, - request_inference_params=request_inference_params, - ) - agent = await get_agent( - session_id=input_data.session_id, - user_id=user_id, - auth_token=auth_token, - # Same auto-enable seam as the main turn, so a spreadsheet - # session's dispatch reads the slot the real turns fill — and - # the same always-on union, or the dispatch would compute a - # different effective list and miss into its own agent-cache - # slot on every App call. - enabled_tools=await _apply_admin_always_on_tools( - await _apply_attachment_tool_autoenable( - input_data.enabled_tools, current_user, input_data.session_id, user_id - ), - current_user, - ), - model_id=dispatch_model_id, - system_prompt=await _plain_turn_prompt(input_data, user_id), - caching_enabled=caching_enabled, - provider=dispatch_provider or registry_provider, - inference_params=inference_params, - mantle_api_mode=mantle_api_mode, - mantle_region=mantle_region, - agent_type=effective_agent_type, - is_resume=False, - accessible_skill_ids=effective_skill_ids, - # Same partial-toolset hazard as app_tool_call above. - cache_write=False, - assistant_id=input_data.rag_assistant_id, - ) - payload = dispatch_app_context_update( - agent, - resource_uri=acu.resource_uri, - content=acu.content, - structured_content=acu.structured_content, - ) - return JSONResponse(payload) - except AppContextUpdateError as e: - # Same AgentCore flattening as the app_tool_call path above. - return app_tool_error_response(e.message, e.code) - except HTTPException: - raise - except Exception: - logger.error("app context update invocation failed", exc_info=True) - return JSONResponse({"error": "Internal error"}, status_code=500) + ``files_to_send`` is the inline set the model receives as document/image + blocks. The three diverted lists never go inline (spreadsheets and decks + route through their tools; oversized and over-budget files are dropped with + a note). ``marker_names`` is every filename that still exists in the + session, in attachment order, for the ``[Attached files: …]`` marker the + SPA replays on reload. ``turn_has_document`` is the *classified* answer to + "did this turn attach something ``document_read`` can read", which feeds + the tool-injection gate and therefore ``toolConfig``. + """ - if input_data.enabled_tools: - logger.info(f"Enabled tools ({len(input_data.enabled_tools)})") + recovered_upload_ids: list = field(default_factory=list) + files_to_send: list = field(default_factory=list) + diverted_tabular: list = field(default_factory=list) + diverted_presentations: list = field(default_factory=list) + oversized_inline: list = field(default_factory=list) + over_budget_inline: list = field(default_factory=list) + dropped_over_count_names: list = field(default_factory=list) + dropped_over_count_total: int = 0 + turn_has_document: bool = False + marker_names: list = field(default_factory=list) + # Inline (base64 ``files``) spreadsheets/decks that could not be stored as + # session files — ``PersistFailure`` entries (file + user-facing reason). + # Dropped from the turn: not diverted, not in the marker, named in the note. + unpersisted_inline: list = field(default_factory=list) + + +async def _persist_inline_diverted( + input_data: InvocationRequest, + user_id: str, + diverted_tabular: list, + diverted_presentations: list, +) -> tuple[list, list, list]: + """Give inline bytes of a diverted class the S3 object + row an upload has. + + Diverted files are dropped from the prompt on the promise that the + Spreadsheet Analysis / PowerPoint tools can reach them — and those tools + find session files only through ``FileUploadRepository.list_session_files``. + Uploads (``file_upload_ids``) are already there; base64 ``files`` are not, + so until this ran the note was a lie for every headless caller. Identity, + not filename, picks the inline ones: a resolved upload may share a name. + + Returns the two diverted lists with the failures removed, plus the failures. + Never raises; a failure is reported to the user and the turn continues. + """ + inline_identity = {id(f) for f in (input_data.files or [])} + inline_diverted = [f for f in diverted_tabular + diverted_presentations if id(f) in inline_identity] + if not inline_diverted or not inline_attachment_persist_enabled(): + return diverted_tabular, diverted_presentations, [] + + started = time.perf_counter() + persisted, failures = await get_inline_attachment_persister().persist( + user_id=user_id, session_id=input_data.session_id, files=inline_diverted + ) + logger.info( + "Persisted %d/%d inline diverted attachment(s) as session files in %.0fms", + len(persisted), + len(inline_diverted), + (time.perf_counter() - started) * 1000, + ) + if not failures: + return diverted_tabular, diverted_presentations, [] + failed = {id(x.file) for x in failures} + return ( + [f for f in diverted_tabular if id(f) not in failed], + [f for f in diverted_presentations if id(f) not in failed], + failures, + ) + +async def _resolve_turn_attachments( + input_data: InvocationRequest, + user_id: str, + session_meta, + *, + is_resume: bool, + is_continuation: bool, +) -> TurnAttachments: + """Map stage B4 (``preamble.files``): recover, fetch, dedupe, partition, budget. + + Mutates ``input_data.file_upload_ids`` when it re-attaches a previous + turn's unanswered uploads, because the write-ahead marker recorded in the + session-state phase reads the ids off the request. + """ # Recover attachments the PREVIOUS turn sent but never got an answer for. # # Inline document bytes are one-shot: they are stripped out of restored @@ -2429,6 +2385,10 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g f"{[(f.filename, _estimate_decoded_size(f)) for f in oversized_inline]}" ) + diverted_tabular, diverted_presentations, unpersisted_inline = await _persist_inline_diverted( + input_data, user_id, diverted_tabular, diverted_presentations + ) + # Aggregate budget for the turn (spec §4E / PR-6): the inline set is one # persisted message, and a message over ~7.5 MB raw fails the AgentCore # Memory write with SessionException. Trim first-fit in attachment order; @@ -2454,94 +2414,843 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g dropped_count=len(over_budget_inline), ) - # Both classes were dropped from the turn entirely; the marker must not - # promise a card for either. + # All three classes were dropped from the turn entirely; the marker must + # not promise a card for any of them. attachment_marker_names = _attachment_marker_names( - all_files, oversized_inline + over_budget_inline + all_files, oversized_inline + over_budget_inline + [x.file for x in unpersisted_inline] ) - # Covers the unconsumed-attachment recovery read, the S3 fetch behind - # `resolve_files`, and the inline/tabular/oversized partitioning. Expected - # to be ~0 on a turn with no attachments; if it is not, the hypothesis in - # docs/specs/turn-latency-preamble.md is wrong about where the time is. - prelude.mark("preamble.files") + return TurnAttachments( + recovered_upload_ids=recovered_upload_ids, + files_to_send=files_to_send, + diverted_tabular=diverted_tabular, + diverted_presentations=diverted_presentations, + oversized_inline=oversized_inline, + over_budget_inline=over_budget_inline, + dropped_over_count_names=dropped_over_count_names, + dropped_over_count_total=dropped_over_count_total, + turn_has_document=turn_has_document, + marker_names=attachment_marker_names, + unpersisted_inline=unpersisted_inline, + ) + + +async def _prepare_session_state( + input_data: InvocationRequest, + user_id: str, + session_meta, + *, + is_resume: bool, + is_continuation: bool, +) -> Tuple[bool, Optional[str]]: + """Map stage B5 (``preamble.session_state``). + + Pre-creates the session row on a first turn, clears the markers a previous + turn may have left (paused turn, pending interrupts, truncation, + interruption) and records the write-ahead attachment marker. Every read is + answered from ``session_meta`` (PR-2 of the preamble spec); only a marker + that actually exists costs a write. + + Returns ``(is_new_session, interrupted_turn_reason)``: whether this is the + session's first turn (it drives title generation), and the settled reason + the previous turn was interrupted, if it was, for the note prepended to + this turn's prompt. + """ + # Pre-create session metadata so OAuth interrupts and other state can + # attach to the session row from turn one. Best-effort; on failure the + # post-stream lazy-create in StreamCoordinator still covers it. + # + # Also clear any stale paused_turn snapshot at the start of a fresh turn. + # If the user abandoned a paused turn and started a new one, the prior + # snapshot is no longer authorized — letting it survive would let a + # later (mistaken) resume request pick up against a turn the user + # already moved past. + is_new_session = False + if not is_resume and not is_continuation: + is_new_session = await ensure_session_metadata_exists( + input_data.session_id, user_id, snapshot=session_meta + ) + try: + from apis.shared.sessions.metadata import ( + clear_paused_turn, + clear_pending_interrupts, + ) + await clear_paused_turn(input_data.session_id, user_id, snapshot=session_meta) + # The snapshot's breadcrumbs go with it. They are the other half of + # the same record, and a breadcrumb that outlives the snapshot + # re-renders a prompt the user can no longer answer: the resume + # route 400s on an interrupt id the rebuilt agent never saw. Safe + # here specifically because this runs at the *head* of a non-resume + # turn — any breadcrumb this turn goes on to write lands later, on + # its own `done` event. + await clear_pending_interrupts( + input_data.session_id, user_id, snapshot=session_meta + ) + except Exception as e: + logger.error("Failed to clear stale paused_turn on new turn: %s", e, exc_info=True) + + # Invalidate any prior max_tokens "Continue" marker on every new model + # turn that isn't an interrupt-resume — both a fresh turn and a + # continuation supersede it. If a continuation itself re-truncates, the + # stream_coordinator intercept re-sets the marker. + interrupted_turn_reason: Optional[str] = None + if not is_resume: + try: + from apis.shared.sessions.metadata import clear_truncated_turn + await clear_truncated_turn(input_data.session_id, user_id, snapshot=session_meta) + except Exception as e: + logger.error("Failed to clear stale truncated_turn on new turn: %s", e, exc_info=True) + + # Same lifecycle for the interrupted-turn marker: any new non-resume + # turn supersedes a prior interruption, so a stale marker can't + # resurrect the "response interrupted" state against a turn the user + # has moved past. The pop returns the settled reason (user_stopped + # beats connection_lost via write precedence) so this same read+write + # also drives the one-turn interruption note prepended to the prompt + # in the stream generator below. + try: + from apis.shared.sessions.metadata import clear_interrupted_turn + interrupted_turn_reason = await clear_interrupted_turn( + input_data.session_id, user_id, snapshot=session_meta + ) + except Exception as e: + logger.error("Failed to clear stale interrupted_turn on new turn: %s", e, exc_info=True) + + # Write-ahead half of the unconsumed-attachment pair (the pop lives + # up with the file-resolution block). Recorded BEFORE the model call + # so it survives every way a turn can die — including the ones no + # error handler sees, like the AgentCore data plane dropping the + # stream. Cleared by StreamCoordinator the moment the turn produces + # assistant content, so a turn that succeeds never leaves a marker + # behind. Runs after ensure_session_metadata_exists above, which is + # what guarantees the session row exists to update. + if input_data.file_upload_ids: + try: + from apis.shared.sessions.metadata import set_pending_attachments + await set_pending_attachments( + input_data.session_id, user_id, input_data.file_upload_ids + ) + except Exception as e: + logger.error("Failed to record pending attachments: %s", e, exc_info=True) + + return is_new_session, interrupted_turn_reason + + +@dataclass +class TurnQuota: + """Map stage B6 (``preamble.quota``): what the quota check decided.""" + + warning_event: Any = None + session_notice_event: Any = None + exceeded_event: Any = None + + +async def _check_turn_quota( + current_user: User, + input_data: InvocationRequest, + session_meta, + *, + is_resume: bool, + is_continuation: bool, +) -> TurnQuota: + """Map stage B6 (``preamble.quota``). Fails open: a quota-service error + lets the turn run rather than blocking it.""" + # Check quota if enforcement is enabled + quota_warning_event = None + quota_session_notice_event = None + quota_exceeded_event = None + if is_quota_enforcement_enabled() and not is_resume and not is_continuation: + try: + quota_checker = get_quota_checker() + # Hand the quota checker the session cost we already read (PR-2b). + # ONLY when the row actually carries `totalCost`: absent means a + # legacy row that `get_session_metadata` still needs to backfill, + # and `None` routes the checker back to that read. Passing 0.0 for + # a missing attribute would silence the notice on exactly the + # long-lived conversations it exists to catch. + session_total_cost = None + if session_meta.row is not None and "totalCost" in session_meta.row: + try: + session_total_cost = float(session_meta.row["totalCost"]) + except (TypeError, ValueError): + session_total_cost = None + quota_result = await quota_checker.check_quota( + user=current_user, + session_id=input_data.session_id, + session_total_cost=session_total_cost, + ) + + if not quota_result.allowed: + # Quota blocked - stream as SSE instead of 429 for better UX + logger.warning("Quota blocked for user") + if quota_result.tier is None: + # No quota tier configured for this user + quota_exceeded_event = build_no_quota_configured_event(quota_result) + else: + # Quota limit exceeded + quota_exceeded_event = build_quota_exceeded_event(quota_result) + else: + # Check for warning level + quota_warning_event = build_quota_warning_event(quota_result) + if quota_warning_event: + logger.info("Quota warning for user") + + # Independent of the per-user ladder: is THIS conversation + # eating the month? (#833 PR-5 — the incident session spent + # 90% of a user's quota while every per-user warning stayed + # quiet until the day the block landed.) + quota_session_notice_event = build_quota_session_notice_event(quota_result) + if quota_session_notice_event: + logger.info("Quota session notice for user") + + except Exception as e: + # Log error but don't block request - fail open for quota errors + logger.error("Error checking quota for user", exc_info=True) + + return TurnQuota( + warning_event=quota_warning_event, + session_notice_event=quota_session_notice_event, + exceeded_event=quota_exceeded_event, + ) + + +@dataclass +class TurnModel: + """Map stage B11: the model this turn runs on and the knobs resolved for it.""" + + model_id: Optional[str] + provider: Optional[str] + caching_enabled: Optional[bool] + inference_params: dict + mantle_api_mode: Optional[str] + mantle_region: Optional[str] + + +async def _resolve_turn_model( + input_data: InvocationRequest, + current_user: User, + user_id: str, + user_settings: dict, + agent_model_override, +) -> TurnModel: + """Map stage B11 (in ``tools``): which model, and with what settings. + + Precedence for the id: the Agent's governed ``modelConfig`` (already + access-checked against the invoker), else the request's, else the user's + saved default, else the catalog's default. The registry then supplies + caching, admin bounds and locks on the inference params, the Mantle + transport fields, and the provider when nothing else named one. + """ + # Build the canonical request inference-params dict. The frontend + # sends ``inference_params`` directly; legacy ``temperature`` / + # ``max_tokens`` fields are folded in for older clients and + # treated as defaults that lose to anything in ``inference_params``. + request_inference_params: dict = dict(input_data.inference_params or {}) + if input_data.temperature is not None: + request_inference_params.setdefault("temperature", input_data.temperature) + if input_data.max_tokens is not None: + request_inference_params.setdefault("max_tokens", input_data.max_tokens) + + # Resolve the user's persisted default when the request does + # not pin a model. Without this, a "no default selected" client + # always lands on the hardcoded factory default and the user's + # saved preference is silently ignored at chat time (#161). + effective_model_id = input_data.model_id + effective_provider = input_data.provider + if agent_model_override is not None: + # The Agent's governed modelConfig wins over the request / user-default + # chain. Already access-checked against the invoker in the resolver (R2), + # so the earlier request-only gate at the top doesn't leave a hole. + effective_model_id = agent_model_override.model_id + effective_provider = agent_model_override.provider or effective_provider + if not effective_model_id: + effective_model_id, effective_provider = await _resolve_fallback_model( + user_id, current_user, effective_provider, settings=user_settings + ) + + # Agent-authored params sit as defaults BENEATH explicit request params, + # then flow through _resolve_model_settings' admin bounds/locks like any + # other request params — an author can't smuggle out-of-bounds values. + if agent_model_override is not None and agent_model_override.params: + request_inference_params = {**agent_model_override.params, **request_inference_params} + + # Single registry lookup resolves caching + inference params + + # the Mantle endpoint path + provider, merging admin defaults with + # request overrides. + caching_enabled, inference_params, mantle_api_mode, mantle_region, registry_provider = await _resolve_model_settings( + model_id=effective_model_id, + explicit_caching_enabled=input_data.caching_enabled, + request_inference_params=request_inference_params, + ) + + # Recover the provider from the registry when neither the request nor + # the Agent's model binding carried one. Agent bindings persist only + # ``model_id`` (no provider), so without this a Mantle model like + # ``openai.gpt-5.4`` resolves to provider=None → Bedrock and blows up + # in ConverseStream with "invalid model identifier" — even though the + # same model works from the normal chat path, which always sends + # ``provider`` alongside ``model_id``. + if not effective_provider and registry_provider: + effective_provider = registry_provider + + if caching_enabled is False: + logger.info("Prompt caching disabled for model") + + return TurnModel( + model_id=effective_model_id, + provider=effective_provider, + caching_enabled=caching_enabled, + inference_params=inference_params, + mantle_api_mode=mantle_api_mode, + mantle_region=mantle_region, + ) + + +async def _resolve_effective_tools( + input_data: InvocationRequest, + current_user: User, + user_id: str, + agent_tools_override, + *, + turn_has_tabular: bool, +) -> Optional[list]: + """Map stage B11 (in ``tools``): the tool ids this turn actually carries. + + The Agent's bindings if it has any, else the request's picker; then the + spreadsheet auto-enable for a session holding a spreadsheet, then the + admin's always-on set and the platform's system tools. One value, so the + cache key, every builder, the attachment guidance and the paused-turn + snapshot all see the same list. + """ + # Get agent instance with user-specific configuration + # AgentCore Memory tracks preferences across sessions per user_id + # Supports multiple LLM providers: AWS Bedrock, OpenAI, and Google Gemini + # Use augmented message and assistant system prompt if assistant RAG was applied + + # Spreadsheet tools scoped to the assistant's document corpus, + # when an assistant is attached to this request. The frontend + # keeps the assistant id in the URL for the whole session's + # lifetime, so we can trust `input_data.rag_assistant_id` + # directly; no preferences fallback needed. + # An Agent's tool bindings replace the request's enabled_tools for this + # turn (D5, resolved per invoker above). None ⇒ no tool binding ⇒ the + # request drives the toolset exactly as today. Drives both the built-in + # extra tools (spreadsheet/artifact gate on specific ids) and get_agent. + effective_enabled_tools = ( + agent_tools_override.tool_ids + if agent_tools_override is not None + else input_data.enabled_tools + ) + # A session holding a spreadsheet gets the Spreadsheet Analysis + # tools whether or not the picker has them on, gated on the + # caller's RBAC grant. Applied to the *effective* list so it + # flows into the cache key, every builder below, the attachment + # guidance and the paused-turn snapshot as one value. Sticky + # across the session (see `_session_has_tabular`), so the key + # does not flip between the attach turn and the follow-up. + effective_enabled_tools = await _apply_attachment_tool_autoenable( + effective_enabled_tools, + current_user, + input_data.session_id, + user_id, + turn_has_tabular=turn_has_tabular, + ) + + # Tools an admin pinned are unioned in for users whose roles grant + # them, unless this Agent binds its own toolset (D4). Applied to + # the same *effective* list for the same reason as the line above: + # one value flows into the cache key, every builder, and the + # paused-turn snapshot. The set depends only on the catalog and the + # user's roles, so it is constant across a session and does not + # flip the key turn to turn. + effective_enabled_tools = await _apply_admin_always_on_tools( + effective_enabled_tools, + current_user, + agent_bound_tools=agent_tools_override is not None, + ) + + return effective_enabled_tools + + +@dataclass +class TurnTools: + """Map stage B11: the context-bound tools built for this turn. + + ``extra_tools`` is appended to the registry's filtered tools by + ``BaseAgent``. ``document_tools`` is the ``document_read`` subset, whose + presence is a cache-key element of its own. ``extra_tools_key_described`` + says whether every builder that fired closes over values the cache key + already carries (otherwise the agent is not cached). ``memory_binding_key`` + is what the memory tools close over, or None. + """ + + extra_tools: list + document_tools: list + extra_tools_key_described: bool + memory_binding_key: Optional[dict] + + +async def _build_turn_tools( + input_data: InvocationRequest, + current_user: User, + user_id: str, + effective_enabled_tools: Optional[list], + *, + agent_memory, + project_memory, + turn_has_document: bool, +) -> TurnTools: + """Map stage B11 (in ``tools``): every injected tool, and the key material + that describes them. Closures only — no IO except the ``document_read`` + gate, which is memoized per session once it answers yes.""" + extra_tools = _build_spreadsheet_tools( + enabled_tools=effective_enabled_tools, + assistant_id=input_data.rag_assistant_id, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_artifact_tools( + enabled_tools=effective_enabled_tools, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_word_document_tools( + enabled_tools=effective_enabled_tools, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_workspace_tools( + enabled_tools=effective_enabled_tools, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_excel_spreadsheet_tools( + enabled_tools=effective_enabled_tools, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_powerpoint_presentation_tools( + enabled_tools=effective_enabled_tools, + session_id=input_data.session_id, + user_id=user_id, + ) + _build_account_tools(effective_enabled_tools, current_user) + + memory_tools = _build_memory_tools( + agent_memory=agent_memory, + user_id=user_id, + user_email=current_user.email, + ) + # A project harness addresses its spaces by scope instead (2.4b). + if project_memory is not None: + from apis.inference_api.chat.project_memory import build_project_memory_tools + + memory_tools = build_project_memory_tools(project_memory, current_user) + extra_tools = extra_tools + memory_tools + + # document_read for any session that carries a readable attachment + # (this turn's uploads count). Gated on session state, not the + # picker; its presence goes into the cache key below rather than + # vetoing the cache, so an attachment session that could keep a + # warm agent still does. + document_tools = await _build_document_tools( + session_id=input_data.session_id, + user_id=user_id, + turn_has_document=turn_has_document, + ) + extra_tools = extra_tools + document_tools + + # Can this turn's agent be cached despite carrying injected tools? + # Only when every builder that fired closes over values the cache + # key already carries (session, user, enabled_tools). Derived from + # the same `effective_enabled_tools` that goes into the key below — + # passing the request's list here instead would let the predicate + # and the key disagree about which builders ran. + extra_tools_key_described = injected_tools_are_key_described( + enabled_tools=effective_enabled_tools, + ) + # Memory tools close over the resolved binding, so it is a cache-key + # element (Shared Projects 2.1). Only set when the tools were built, + # so the key and the toolset cannot disagree. + if project_memory is not None: + memory_binding_key = project_memory.binding_key() + elif memory_tools: + memory_binding_key = { + "spaceId": agent_memory.space_id, + "spaceName": agent_memory.space_name, + "access": agent_memory.access, + } + else: + memory_binding_key = None + + return TurnTools( + extra_tools=extra_tools, + document_tools=document_tools, + extra_tools_key_described=extra_tools_key_described, + memory_binding_key=memory_binding_key, + ) + + +def _validate_resume_interrupts(agent, input_data: InvocationRequest) -> None: + """Reject a resume that names interrupts the cached agent has not paused. + + Cache eviction, a process restart, or a forged request would otherwise be + silently accepted by Strands and drop the client's response. Raising up + front lets the client see a 400 and restart the turn cleanly. + """ + strands_agent = getattr(agent, "agent", None) + interrupt_state = getattr(strands_agent, "_interrupt_state", None) if strands_agent else None + known_ids: set[str] = set() + if interrupt_state and getattr(interrupt_state, "activated", False): + interrupts = getattr(interrupt_state, "interrupts", None) or {} + known_ids = set(interrupts.keys()) + submitted_ids = [entry.interruptId for entry in (input_data.interrupt_responses or [])] + unknown_ids = [iid for iid in submitted_ids if iid not in known_ids] + if unknown_ids: + logger.warning( + "Resume rejected: submitted interrupt ids not in paused state" + ) + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Unknown or expired interrupt ids; restart the turn.", + ) + + +async def _search_and_augment( + *, assistant_id: str, message: str, access: Any +) -> Tuple[Optional[list], str]: + """Search an assistant's knowledge base and augment ``message`` with it. + + Returns ``(context_chunks, augmented_message)``. Fail-open as it always + was: a failed search yields ``(None, message)`` and the turn runs without + retrieved context. Never raises, which also means a turn that exits before + awaiting it (a refusal after the search started) leaves no unretrieved + exception behind. + """ + from apis.shared.assistants.rag_service import ( + augment_prompt_with_context, + search_assistant_knowledgebase_with_cap, + ) + + try: + logger.info("Searching knowledge base for assistant...") + # The cap comes back with the results: both are decided from the one + # KB_Record read the search makes, rather than reading it again here. + context_chunks, cap = await search_assistant_knowledgebase_with_cap( + assistant_id=assistant_id, + query=message, + top_k=5, + access=access, + ) + logger.info(f"Knowledge base search returned {len(context_chunks) if context_chunks else 0} chunks") + if not context_chunks: + logger.info("No context chunks found for assistant - using original message without augmentation") + return context_chunks, message + for i, _chunk in enumerate(context_chunks): + logger.info(f"Chunk {i + 1} retrieved") + logger.info(f"Chunk {i + 1} metadata retrieved") + # Engine-aware cap (Requirement 3.2): managed gets 8,000 so + # reranking's top_k chunks actually reach the model; legacy keeps + # 2,000. See rag_service.resolve_context_cap / HANDOFF §5.40. + augmented = augment_prompt_with_context( + user_message=message, context_chunks=context_chunks, max_context_length=cap + ) + logger.info(f"Augmented message with {len(context_chunks)} context chunks") + logger.info("Augmented message preview available") + return context_chunks, augmented + except Exception as e: + logger.error("Error searching assistant knowledge base", exc_info=True) + logger.error(f"Exception type: {type(e).__name__}") + # Continue without RAG context rather than failing + return None, message + + +def _build_citations(assistant, context_chunks, rag_assistant_id: Optional[str]) -> list: + """Citations to persist and to stream ahead of the answer.""" + # #111: when the agent's ``show_citations`` flag is off, suppress citations + # entirely — leaving this list empty is a single choke point that turns off all + # three downstream consumers at once: the ``event: citation`` SSE below, the + # ``citations=...`` persisted on the stored message, and the copy handed to + # ``agent.stream_async``. RAG retrieval and prompt augmentation above are + # deliberately untouched: the model still receives the context chunks, the user + # just is not shown (or able to download) the sources. + show_citations = getattr(assistant, "show_citations", True) + citations_for_storage = [] + if context_chunks and show_citations: + for chunk in context_chunks: + citations_for_storage.append( + { + "assistantId": rag_assistant_id, + "documentId": chunk.get("metadata", {}).get("document_id", ""), + # Managed KBs carry the filename under ``filename`` (set at ingest, + # managed_backend.py); legacy S3-Vectors used ``source``. Read + # managed first, fall back to legacy, then the placeholder — before + # this, every managed-KB citation rendered "Unknown Source". + "fileName": ( + chunk.get("metadata", {}).get("filename") + or chunk.get("metadata", {}).get("source") + or "Unknown Source" + ), + "text": chunk.get("text", "")[:500], # Limit excerpt length + } + ) + return citations_for_storage + + +@router.post("/invocations") +async def invocations(request: InvocationRequest, current_user: User = Depends(get_current_user_trusted)): + """The AgentCore Runtime invocation endpoint: one conversation turn. + + Runs the sequence docs/specs/turn-path-ttft.md maps, in this order. The + marks are the `turn_prelude` stage names; everything up to `stream_setup` + happens before the response opens, with no channel to the client. + + 1. `preamble.ownership` — one read of the session's META row; 404 if it is + another user's. + 2. `preamble.skills` — the turn's effective skills (opt-in; ~0 otherwise). + 3. *(no mark)* — model retirement; the two MCP App dispatches + (`app_tool_call`, `app_context_update`) return here, JSON not SSE. + 4. `preamble.files` — `_resolve_turn_attachments`. + 5. `preamble.session_state` — `_prepare_session_state`; the title task on a + first turn. + 6. `preamble.quota` — `_check_turn_quota`; a block streams as an + assistant message. + 7. *(no mark)* — retired-model denial, model access, user settings. + 8. `rag` — Agent turns: access, version, bindings, the + knowledge-base search, prompt composition, memory, binding persistence. + 9. *(no mark)* — custom prompt, personal instructions, the + single-flight lease. + 10. `tools` — resume: rebuild from the paused snapshot; else + `_resolve_turn_model`, `_resolve_effective_tools`, `_build_turn_tools`. + 11. `agent_build.*` — `get_agent`, deferred into the stream and + narrated (`preparing` / `prepared`) unless the preparing phase is off. + 12. `stream_setup` — citations, the generators, the response. + + Then `_guarded_stream` opens the response, builds the agent, emits + `turn_prelude`, starts the lease heartbeat and yields + `stream_with_quota_warning`, which composes the final message and streams + the agent. The SSE contract is in CLAUDE.md § SSE Event Types. + """ + input_data = request + user_id = current_user.user_id + auth_token = current_user.raw_token + + # Where the pre-stream time goes. Everything between here and the + # `StreamingResponse` return happens with NO channel open to the client — + # measured at 3.75s on a warm turn — so this is the only way to see which + # stage owns it. Pure timing: nothing reaches the model. + # See `turn_timing.py` and docs/specs/agent-state-feedback.md. + prelude = TurnPrelude() + # Whether this turn's agent is built inside the stream (PR-3). Recorded on + # the `turn_prelude` line so the two shapes stay distinguishable in the + # logs once the flag has been on for a while. + deferred_build = False + + # Refuse a turn against a session id another user already owns. + # + # Session ids travel in shareable URLs (`/s/{sessionId}`). Opening someone + # else's link 404s on the metadata read, but the SPA then treats the + # session as new and lets the user send — which used to fork the id: a + # SECOND metadata row under the requester, on the same session, invisible + # to both parties. In prod on 2026-08-31 that also left the original + # owner's session resolving non-deterministically between the two rows. + # + # Not a confidentiality fix — conversation content is keyed by actor id in + # AgentCore Memory, so the second user only ever saw an empty thread. This + # stops the id from being forked at all. 404 rather than 403 so the + # response says nothing about whether the session exists, matching what + # `GET /sessions/{id}/metadata` already returns for the same case. + # + # ONE read of the session's META row, shared by everything in the preamble + # that used to fetch it again (PR-2, docs/specs/turn-latency-preamble.md). + # Measured on dev: eight separate reads of this item cost ~445ms of a + # ~455ms stage. That was later traced (PR-3 of the same spec) to boto3 + # client construction on a throttled container CPU — ~47ms per read site — + # not to the DynamoDB round trip, which is ~6ms. Both fixes stand: one read, + # on a cached client. + # + # Deliberately explicit rather than a per-request memo inside + # `_get_session_by_gsi`: CLAUDE.md's "never cache session state" rule has + # been paid for twice (#741, #751), and a snapshot callers opt into cannot + # leak into one that needs a fresh read. + session_meta = await load_session_meta(input_data.session_id, user_id) + if session_meta.owned_by_other: + logger.warning( + "Rejected invocation for session %s — owned by a different user", + _sanitize_log(input_data.session_id), + ) + raise HTTPException(status_code=404, detail="Session not found") + # First of the preamble's five sub-stages (docs/specs/turn-latency-preamble.md). + # The coarse `preamble` number survives as `groups.preamble` in the emitted + # line, so the four-turn baseline in the agent-state-feedback spec stays + # comparable across this split. + prelude.mark("preamble.ownership") + # Resume requests reuse the cached agent and its paused interrupt state; + # they bypass quota, file resolution, and RAG augmentation because those + # already ran on the original turn that got paused. + is_resume = bool(input_data.interrupt_responses) + # Resolve the effective agent type: the client's explicit choice, else the + # compiled-in default ("chat"). Used for the skill resolution below and the + # non-resume get_agent calls (resume reuses the snapshot's type). An Agent + # that binds skills is coerced to "skill" later by the agent-binding + # resolver. + effective_agent_type = input_data.agent_type or DEFAULT_AGENT_TYPE + # Skills feature deferred for this environment: neutralize the legacy + # "skill" agent type, which is a ChatAgent alias since v2 PR-2. Voice and + # other agent types pass through untouched. + if not skills_enabled() and effective_agent_type == "skill": + effective_agent_type = "chat" + # Resolve the user's *effective* skills once for the whole request: the + # accessible set (catalog ∪ own), narrowed by the client's per-turn + # enabled_skills selection. Threaded into every get_agent call below so they + # share one skills_hash cache key (otherwise the app-tool-call / resume paths + # would miss the main turn's cached agent). + # + # Skills v2: this is no longer gated on agent_type == "skill". Skills are a + # plain-chat capability now — the picker in model settings sends + # enabled_skills on an ordinary turn and ChatAgent mounts the AgentSkills + # plugin. The opt-in default (D6) is what keeps this cheap: an absent or + # empty selection short-circuits to [] without touching RBAC or the skill + # table, so every turn that doesn't ask for skills costs exactly what it did + # before. An Agent's skill bindings override this further down. + effective_skill_ids = None + if skills_enabled() and input_data.enabled_skills: + effective_skill_ids = _apply_enabled_skills_filter( + await _resolve_accessible_skill_ids(current_user), + input_data.enabled_skills, + ) + # Near-zero on a turn that selects no skills — the opt-in default (D6) + # short-circuits before touching RBAC or the skill table. A non-trivial + # number here means the RBAC cache missed or the owner-index query is slow. + prelude.mark("preamble.skills") + # A "Continue" after a max_tokens truncation. Like resume, it bypasses + # quota / RAG / file resolution and does NOT clear the turn state; unlike + # resume there is no interrupt to validate — the agent is rebuilt from the + # resent params and re-entered with an empty prompt (assistant-prefill). + is_continuation = bool(input_data.continue_truncated) + # Marketplace D11: the Agent was `@`-mentioned in the composer, so it runs + # this turn only — it does not bind the conversation. Only meaningful + # alongside `rag_assistant_id`; on its own it does nothing. + is_agent_mention = bool(input_data.agent_mention) and bool(input_data.rag_assistant_id) + logger.info( + "Invocation request received (resume=%s, continue_truncated=%s, agent_mention=%s)" + % (is_resume, is_continuation, is_agent_mention) + ) + logger.info("Message received") + + # Model retirement (docs/specs/model-retirement.md §7). Resolved before anything + # builds an agent from ``input_data.model_id``, so the App tool-call / context / + # continuation paths land in the same agent-cache slot as the turn itself. A + # redirect swaps the provider as well: the request's described the retired + # model, and a successor on another transport misroutes with it. A denial is + # streamed at the access check below, once there is a turn to answer. + retired_model_denial: Optional[str] = None + requested_model = await resolve_effective_model(input_data.model_id) + if requested_model is not None: + if requested_model.denied: + retired_model_denial = retired_model_message(requested_model.retired) + elif requested_model.redirected: + input_data.model_id = requested_model.model_id + input_data.provider = requested_model.provider + + # App-initiated tools/call (MCP Apps PR #5). Like resume/continuation it + # bypasses quota / RAG / file resolution / title — there is no model + # turn. We rebuild the conversation agent (so the MCP client session + + # auth are wired exactly as for a model-driven call), dispatch the one + # named tool, publish synthesized tool_use/tool_result into the thread + # via the per-session broker, and return the CallToolResult as JSON for + # app-api to relay back to the iframe. Inert behind the host flag (the + # UIToolCatalog is empty, so dispatch rejects every call as not + # app-visible). + if input_data.app_tool_call is not None: + atc = input_data.app_tool_call + try: + agent = await _agent_for_app_dispatch( + input_data, + current_user, + user_id, + auth_token, + agent_type=effective_agent_type, + accessible_skill_ids=effective_skill_ids, + ) + payload = await dispatch_app_tool_call( + agent, + session_id=input_data.session_id, + user_id=user_id, + tool_use_id=atc.tool_use_id, + tool_name=atc.tool_name, + arguments=atc.arguments, + ) + return JSONResponse(payload) + except AppToolCallError as e: + # 200 + envelope, not `status_code=e.code`: AgentCore Runtime + # rewrites any non-2xx to a generic 424 and discards the + # message, so a deliberate 409 ("connect the account") reached + # the SPA as "check your CloudWatch logs". app-api restores the + # real status. See `mcp_apps.error_envelope`. + return app_tool_error_response(e.message, e.code) + except HTTPException: + raise + except Exception: + logger.error("app tools/call invocation failed", exc_info=True) + return JSONResponse({"error": "Internal error"}, status_code=500) - # Pre-create session metadata so OAuth interrupts and other state can - # attach to the session row from turn one. Best-effort; on failure the - # post-stream lazy-create in StreamCoordinator still covers it. - # - # Also clear any stale paused_turn snapshot at the start of a fresh turn. - # If the user abandoned a paused turn and started a new one, the prior - # snapshot is no longer authorized — letting it survive would let a - # later (mistaken) resume request pick up against a turn the user - # already moved past. - is_new_session = False - if not is_resume and not is_continuation: - is_new_session = await ensure_session_metadata_exists( - input_data.session_id, user_id, snapshot=session_meta - ) + # App-pushed model context (MCP Apps PR #6, `ui/update-model-context`). + # Like app_tool_call it bypasses quota / RAG / file resolution / title + # and runs NO model turn — we rebuild the conversation agent (so the + # same cached `agent.state` is reused) and stash the payload under + # `mcp_apps.context[resource_uri]`. The next real user turn merges and + # clears it. Inert behind the host flag (no live App ever calls this). + if input_data.app_context_update is not None: + acu = input_data.app_context_update try: - from apis.shared.sessions.metadata import ( - clear_paused_turn, - clear_pending_interrupts, + agent = await _agent_for_app_dispatch( + input_data, + current_user, + user_id, + auth_token, + agent_type=effective_agent_type, + accessible_skill_ids=effective_skill_ids, ) - await clear_paused_turn(input_data.session_id, user_id, snapshot=session_meta) - # The snapshot's breadcrumbs go with it. They are the other half of - # the same record, and a breadcrumb that outlives the snapshot - # re-renders a prompt the user can no longer answer: the resume - # route 400s on an interrupt id the rebuilt agent never saw. Safe - # here specifically because this runs at the *head* of a non-resume - # turn — any breadcrumb this turn goes on to write lands later, on - # its own `done` event. - await clear_pending_interrupts( - input_data.session_id, user_id, snapshot=session_meta + payload = dispatch_app_context_update( + agent, + resource_uri=acu.resource_uri, + content=acu.content, + structured_content=acu.structured_content, ) - except Exception as e: - logger.error("Failed to clear stale paused_turn on new turn: %s", e, exc_info=True) + return JSONResponse(payload) + except AppContextUpdateError as e: + # Same AgentCore flattening as the app_tool_call path above. + return app_tool_error_response(e.message, e.code) + except HTTPException: + raise + except Exception: + logger.error("app context update invocation failed", exc_info=True) + return JSONResponse({"error": "Internal error"}, status_code=500) - # Invalidate any prior max_tokens "Continue" marker on every new model - # turn that isn't an interrupt-resume — both a fresh turn and a - # continuation supersede it. If a continuation itself re-truncates, the - # stream_coordinator intercept re-sets the marker. - interrupted_turn_reason: Optional[str] = None - if not is_resume: - try: - from apis.shared.sessions.metadata import clear_truncated_turn - await clear_truncated_turn(input_data.session_id, user_id, snapshot=session_meta) - except Exception as e: - logger.error("Failed to clear stale truncated_turn on new turn: %s", e, exc_info=True) - # Same lifecycle for the interrupted-turn marker: any new non-resume - # turn supersedes a prior interruption, so a stale marker can't - # resurrect the "response interrupted" state against a turn the user - # has moved past. The pop returns the settled reason (user_stopped - # beats connection_lost via write precedence) so this same read+write - # also drives the one-turn interruption note prepended to the prompt - # in the stream generator below. - try: - from apis.shared.sessions.metadata import clear_interrupted_turn - interrupted_turn_reason = await clear_interrupted_turn( - input_data.session_id, user_id, snapshot=session_meta - ) - except Exception as e: - logger.error("Failed to clear stale interrupted_turn on new turn: %s", e, exc_info=True) + if input_data.enabled_tools: + logger.info(f"Enabled tools ({len(input_data.enabled_tools)})") - # Write-ahead half of the unconsumed-attachment pair (the pop lives - # up with the file-resolution block). Recorded BEFORE the model call - # so it survives every way a turn can die — including the ones no - # error handler sees, like the AgentCore data plane dropping the - # stream. Cleared by StreamCoordinator the moment the turn produces - # assistant content, so a turn that succeeds never leaves a marker - # behind. Runs after ensure_session_metadata_exists above, which is - # what guarantees the session row exists to update. - if input_data.file_upload_ids: - try: - from apis.shared.sessions.metadata import set_pending_attachments - await set_pending_attachments( - input_data.session_id, user_id, input_data.file_upload_ids - ) - except Exception as e: - logger.error("Failed to record pending attachments: %s", e, exc_info=True) + # Attachments: recover the previous turn's unanswered uploads, fetch this + # turn's from S3, dedupe, partition (inline / spreadsheet / deck / + # oversized) and hold the inline set to one message's byte budget. + attachments = await _resolve_turn_attachments( + input_data, + user_id, + session_meta, + is_resume=is_resume, + is_continuation=is_continuation, + ) + + # Covers the unconsumed-attachment recovery read (answered from the + # snapshot), the S3 fetch behind `resolve_files`, and the partitioning. + # ~1ms on a turn with no attachments; otherwise the S3 GETs are the cost. + prelude.mark("preamble.files") + + # Session state: the row pre-create, the four stale-marker clears and the + # write-ahead attachment marker, all answered from `session_meta`. + is_new_session, interrupted_turn_reason = await _prepare_session_state( + input_data, + user_id, + session_meta, + is_resume=is_resume, + is_continuation=is_continuation, + ) # First turn → kick off title generation concurrently with the stream. # Runs as a background task so it doesn't add latency to TTFT. The @@ -2559,84 +3268,38 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g ) ) - # The prime suspect: five of the preamble's eight reads of the session's - # META row live in this stage (pre-create plus the four stale-marker - # clears), each on its own round trip, and four of them short-circuit - # without writing anything. The title task is inside the boundary because - # spawning it is first-turn session state; it is an `asyncio.create_task`, - # so it contributes nothing to the number. + # Five reads of the META row used to live here (pre-create plus the four + # stale-marker clears), each on its own round trip; since PR-2 all five are + # answered from `session_meta`, so a non-trivial number means a marker + # write or a first turn's pre-create. The title task is inside the boundary + # because spawning it is first-turn session state; it is an + # `asyncio.create_task`, so it contributes nothing to the number. prelude.mark("preamble.session_state") - # Check quota if enforcement is enabled - quota_warning_event = None - quota_session_notice_event = None - quota_exceeded_event = None - if is_quota_enforcement_enabled() and not is_resume and not is_continuation: - try: - quota_checker = get_quota_checker() - # Hand the quota checker the session cost we already read (PR-2b). - # ONLY when the row actually carries `totalCost`: absent means a - # legacy row that `get_session_metadata` still needs to backfill, - # and `None` routes the checker back to that read. Passing 0.0 for - # a missing attribute would silence the notice on exactly the - # long-lived conversations it exists to catch. - session_total_cost = None - if session_meta.row is not None and "totalCost" in session_meta.row: - try: - session_total_cost = float(session_meta.row["totalCost"]) - except (TypeError, ValueError): - session_total_cost = None - quota_result = await quota_checker.check_quota( - user=current_user, - session_id=input_data.session_id, - session_total_cost=session_total_cost, - ) - - if not quota_result.allowed: - # Quota blocked - stream as SSE instead of 429 for better UX - logger.warning("Quota blocked for user") - if quota_result.tier is None: - # No quota tier configured for this user - quota_exceeded_event = build_no_quota_configured_event(quota_result) - else: - # Quota limit exceeded - quota_exceeded_event = build_quota_exceeded_event(quota_result) - else: - # Check for warning level - quota_warning_event = build_quota_warning_event(quota_result) - if quota_warning_event: - logger.info("Quota warning for user") - - # Independent of the per-user ladder: is THIS conversation - # eating the month? (#833 PR-5 — the incident session spent - # 90% of a user's quota while every per-user warning stayed - # quiet until the day the block landed.) - quota_session_notice_event = build_quota_session_notice_event(quota_result) - if quota_session_notice_event: - logger.info("Quota session notice for user") - - except Exception as e: - # Log error but don't block request - fail open for quota errors - logger.error("Error checking quota for user", exc_info=True) + quota = await _check_turn_quota( + current_user, + input_data, + session_meta, + is_resume=is_resume, + is_continuation=is_continuation, + ) + quota_warning_event = quota.warning_event + quota_session_notice_event = quota.session_notice_event + quota_exceeded_event = quota.exceeded_event - # The quota round trip: a cached tier resolve, a cached O(1) cost-summary - # GetItem, and — the uncached one — the per-session notice, which reads the - # META row for the eighth time this turn. + # The quota check: a cached tier resolve, a cached O(1) cost-summary + # GetItem, and the per-session notice, answered from the snapshot's + # `totalCost` since PR-2b (or by its own read on a legacy row without one). prelude.mark("preamble.quota") # If quota exceeded, stream the quota exceeded message instead of agent response if quota_exceeded_event: - return StreamingResponse( - stream_conversational_message( - message=quota_exceeded_event.message, - stop_reason="quota_exceeded", - metadata_event=quota_exceeded_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, + return _refuse_turn( + input_data, + user_id, + message=quota_exceeded_event.message, + stop_reason="quota_exceeded", + metadata_event=quota_exceeded_event, ) # A retired model with no successor is denied for everyone, wildcard holders @@ -2644,21 +3307,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # resume: that turn finishes on its paused snapshot's model, whatever the # request carries. if retired_model_denial and not is_resume: - retired_event = ConversationalErrorEvent( - code=ErrorCode.FORBIDDEN, message=retired_model_denial, recoverable=False - ) - return StreamingResponse( - stream_conversational_message( - message=retired_model_denial, - stop_reason="error", - metadata_event=retired_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, - ) + return _forbidden_turn(input_data, user_id, retired_model_denial) # Check model access if a specific model_id is requested if input_data.model_id: @@ -2674,6 +3323,10 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g assistant = None context_chunks = None augmented_message = input_data.message + # The knowledge-base search, started where it used to be awaited and + # awaited after the agent build instead (turn-path spec §5 P3b). None on a + # plain turn, a continuation, or once its result has been applied. + kb_search: Optional["asyncio.Task[Tuple[Optional[list], str]]"] = None system_prompt = input_data.system_prompt # Start with provided system prompt # One settings read per turn: the default model and personal instructions both # come from it. A preview is an author testing an agent, so it gets none of theirs. @@ -2747,11 +3400,6 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if input_data.rag_assistant_id and not is_resume: # Local imports to avoid circular dependency from apis.shared.assistants.kb_access import granted - from apis.shared.assistants.rag_service import ( - augment_prompt_with_context, - resolve_context_cap, - search_assistant_knowledgebase_with_formatting, - ) from apis.shared.assistants.service import ( get_assistant_with_access_check, mark_share_as_interacted, @@ -2917,22 +3565,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g from apis.shared.assistants.service import is_disabled_project_harness if await is_disabled_project_harness(input_data.rag_assistant_id): - refusal = PROJECTS_DISABLED_MESSAGE - refused_event = ConversationalErrorEvent( - code=ErrorCode.FORBIDDEN, message=refusal, recoverable=False - ) - return StreamingResponse( - stream_conversational_message( - message=refusal, - stop_reason="error", - metadata_event=refused_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, - ) + return _forbidden_turn(input_data, user_id, PROJECTS_DISABLED_MESSAGE) # Check if assistant exists at all to provide better error message from apis.shared.assistants.service import assistant_exists @@ -3035,21 +3668,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if runs_project_harness: refusal, turn_project = await _project_turn_gate(assistant.project_id) if refusal: - refused_event = ConversationalErrorEvent( - code=ErrorCode.FORBIDDEN, message=refusal, recoverable=False - ) - return StreamingResponse( - stream_conversational_message( - message=refusal, - stop_reason="error", - metadata_event=refused_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, - ) + return _forbidden_turn(input_data, user_id, refusal) turn_project_id = assistant.project_id # Shared Projects 2.4b: the project's memory spaces, read now and awaited at # prompt assembly (5b), so the reads overlap binding resolution and the @@ -3083,59 +3702,28 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g project_id=turn_project_id, ) except AgentBindingBlockedError as block: - blocked_event = ConversationalErrorEvent( - code=ErrorCode.FORBIDDEN, message=block.message, recoverable=False - ) - return StreamingResponse( - stream_conversational_message( - message=block.message, - stop_reason="error", - metadata_event=blocked_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, - ) + return _forbidden_turn(input_data, user_id, block.message) # Skipped on a continuation: the turn carries an empty message, so a # knowledge-base search would spend a query on "" and augment nothing. The # context the original turn retrieved is already in the history being continued. if not is_continuation: - # 3. Search assistant knowledge base + # 3–4. Search the assistant's knowledge base and augment the message. + # Started here, awaited after the agent build: the build reads + # neither the chunks nor the augmented message — only the stream + # does, for the citation frames and `final_message` — so the two + # can overlap (turn-path spec §5 P3b). Off, it is awaited right here. logger.info("Starting knowledge base search for assistant...") - try: - logger.info("Searching knowledge base for assistant...") - context_chunks = await search_assistant_knowledgebase_with_formatting( + kb_search = asyncio.create_task( + _search_and_augment( assistant_id=input_data.rag_assistant_id, - query=input_data.message, - top_k=5, + message=input_data.message, access=granted(input_data.rag_assistant_id, user_id, assistant_permission), ) - logger.info(f"Knowledge base search returned {len(context_chunks) if context_chunks else 0} chunks") - if context_chunks: - for i, chunk in enumerate(context_chunks): - logger.info(f"Chunk {i + 1} retrieved") - logger.info(f"Chunk {i + 1} metadata retrieved") - - # 4. Augment message with context - if context_chunks: - # Engine-aware cap (Requirement 3.2): managed gets 8,000 so - # reranking's top_k chunks actually reach the model; legacy keeps - # 2,000. See rag_service.resolve_context_cap / HANDOFF §5.40. - cap = resolve_context_cap(input_data.rag_assistant_id) - augmented_message = augment_prompt_with_context(user_message=input_data.message, context_chunks=context_chunks, max_context_length=cap) - logger.info( - f"Augmented message with {len(context_chunks)} context chunks" - ) - logger.info("Augmented message preview available") - else: - logger.info("No context chunks found for assistant - using original message without augmentation") - except Exception as e: - logger.error("Error searching assistant knowledge base", exc_info=True) - logger.error(f"Exception type: {type(e).__name__}") - # Continue without RAG context rather than failing + ) + if not kb_search_ahead_enabled(): + context_chunks, augmented_message = await kb_search + kb_search = None # 5. Append assistant's instructions to the base system prompt (don't replace) # For preview sessions, prefer the system_prompt from the request (live form edits) @@ -3499,106 +4087,20 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # snapshot's toolset, the same source the resume `get_agent` used. effective_enabled_tools = snapshot.enabled_tools else: - # Build the canonical request inference-params dict. The frontend - # sends ``inference_params`` directly; legacy ``temperature`` / - # ``max_tokens`` fields are folded in for older clients and - # treated as defaults that lose to anything in ``inference_params``. - request_inference_params: dict = dict(input_data.inference_params or {}) - if input_data.temperature is not None: - request_inference_params.setdefault("temperature", input_data.temperature) - if input_data.max_tokens is not None: - request_inference_params.setdefault("max_tokens", input_data.max_tokens) - - # Resolve the user's persisted default when the request does - # not pin a model. Without this, a "no default selected" client - # always lands on the hardcoded factory default and the user's - # saved preference is silently ignored at chat time (#161). - effective_model_id = input_data.model_id - effective_provider = input_data.provider - if agent_model_override is not None: - # The Agent's governed modelConfig wins over the request / user-default - # chain. Already access-checked against the invoker in the resolver (R2), - # so the earlier request-only gate at the top doesn't leave a hole. - effective_model_id = agent_model_override.model_id - effective_provider = agent_model_override.provider or effective_provider - if not effective_model_id: - effective_model_id, effective_provider = await _resolve_fallback_model( - user_id, current_user, effective_provider, settings=user_settings - ) - - # Agent-authored params sit as defaults BENEATH explicit request params, - # then flow through _resolve_model_settings' admin bounds/locks like any - # other request params — an author can't smuggle out-of-bounds values. - if agent_model_override is not None and agent_model_override.params: - request_inference_params = {**agent_model_override.params, **request_inference_params} - - # Single registry lookup resolves caching + inference params + - # the Mantle endpoint path + provider, merging admin defaults with - # request overrides. - caching_enabled, inference_params, mantle_api_mode, mantle_region, registry_provider = await _resolve_model_settings( - model_id=effective_model_id, - explicit_caching_enabled=input_data.caching_enabled, - request_inference_params=request_inference_params, + # The model: the Agent's binding, else the request's, else the + # user's default, else the catalog's; then the registry's settings. + turn_model = await _resolve_turn_model( + input_data, current_user, user_id, user_settings, agent_model_override ) - # Recover the provider from the registry when neither the request nor - # the Agent's model binding carried one. Agent bindings persist only - # ``model_id`` (no provider), so without this a Mantle model like - # ``openai.gpt-5.4`` resolves to provider=None → Bedrock and blows up - # in ConverseStream with "invalid model identifier" — even though the - # same model works from the normal chat path, which always sends - # ``provider`` alongside ``model_id``. - if not effective_provider and registry_provider: - effective_provider = registry_provider - - if caching_enabled is False: - logger.info("Prompt caching disabled for model") - - # Get agent instance with user-specific configuration - # AgentCore Memory tracks preferences across sessions per user_id - # Supports multiple LLM providers: AWS Bedrock, OpenAI, and Google Gemini - # Use augmented message and assistant system prompt if assistant RAG was applied - - # Spreadsheet tools scoped to the assistant's document corpus, - # when an assistant is attached to this request. The frontend - # keeps the assistant id in the URL for the whole session's - # lifetime, so we can trust `input_data.rag_assistant_id` - # directly; no preferences fallback needed. - # An Agent's tool bindings replace the request's enabled_tools for this - # turn (D5, resolved per invoker above). None ⇒ no tool binding ⇒ the - # request drives the toolset exactly as today. Drives both the built-in - # extra tools (spreadsheet/artifact gate on specific ids) and get_agent. - effective_enabled_tools = ( - agent_tools_override.tool_ids - if agent_tools_override is not None - else input_data.enabled_tools - ) - # A session holding a spreadsheet gets the Spreadsheet Analysis - # tools whether or not the picker has them on, gated on the - # caller's RBAC grant. Applied to the *effective* list so it - # flows into the cache key, every builder below, the attachment - # guidance and the paused-turn snapshot as one value. Sticky - # across the session (see `_session_has_tabular`), so the key - # does not flip between the attach turn and the follow-up. - effective_enabled_tools = await _apply_attachment_tool_autoenable( - effective_enabled_tools, + # The tool ids this turn carries — one value for the cache key, + # every builder, the attachment guidance and the paused snapshot. + effective_enabled_tools = await _resolve_effective_tools( + input_data, current_user, - input_data.session_id, user_id, - turn_has_tabular=bool(diverted_tabular), - ) - - # Tools an admin pinned are unioned in for users whose roles grant - # them, unless this Agent binds its own toolset (D4). Applied to - # the same *effective* list for the same reason as the line above: - # one value flows into the cache key, every builder, and the - # paused-turn snapshot. The set depends only on the catalog and the - # user's roles, so it is constant across a session and does not - # flip the key turn to turn. - effective_enabled_tools = await _apply_admin_always_on_tools( - effective_enabled_tools, - current_user, - agent_bound_tools=agent_tools_override is not None, + agent_tools_override, + turn_has_tabular=bool(attachments.diverted_tabular), ) # An Agent's skill bindings replace the request's skills for this turn so @@ -3611,79 +4113,15 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g effective_agent_type = "skill" effective_skill_ids = agent_skills_override.skill_ids - extra_tools = _build_spreadsheet_tools( - enabled_tools=effective_enabled_tools, - assistant_id=input_data.rag_assistant_id, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_artifact_tools( - enabled_tools=effective_enabled_tools, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_word_document_tools( - enabled_tools=effective_enabled_tools, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_workspace_tools( - enabled_tools=effective_enabled_tools, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_excel_spreadsheet_tools( - enabled_tools=effective_enabled_tools, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_powerpoint_presentation_tools( - enabled_tools=effective_enabled_tools, - session_id=input_data.session_id, - user_id=user_id, - ) + _build_account_tools(effective_enabled_tools, current_user) - - memory_tools = _build_memory_tools( + turn_tools = await _build_turn_tools( + input_data, + current_user, + user_id, + effective_enabled_tools, agent_memory=agent_memory, - user_id=user_id, - user_email=current_user.email, - ) - # A project harness addresses its spaces by scope instead (2.4b). - if project_memory is not None: - from apis.inference_api.chat.project_memory import build_project_memory_tools - - memory_tools = build_project_memory_tools(project_memory, current_user) - extra_tools = extra_tools + memory_tools - - # document_read for any session that carries a readable attachment - # (this turn's uploads count). Gated on session state, not the - # picker; its presence goes into the cache key below rather than - # vetoing the cache, so an attachment session that could keep a - # warm agent still does. - document_tools = await _build_document_tools( - session_id=input_data.session_id, - user_id=user_id, - turn_has_document=turn_has_document, + project_memory=project_memory, + turn_has_document=attachments.turn_has_document, ) - extra_tools = extra_tools + document_tools - - # Can this turn's agent be cached despite carrying injected tools? - # Only when every builder that fired closes over values the cache - # key already carries (session, user, enabled_tools). Derived from - # the same `effective_enabled_tools` that goes into the key below — - # passing the request's list here instead would let the predicate - # and the key disagree about which builders ran. - extra_tools_key_described = injected_tools_are_key_described( - enabled_tools=effective_enabled_tools, - ) - # Memory tools close over the resolved binding, so it is a cache-key - # element (Shared Projects 2.1). Only set when the tools were built, - # so the key and the toolset cannot disagree. - if project_memory is not None: - memory_binding_key = project_memory.binding_key() - elif memory_tools: - memory_binding_key = { - "spaceId": agent_memory.space_id, - "spaceName": agent_memory.space_name, - "access": agent_memory.access, - } - else: - memory_binding_key = None # System-prompt assembly, the single-flight lease, skill # resolution and every tool builder (documents, attachments, @@ -3707,22 +4145,23 @@ async def _build_main_agent(): user_id=user_id, auth_token=auth_token, enabled_tools=effective_enabled_tools, - model_id=effective_model_id, + model_id=turn_model.model_id, system_prompt=system_prompt, # Use assistant's instructions if available - caching_enabled=caching_enabled, - provider=effective_provider, - inference_params=inference_params, - mantle_api_mode=mantle_api_mode, - mantle_region=mantle_region, + caching_enabled=turn_model.caching_enabled, + provider=turn_model.provider, + inference_params=turn_model.inference_params, + mantle_api_mode=turn_model.mantle_api_mode, + mantle_region=turn_model.mantle_region, agent_type=effective_agent_type, - extra_tools=extra_tools, + extra_tools=turn_tools.extra_tools, is_resume=False, accessible_skill_ids=effective_skill_ids, - extra_tools_key_described=extra_tools_key_described, - has_document_tools=bool(document_tools), + extra_tools_key_described=turn_tools.extra_tools_key_described, + has_document_tools=bool(turn_tools.document_tools), assistant_id=input_data.rag_assistant_id, build_stage_recorder=_mark_build_stage, - memory_binding=memory_binding_key, + build_detail_recorder=prelude.detail, + memory_binding=turn_tools.memory_binding_key, memory_context=memory_context, ) @@ -3749,58 +4188,15 @@ async def _build_main_agent(): prelude.mark("agent_build.rest") # Resume requests must target interrupts that the cached agent - # actually has paused. Cache eviction, a process restart, or a - # forged request will otherwise be silently accepted by Strands - # and drop the client's response. Reject up front so the client - # sees a 400 and can restart the turn cleanly. + # actually has paused; see `_validate_resume_interrupts`. if is_resume: - strands_agent = getattr(agent, "agent", None) - interrupt_state = getattr(strands_agent, "_interrupt_state", None) if strands_agent else None - known_ids: set[str] = set() - if interrupt_state and getattr(interrupt_state, "activated", False): - interrupts = getattr(interrupt_state, "interrupts", None) or {} - known_ids = set(interrupts.keys()) - submitted_ids = [entry.interruptId for entry in (input_data.interrupt_responses or [])] - unknown_ids = [iid for iid in submitted_ids if iid not in known_ids] - if unknown_ids: - logger.warning( - "Resume rejected: submitted interrupt ids not in paused state" - ) - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail="Unknown or expired interrupt ids; restart the turn.", - ) + _validate_resume_interrupts(agent, input_data) - # Build citations list for persistence (convert context chunks to citation format) - # Build citations list for persistence (convert context chunks to citation format) - # - # #111: when the agent's ``show_citations`` flag is off, suppress citations - # entirely — leaving this list empty is a single choke point that turns off all - # three downstream consumers at once: the ``event: citation`` SSE below, the - # ``citations=...`` persisted on the stored message, and the copy handed to - # ``agent.stream_async``. RAG retrieval and prompt augmentation above are - # deliberately untouched: the model still receives the context chunks, the user - # just is not shown (or able to download) the sources. - show_citations = getattr(assistant, "show_citations", True) - citations_for_storage = [] - if context_chunks and show_citations: - for chunk in context_chunks: - citations_for_storage.append( - { - "assistantId": input_data.rag_assistant_id, - "documentId": chunk.get("metadata", {}).get("document_id", ""), - # Managed KBs carry the filename under ``filename`` (set at ingest, - # managed_backend.py); legacy S3-Vectors used ``source``. Read - # managed first, fall back to legacy, then the placeholder — before - # this, every managed-KB citation rendered "Unknown Source". - "fileName": ( - chunk.get("metadata", {}).get("filename") - or chunk.get("metadata", {}).get("source") - or "Unknown Source" - ), - "text": chunk.get("text", "")[:500], # Limit excerpt length - } - ) + # Citations to persist and to stream ahead of the answer. Empty when + # the Agent hides them, which switches off every downstream consumer. + citations_for_storage = _build_citations( + assistant, context_chunks, input_data.rag_assistant_id + ) # Create stream with optional quota warning injection async def stream_with_quota_warning() -> AsyncGenerator[str, None]: @@ -3809,9 +4205,13 @@ async def stream_with_quota_warning() -> AsyncGenerator[str, None]: # (kicked off before the quota check on first turns) finishes, # push the title to the client so the sidebar/header rename in # parallel with the pending response instead of at stream end. - # Checked between agent events — never awaited, so it adds no - # latency; a stream that outruns Nova Micro simply never emits - # and the SPA's post-close metadata refresh covers it. + # Never awaited, so it adds no latency. Polled from two places: + # the coordinator's live status merge (every 100ms, so a title + # that lands during the model's time-to-first-token or a long + # tool call goes out right away) and between agent events below + # (the only route while that merge is switched off). A stream + # that outruns Nova Micro simply never emits and the SPA's + # post-close metadata refresh covers it. title_emitted = False def _session_title_sse() -> Optional[str]: @@ -3867,14 +4267,15 @@ def _session_title_sse() -> Optional[str]: # The original text becomes the single source of truth for UI display, # while the full augmented prompt stays in AgentCore Memory for the LLM. attachment_guidance = _build_attachment_guidance( - diverted_tabular, - diverted_presentations, - oversized_inline, + attachments.diverted_tabular, + attachments.diverted_presentations, + attachments.oversized_inline, effective_enabled_tools, - over_budget=over_budget_inline, - dropped_over_count_names=dropped_over_count_names, - dropped_over_count_total=dropped_over_count_total, + over_budget=attachments.over_budget_inline, + dropped_over_count_names=attachments.dropped_over_count_names, + dropped_over_count_total=attachments.dropped_over_count_total, max_files=MAX_FILES_PER_MESSAGE, + unpersisted=attachments.unpersisted_inline, ) # When multiple spreadsheets are visible, ship the full inventory # up front so the agent can disambiguate intentionally instead of @@ -3919,9 +4320,9 @@ def _session_title_sse() -> Optional[str]: # files the previous turn never got an answer for. Say so, or # the model has to guess why documents it was not asked about # are attached to the message. - if recovered_upload_ids and attachment_marker_names: + if attachments.recovered_upload_ids and attachments.marker_names: final_message = ( - f"{_build_attachment_recovery_note(attachment_marker_names)}\n\n{final_message}" + f"{_build_attachment_recovery_note(attachments.marker_names)}\n\n{final_message}" ) if interrupted_turn_reason: @@ -3946,11 +4347,11 @@ def _session_title_sse() -> Optional[str]: message_will_be_modified = ( final_message != input_data.message # RAG augmentation / attachment guidance / inventory - or bool(files_to_send) # File attachments + or bool(attachments.files_to_send) # File attachments # The `[Attached files: …]` marker is appended for diverted # attachments too, so the persisted text differs from what the # user typed even when nothing went inline (a lone .pptx). - or bool(attachment_marker_names) + or bool(attachments.marker_names) ) # Strands' resume protocol wants each entry wrapped as # {"interruptResponse": {...}}. The InvocationRequest schema @@ -3965,8 +4366,8 @@ def _session_title_sse() -> Optional[str]: async for event in agent.stream_async( final_message, session_id=input_data.session_id, - files=files_to_send if files_to_send else None, - attachment_names=attachment_marker_names or None, + files=attachments.files_to_send if attachments.files_to_send else None, + attachment_names=attachments.marker_names or None, citations=citations_for_storage if citations_for_storage else None, original_message=input_data.message if message_will_be_modified else None, interrupt_responses=interrupt_responses_payload, @@ -3993,6 +4394,10 @@ def _session_title_sse() -> Optional[str]: # is not. None for preview sessions and the local # no-DynamoDB path, where steering is simply inert. turn_lease=session_lease, + poll_side_frame=_session_title_sse if title_task is not None else None, + # The same clock as `turn_prelude`, continued to the first model + # output and emitted as `turn_first_token` (turn-path P1b). + turn_clock=prelude, ): yield event # Interleave the finished title between agent events (same @@ -4054,7 +4459,7 @@ def _session_title_sse() -> Optional[str]: # generator's finally is the release site for the happy path (the two # except handlers below cover pre-stream failures). async def _guarded_stream() -> AsyncGenerator[str, None]: - nonlocal agent + nonlocal agent, kb_search, context_chunks, augmented_message, citations_for_storage heartbeat_task = None try: @@ -4149,6 +4554,20 @@ async def _guarded_stream() -> AsyncGenerator[str, None]: + "\n\n" ) + # The knowledge-base search ran alongside the build; take its + # result now, ahead of `stream_with_quota_warning`, which is + # the first reader of the chunks (citation frames) and of the + # augmented message (`final_message`). `rag_wait` is what was + # left of the search once the build was done — ~0 when the + # build outlasted it. + if kb_search is not None: + context_chunks, augmented_message = await kb_search + kb_search = None + citations_for_storage = _build_citations( + assistant, context_chunks, input_data.rag_assistant_id + ) + prelude.mark("rag_wait") + # Emitted here rather than before the return: with the build # deferred, "the window before the client can hear anything" # ends at the agent, not at the response. @@ -4159,6 +4578,13 @@ async def _guarded_stream() -> AsyncGenerator[str, None]: "isResume": is_resume, "hasAssistant": bool(input_data.rag_assistant_id), "deferredBuild": deferred_build, + # Whether the build's SDK clients came from the + # process-wide session (`agent_build_shared_session_enabled`, + # a kill switch), and how many builds this process has + # run: 1 on a conversation's first turn, which is + # always a fresh Runtime process. + "sharedSession": agent_build_shared_session_enabled(), + "processBuilds": process_build_count(), }, ) @@ -4188,7 +4614,7 @@ async def _guarded_stream() -> AsyncGenerator[str, None]: return StreamingResponse( _guarded_stream(), media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, + headers=_sse_headers(input_data.session_id), ) except HTTPException: @@ -4208,15 +4634,10 @@ async def _guarded_stream() -> AsyncGenerator[str, None]: error_event = build_conversational_error_event(code=ErrorCode.AGENT_ERROR, error=e, session_id=input_data.session_id, recoverable=True) - return StreamingResponse( - stream_conversational_message( - message=error_event.message, - stop_reason="error", - metadata_event=error_event, - session_id=input_data.session_id, - user_id=user_id, - user_input=input_data.message, - ), - media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no", "X-Session-ID": input_data.session_id}, + return _refuse_turn( + input_data, + user_id, + message=error_event.message, + stop_reason="error", + metadata_event=error_event, ) diff --git a/backend/src/apis/inference_api/chat/service.py b/backend/src/apis/inference_api/chat/service.py index 76851b8d7..8c578482d 100644 --- a/backend/src/apis/inference_api/chat/service.py +++ b/backend/src/apis/inference_api/chat/service.py @@ -8,6 +8,7 @@ import logging import hashlib import os +import threading from typing import Any, Callable, Dict, Optional, List, Tuple import boto3 @@ -294,6 +295,16 @@ def _adopt_session_conversation(agent: BaseAgent, session_id: str) -> None: logger.debug("Session %s: could not sync compaction live offset", scrub_log(session_id), exc_info=True) +# Agent builds (cache misses) this process has run. Every conversation's first +# turn runs in a fresh Runtime process, so 1 marks exactly the cold-process +# build the agent-build experiment is about (stamped on `turn_prelude`). +_process_build_count = 0 + + +def process_build_count() -> int: + return _process_build_count + + async def get_agent( session_id: str, user_id: Optional[str] = None, @@ -317,6 +328,7 @@ async def get_agent( has_document_tools: bool = False, assistant_id: Optional[str] = None, build_stage_recorder: Optional[Callable[[str], None]] = None, + build_detail_recorder: Optional[Callable[[str, Any], None]] = None, memory_binding: Optional[Dict[str, Any]] = None, memory_context: Optional[str] = None, ) -> BaseAgent: @@ -493,7 +505,9 @@ async def get_agent( set_stage_recorder, ) - _stage_token = set_stage_recorder(build_stage_recorder) + global _process_build_count + _process_build_count += 1 + _stage_token = set_stage_recorder(build_stage_recorder, build_detail_recorder) try: agent = create_agent(**create_kwargs) finally: @@ -596,6 +610,44 @@ def clear_agent_cache(): Focus on being informative and scannable. The title should allow users to quickly identify this conversation in a list.""" +# One Bedrock client for every title, built on first use. A client per call +# cost ~250ms the first time in a process (botocore loads the service model) +# and, every time, a fresh connection pool — a new TCP+TLS handshake to +# bedrock-runtime on the path of a title the user is watching for. boto3 +# clients are thread-safe, so the worker threads below can share it. +_title_bedrock_client: Any = None +# Guards the first build: boto3's default session is not thread-safe, and two +# first-turn titles can reach it from separate worker threads at once. +_title_client_lock = threading.Lock() + +# Strong references to in-flight title writes. The event loop only holds weak +# references to tasks, so a fire-and-forget write could otherwise be +# garbage-collected before it lands. +_pending_title_writes: set = set() + + +def _get_title_bedrock_client() -> Any: + global _title_bedrock_client + if _title_bedrock_client is None: + with _title_client_lock: + if _title_bedrock_client is None: + bedrock_region = os.environ.get('AWS_REGION', 'us-east-1') + _title_bedrock_client = boto3.client('bedrock-runtime', region_name=bedrock_region) + return _title_bedrock_client + + +def _converse_for_title(**kwargs: Any) -> Dict[str, Any]: + """Run in a worker thread, so a first-use client build stays off the event loop.""" + return _get_title_bedrock_client().converse(**kwargs) + + +def _schedule_title_write(session_id: str, user_id: str, title: str) -> None: + task = asyncio.create_task( + update_session_title(session_id=session_id, user_id=user_id, title=title) + ) + _pending_title_writes.add(task) + task.add_done_callback(_pending_title_writes.discard) + async def generate_conversation_title( session_id: str, @@ -608,8 +660,8 @@ async def generate_conversation_title( This function: 1. Truncates user input to ~500 tokens (2000 chars as rough approximation) 2. Calls Nova Micro with optimized system prompt - 3. Updates session metadata both locally and in cloud - 4. Returns generated title or fallback on error + 3. Returns the title as soon as Nova does, persisting it in the background + 4. Returns fallback on error Args: session_id: Session identifier @@ -628,10 +680,6 @@ async def generate_conversation_title( logger.debug(f"Truncated input from {len(user_input)} to {MAX_INPUT_LENGTH} chars") try: - # Initialize Bedrock Runtime client - bedrock_region = os.environ.get('AWS_REGION', 'us-east-1') - bedrock_client = boto3.client('bedrock-runtime', region_name=bedrock_region) - # Prepare request for Nova Micro # us.amazon.nova-micro-v1:0 is the fastest, most cost-effective model request_body = { @@ -657,13 +705,21 @@ async def generate_conversation_title( # whole Nova round-trip, stalling the agent stream this task runs # concurrently with. response = await asyncio.to_thread( - bedrock_client.converse, + _converse_for_title, modelId="us.amazon.nova-micro-v1:0", messages=request_body["messages"], system=request_body["system"], inferenceConfig=request_body["inferenceConfig"], ) + # A guardrail or content-filter stop may carry the refusal as its + # text, which must not become the sidebar title. `max_tokens` is + # still accepted: the 50-char clip below already handles an overrun. + stop_reason = response.get("stopReason") + if stop_reason not in ("end_turn", "max_tokens"): + logger.warning("Title generation stopped with stopReason=%s; keeping the placeholder", stop_reason) + return "New Conversation" + # Extract generated title from response title = response["output"]["message"]["content"][0]["text"].strip() @@ -674,10 +730,13 @@ async def generate_conversation_title( logger.info("✅ Generated title successfully") - # Targeted update — only writes the title attribute. The post-stream - # update_session_activity write is also targeted and disjoint, so the - # two cannot clobber each other on overlapping turns. - await update_session_title(session_id=session_id, user_id=user_id, title=title) + # Persist without holding the title back. This task finishing is what + # lets the stream push `session_title`, so awaiting the DynamoDB write + # here would put its round trips between Nova answering and the user + # seeing the name. Targeted update — only writes the title attribute. + # The post-stream update_session_activity write is also targeted and + # disjoint, so the two cannot clobber each other on overlapping turns. + _schedule_title_write(session_id, user_id, title) return title diff --git a/backend/src/apis/inference_api/chat/turn_timing.py b/backend/src/apis/inference_api/chat/turn_timing.py index bd3295df1..dfe622939 100644 --- a/backend/src/apis/inference_api/chat/turn_timing.py +++ b/backend/src/apis/inference_api/chat/turn_timing.py @@ -110,6 +110,17 @@ def _metric_name(stage: str) -> str: return "".join(p[:1].upper() + p[1:] for p in parts) + "Ms" +def _group_marks(marks: List[Tuple[str, float]]) -> Dict[str, int]: + """Sum ``marks`` into every proper dotted prefix (see ``TurnPrelude._groups``).""" + totals: Dict[str, int] = {} + for name, ms in marks: + parts = name.split(".") + for depth in range(1, len(parts)): + prefix = ".".join(parts[:depth]) + totals[prefix] = totals.get(prefix, 0) + int(ms) + return totals + + class TurnPrelude: """Records named marks across the pre-stream stages of one turn. @@ -118,7 +129,15 @@ class TurnPrelude: everything. A measurement must never be able to break the turn it measures. """ - __slots__ = ("_t0", "_wall_t0", "_last", "_marks") + __slots__ = ( + "_t0", + "_wall_t0", + "_last", + "_marks", + "_details", + "_emitted", + "_first_token_emitted", + ) def __init__(self) -> None: now = time.perf_counter() @@ -131,6 +150,24 @@ def __init__(self) -> None: # domain, so the value handed to it is captured here in ITS domain. self._wall_t0 = time.time() self._marks: List[Tuple[str, float]] = [] + self._details: Dict[str, Any] = {} + # Set by `emit`: (marks already reported, totalMs, sessionId, + # streamKind, extra). `emit_first_token` reports only the marks after + # it, under the same identity, so the coordinator needs no arguments. + self._emitted: Optional[Tuple[int, int, str, str, Optional[Dict[str, Any]]]] = None + self._first_token_emitted = False + + def detail(self, key: str, value: Any) -> None: + """Attach a log-only property describing a stage, e.g. which MCP + server owned the time inside ``agent_build.tools.mcp``. + + Log line only, never a metric: a detail is for reading one turn, and + its values (a per-server list) have no percentile. Last write wins. + """ + try: + self._details[key] = value + except Exception: # noqa: BLE001 - never break a turn to measure it + logger.debug("Turn prelude detail skipped", exc_info=True) def mark(self, stage: str) -> None: """Close the stage that just finished and open the next one. @@ -145,6 +182,20 @@ def mark(self, stage: str) -> None: except Exception: # noqa: BLE001 - never break a turn to measure it logger.debug("Turn prelude mark skipped", exc_info=True) + def mark_at(self, stage: str, at: float) -> None: + """``mark``, but closing the stage at an earlier ``perf_counter`` + reading taken somewhere a mark could not be (a Strands hook). + + A reading older than the previous mark clamps to it — a zero stage, + never a negative one. + """ + try: + end = max(float(at), self._last) + self._marks.append((stage, (end - self._last) * 1000.0)) + self._last = end + except Exception: # noqa: BLE001 - never break a turn to measure it + logger.debug("Turn prelude mark_at skipped", exc_info=True) + @property def started_at(self) -> float: """``time.time()`` for the moment the handler was entered. @@ -187,16 +238,15 @@ def _groups(self) -> Dict[str, int]: the prefix keeps the coarse series comparable across the split while ``stages`` answers the new question. + Every proper prefix is a group, not just the first: splitting + ``agent_build.tools`` into ``agent_build.tools.filter`` and friends + must keep ``agent_build.tools`` as a number for the same reason the + first split kept ``agent_build``. + Undotted stages are deliberately absent — a group of one is noise, and the reader already has that number in ``stages``. """ - totals: Dict[str, int] = {} - for name, ms in self._marks: - prefix, dot, _ = name.partition(".") - if not dot: - continue - totals[prefix] = totals.get(prefix, 0) + int(ms) - return totals + return _group_marks(self._marks) def emit( self, @@ -215,6 +265,7 @@ def emit( # disagree — by microseconds, which is worse than by a lot: nobody # chases a big difference for long, and everybody chases a small one. total_ms = self.total_ms + self._emitted = (len(self._marks), total_ms, session_id, stream_kind, extra) try: payload: Dict[str, Any] = { "sessionId": session_id, @@ -225,6 +276,8 @@ def emit( groups = self._groups() if groups: payload["groups"] = groups + if self._details: + payload.update(self._details) if extra: payload.update(extra) logger.info("turn_prelude %s", json.dumps(payload, default=str)) @@ -244,6 +297,73 @@ def emit( except Exception: # noqa: BLE001 logger.debug("Turn prelude metrics skipped", exc_info=True) + def emit_first_token(self) -> None: + """Log the turn's first model output against the same clock. + + ``turn_prelude`` is emitted when the agent is ready, before the stream + starts, so it cannot carry what happens next: the coordinator's head of + turn, LTM retrieval and the hooks before the model call, and the + model's own time to first token. Those are marked on this object after + ``emit`` (``head_of_turn.*``, ``pre_model``, ``model``) and reported + here as a second line, ``turn_first_token``, joined to the first by + ``sessionId``. ``firstTokenMs`` runs from handler entry, so it is the + prelude plus everything after it — the server's whole share of the + wait the user sees. + + A second line rather than a delayed first one: holding ``turn_prelude`` + until a token arrives would lose it on every turn that never produces + one (an error, a Stop, an interrupt), which are the turns most worth + reading. Once per turn; a call before ``emit`` reports every mark and + no ``preludeTotalMs``. + + Called by the coordinator AFTER the first token has been yielded to the + client, so the log write and the EMF record are never in front of it. + """ + if self._first_token_emitted: + return + self._first_token_emitted = True + + # At the LAST MARK (`model`), not now: this runs on the coordinator's + # next pass, so `total_ms` would also count the wait for the second + # event — 131ms and 182ms on the first two dev readouts. + first_token_ms = max(0, int((self._last - self._t0) * 1000)) + reported, prelude_total_ms, session_id, stream_kind, extra = self._emitted or ( + 0, + None, + "", + "agent", + None, + ) + marks = self._marks[reported:] + try: + payload: Dict[str, Any] = { + "sessionId": session_id, + "streamKind": stream_kind, + "firstTokenMs": first_token_ms, + "preludeTotalMs": prelude_total_ms, + "stages": {name: int(ms) for name, ms in marks}, + } + groups = _group_marks(marks) + if groups: + payload["groups"] = groups + if extra: + payload.update(extra) + logger.info("turn_first_token %s", json.dumps(payload, default=str)) + except Exception: # noqa: BLE001 + logger.debug("Turn first-token emit skipped", exc_info=True) + + try: + self._emit_metrics( + stream_kind=stream_kind, + session_id=session_id, + total_ms=first_token_ms, + extra=extra, + marks=marks, + total_metric="FirstTokenMs", + ) + except Exception: # noqa: BLE001 + logger.debug("Turn first-token metrics skipped", exc_info=True) + def _emit_metrics( self, *, @@ -251,6 +371,8 @@ def _emit_metrics( session_id: str, total_ms: int, extra: Optional[Dict[str, Any]] = None, + marks: Optional[List[Tuple[str, float]]] = None, + total_metric: str = "PreludeTotalMs", ) -> None: """One EMF record per turn, for percentiles over the fleet. @@ -269,18 +391,21 @@ def _emit_metrics( from apis.shared.observability.emf import emit_emf_metrics - metrics: Dict[str, float] = {"PreludeTotalMs": total_ms} - for name, ms in self._marks: + marks = self._marks if marks is None else marks + metrics: Dict[str, float] = {total_metric: total_ms} + for name, ms in marks: metrics[_metric_name(name)] = int(ms) # Group totals last: on a decomposed stage they are the only place the # coarse number exists, and no stage name can collide with one (a group # key is only ever produced by a dotted stage, which never renders to # the same metric name as its own prefix). - for prefix, total in self._groups().items(): + for prefix, total in _group_marks(marks).items(): metrics.setdefault(_metric_name(prefix), total) properties: Dict[str, Any] = {"streamKind": stream_kind, "sessionId": session_id} - for key in ("isResume", "deferredBuild", "hasAssistant"): + # Properties, never dimensions: `sharedSession` is a kill-switch state + # and `processBuilds` is unbounded. + for key in ("isResume", "deferredBuild", "hasAssistant", "sharedSession", "processBuilds"): if extra and key in extra: properties[key] = extra[key] diff --git a/backend/src/apis/inference_api/chat/voice_routes.py b/backend/src/apis/inference_api/chat/voice_routes.py index 841639116..5a67b3066 100644 --- a/backend/src/apis/inference_api/chat/voice_routes.py +++ b/backend/src/apis/inference_api/chat/voice_routes.py @@ -7,7 +7,7 @@ Protocol: Client → Server: - {"type": "config", "session_id": "...", "auth_token": "...", ...} (first message) + {"type": "config", "session_id": "...", "auth_token": "...", "voice": "tiffany", ...} (first message) {"type": "bidi_audio_input", "audio": "", "sample_rate": 16000} {"type": "bidi_text_input", "text": "..."} {"type": "ping"} @@ -293,7 +293,7 @@ async def _finalize_voice_session(session_id: str, user_id: str, voice_agent: An cost = None if pricing and total_tokens > 0: try: - total_cost, breakdown = CostCalculator.calculate_message_cost(accumulated_usage, pricing) + total_cost, breakdown = CostCalculator.calculate_voice_cost(accumulated_usage, pricing) cost = { "total": total_cost, "inputCost": breakdown.input_cost, @@ -391,6 +391,7 @@ async def voice_stream( user_id = _get_param_from_request(websocket, "user-id", user_id) enabled_tools_list = _get_enabled_tools_from_request(websocket, enabled_tools) auth_token = _get_param_from_request(websocket, "auth-token", token) or "" + requested_voice: Optional[str] = None # Always read config message from client (sent on WebSocket open). # Required for auth_token in AgentCore mode and supplements any @@ -404,6 +405,10 @@ async def voice_stream( user_id = first_msg.get("user_id") or user_id enabled_tools_list = first_msg.get("enabled_tools") or enabled_tools_list auth_token = first_msg.get("auth_token") or auth_token + # The user's chosen Nova 2 Sonic voice. Validated in VoiceAgent + # (an unknown id falls back to the default rather than 4xx-ing a + # WebSocket the user has already heard connect). + requested_voice = first_msg.get("voice") logger.info(f"Voice config received from client message") except asyncio.TimeoutError: logger.warning("No config message received within 10s, using query params") @@ -450,15 +455,23 @@ async def voice_stream( user_id=user_id, auth_token=auth_token, enabled_tools=enabled_tools_list, + voice=requested_voice, ) + if requested_voice and voice_agent.voice_id != str(requested_voice).strip().lower(): + logger.warning( + f"Voice '{_sanitize_log(str(requested_voice))}' is not in the catalog; " + f"using '{voice_agent.voice_id}' for session={_sanitize_log(session_id)}" + ) _active_sessions[session_id] = voice_agent - # Send connection confirmation + # Send connection confirmation. `voice` is the id actually in use, so + # a client whose request fell back can show the real one. await websocket.send_json({ "type": "bidi_connection_start", "connection_id": session_id, "status": "connected", + "voice": voice_agent.voice_id, }) # Start the voice agent diff --git a/backend/src/apis/inference_api/warmup.py b/backend/src/apis/inference_api/warmup.py index 7e530a387..b98b01deb 100644 --- a/backend/src/apis/inference_api/warmup.py +++ b/backend/src/apis/inference_api/warmup.py @@ -53,6 +53,17 @@ "s3", ) +# The parse above is per *session*, and the two SDKs on the agent build (the +# AgentCore Memory session manager, Strands' BedrockModel) build their clients +# on the process-wide session from `apis.shared.aws_clients` +# (`agent_build_shared_session_enabled`, default on). Build those clients on +# it here, so the first turn finds them parsed. +WARM_SHARED_SESSION_SERVICES: tuple[str, ...] = ( + "bedrock-agentcore", + "bedrock-agentcore-control", + "bedrock-runtime", +) + def warmup_enabled() -> bool: """Whether startup warm-up runs. Empty/unset means on.""" @@ -87,12 +98,47 @@ def warm_boto_clients(services: Iterable[str] = WARM_BOTO_SERVICES) -> None: _timed(f"boto:{service}", lambda service=service: boto3.client(service, region_name=region)) +def warm_shared_session(services: Iterable[str] = WARM_SHARED_SESSION_SERVICES) -> None: + """Build the agent build's shared boto3 session, its clients, and the memory strategy ids. + + Everything here is client construction — no sockets — except the last + step. ``warm_strategy_ids`` is a control-plane read of the memory's + strategy ids (static configuration, cached for the life of the process), + and it opens the one connection warm-up otherwise avoids. That is + accepted (docs/specs/turn-path-ttft.md §5 P2): the alternative is paying + the read on the first turn, which is the turn this exists to shorten; the + call is idempotent, so botocore retries a connection error; and on + Runtime V2, where a warmed process is snapshotted and restored, the + restore's first turn is the thing to watch for a pool holding a socket + that did not survive. + """ + from apis.shared.feature_flags import agent_build_shared_session_enabled + + if not agent_build_shared_session_enabled(): + logger.info("warmup step=shared outcome=skipped error=AGENT_BUILD_SHARED_SESSION_ENABLED=false") + return + region = os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") + if not region: + logger.info("warmup step=shared outcome=skipped error=no region configured") + return + from apis.shared.aws_clients import shared_boto_session + + session = shared_boto_session() + for service in services: + _timed(f"shared:{service}", lambda service=service: session.client(service, region_name=region)) + + from agents.main_agent.session.session_factory import warm_strategy_ids + + _timed("shared:strategy_ids", warm_strategy_ids) + + def run_warmup() -> None: """The whole warm-up, synchronously. Exposed for tests and for callers that want it inline.""" started = time.perf_counter() warm_modules() warm_boto_clients() + warm_shared_session() logger.info("warmup complete ms=%d", int((time.perf_counter() - started) * 1000)) diff --git a/backend/src/apis/shared/assistants/__init__.py b/backend/src/apis/shared/assistants/__init__.py index 31a273d2a..158e37890 100644 --- a/backend/src/apis/shared/assistants/__init__.py +++ b/backend/src/apis/shared/assistants/__init__.py @@ -39,6 +39,7 @@ from .rag_service import ( augment_prompt_with_context, resolve_context_cap, + search_assistant_knowledgebase_with_cap, search_assistant_knowledgebase_with_formatting, ) @@ -76,5 +77,6 @@ # RAG service functions "augment_prompt_with_context", "resolve_context_cap", + "search_assistant_knowledgebase_with_cap", "search_assistant_knowledgebase_with_formatting", ] diff --git a/backend/src/apis/shared/assistants/rag_service.py b/backend/src/apis/shared/assistants/rag_service.py index 4dd8a44e1..bfb460cf1 100644 --- a/backend/src/apis/shared/assistants/rag_service.py +++ b/backend/src/apis/shared/assistants/rag_service.py @@ -51,14 +51,13 @@ client already reads. The rename stops at the seam; no caller has to change. """ +import asyncio import logging import os import re import time from typing import Any, Dict, List, Mapping, Optional, Set, Tuple -import boto3 - from apis.shared.assistants.kb_access import KbAccess from apis.shared.kb_backend.dual_read import schedule_observation, start_managed_read from apis.shared.kb_backend.idleness import schedule_activity_touch @@ -114,6 +113,27 @@ def resolve_context_cap(assistant_id: str, *, record: Optional[Mapping[str, Any] return MANAGED_MAX_CONTEXT_CHARS if engine == ENGINE_MANAGED else MAX_CONTEXT_CHARS +async def _search_then_filter( + backend: Any, + assistant_id: str, + query: str, + top_k: int, + record: Optional[Mapping[str, Any]] = None, +) -> Tuple[List[Chunk], float, List[Chunk]]: + """``(raw chunks, search ms, chunks whose document is complete)``. + + Runs on a private event loop in a worker thread (see the caller). Only the + backend call and the filter belong here: neither schedules anything that + must outlive this loop. ``search ms`` times the backend call alone — it is + the number the dual-read observation compares across engines. + """ + started = time.perf_counter() + chunks = await backend.search(assistant_id, query, top_k, record=record) + search_ms = (time.perf_counter() - started) * 1000.0 + complete = _filter_chunks_by_document_status(chunks, assistant_id) if chunks else [] + return chunks, search_ms, complete + + async def search_assistant_knowledgebase_with_formatting( assistant_id: str, query: str, @@ -121,8 +141,30 @@ async def search_assistant_knowledgebase_with_formatting( *, access: Optional[KbAccess], ) -> List[Dict[str, Any]]: + """Search an assistant's knowledge base; see :func:`search_assistant_knowledgebase_with_cap`. + + For callers that do not augment a prompt and so have no use for the cap. + """ + results, _ = await search_assistant_knowledgebase_with_cap( + assistant_id, query, top_k, access=access + ) + return results + + +async def search_assistant_knowledgebase_with_cap( + assistant_id: str, + query: str, + top_k: int = DEFAULT_TOP_K, + *, + access: Optional[KbAccess], +) -> Tuple[List[Dict[str, Any]], int]: """ - Search assistant knowledge base and return formatted results + Search assistant knowledge base and return formatted results, with the context cap + + The cap is :func:`resolve_context_cap` decided from the same KB_Record that + chose the backend. Returning it here is what lets a turn read that record + once: the backend's own lookup of ``awsKbId`` and the cap each used to read it + again, on a fresh client, on the turn's critical path (~50ms apiece on dev). Resolves the knowledge base's backend, delegates the search across the seam, then applies the parity rules that must hold on every backend: the document @@ -140,13 +182,14 @@ async def search_assistant_knowledgebase_with_formatting( ``kb_access.resolve_kb_access`` if it is not. Returns: - List of dictionaries containing: + ``(results, cap)``. ``results`` is a list of dictionaries containing: - text: Chunk text content - distance: Similarity distance (lower = more similar) - metadata: Original metadata from vector store - key: Vector key/ID - Empty when the caller has no grant — no backend is contacted at all. + Empty when the caller has no grant — no backend is contacted at all, and + ``cap`` is the legacy :data:`MAX_CONTEXT_CHARS` (nothing to cap). """ # Authorization first, before the backend resolution, the query clamp, and # any AWS call (Requirement 25.1). Ordering is the requirement: a check that @@ -158,7 +201,7 @@ async def search_assistant_knowledgebase_with_formatting( f"no resolved access grant" ) emit_count(METRIC_ACCESS_DENIED, dimensions={"reason": "no_grant"}) - return [] + return [], MAX_CONTEXT_CHARS if access.assistant_id != assistant_id: # A grant for a different assistant is not a grant for this one. This is @@ -170,14 +213,15 @@ async def search_assistant_knowledgebase_with_formatting( f"{access.assistant_id}, not {assistant_id}" ) emit_count(METRIC_ACCESS_DENIED, dimensions={"reason": "grant_mismatch"}) - return [] + return [], MAX_CONTEXT_CHARS managed_task = None + cap = MAX_CONTEXT_CHARS try: - # One record read serves both questions: which backend to use, and whether - # this knowledge base is in the dual-read pilot. Reading it here rather - # than letting the resolver read it internally is what keeps the pilot - # from costing an extra DynamoDB round trip on every turn. + # One record read serves every question this search asks of it: which + # backend to use, whether this knowledge base is in the dual-read pilot, + # the managed backend's ``awsKbId``, and the context cap. Each of those + # used to be its own DynamoDB round trip on the turn's critical path. record = load_record(assistant_id) backend = resolve_backend(assistant_id, record=record) @@ -189,6 +233,7 @@ async def search_assistant_knowledgebase_with_formatting( # the SAME record ``resolve_backend`` just used (no extra DynamoDB round # trip), so the logged engine can never disagree with the one that ran. engine = resolve_engine_for(assistant_id, record=record) + cap = resolve_context_cap(assistant_id, record=record) logger.info( "knowledge base retrieval for assistant %s served by engine=%s (%s)", assistant_id, @@ -210,9 +255,17 @@ async def search_assistant_knowledgebase_with_formatting( # ``None`` whenever there is no comparison to make. managed_task = start_managed_read(record, assistant_id, query, top_k) - started = time.perf_counter() - chunks = await backend.search(assistant_id, query, top_k) - legacy_ms = (time.perf_counter() - started) * 1000.0 + # The search and the document-status filter run back to back on one + # worker thread, on a loop of its own. An agent turn starts this search + # before the agent build and awaits it after (turn-path spec §5 P3b), + # and a cold build holds the event loop: had the filter run here, after + # the search, its DynamoDB reads would have waited for the build to let + # go (~60ms of residual `rag_wait` on dev). Everything that schedules + # onto THIS loop — the observational read above, the observation and + # the activity touch below — stays on it. + chunks, legacy_ms, complete_chunks = await asyncio.to_thread( + asyncio.run, _search_then_filter(backend, assistant_id, query, top_k, record) + ) # Detach the comparison. Legacy is what gets served either way — including # when it is empty, which is a finding rather than a reason to reach for @@ -234,10 +287,10 @@ async def search_assistant_knowledgebase_with_formatting( if not chunks: logger.info(f"No vectors found for assistant {assistant_id} with query: {query[:50]}...") - return [] + return [], cap # Filter out chunks from documents that are not in "complete" status - chunks = _filter_chunks_by_document_status(chunks, assistant_id) + chunks = complete_chunks # Format results - return document_id for on-demand download URL generation formatted_results = [] @@ -266,7 +319,7 @@ async def search_assistant_knowledgebase_with_formatting( ) logger.info(f"Found {len(deduped_results)} relevant chunks for assistant {assistant_id}") - return deduped_results + return deduped_results, cap except Exception as e: logger.error(f"Error searching knowledge base for assistant {assistant_id}: {e}", exc_info=True) @@ -277,7 +330,19 @@ async def search_assistant_knowledgebase_with_formatting( # exception was never retrieved. managed_task.cancel() # Return empty list on error (graceful degradation) - return [] + return [], cap + + +def _documents_table(table_name: str, region: str) -> Any: + """The assistants table for the status lookups, on the process-cached resource. + + A fresh ``boto3.resource`` per search paid ~45–55ms for its first ``GetItem`` in + the Runtime (a new connection pool) against ~3ms on a reused one, and this + lookup sits between the retrieval and the model's first token. + """ + from apis.shared.aws_clients import get_dynamodb_table + + return get_dynamodb_table(table_name, region) def _filter_chunks_by_document_status(chunks: List[Chunk], assistant_id: str) -> List[Chunk]: @@ -359,8 +424,7 @@ def _filter_vectors_by_document_status(vectors: List[Dict[str, Any]], assistant_ table_name = os.environ.get("DYNAMODB_ASSISTANTS_TABLE_NAME") if table_name: region = os.environ.get("AWS_REGION", "us-west-2") - dynamodb = boto3.resource("dynamodb", region_name=region) - table = dynamodb.Table(table_name) + table = _documents_table(table_name, region) for doc_id in doc_ids: try: response = table.get_item( diff --git a/backend/src/apis/shared/aws_clients.py b/backend/src/apis/shared/aws_clients.py index e77bd2870..050b7c218 100644 --- a/backend/src/apis/shared/aws_clients.py +++ b/backend/src/apis/shared/aws_clients.py @@ -19,6 +19,23 @@ calls, and credential refresh is handled inside the shared session. What is NOT safe is sharing one across a *moto* boundary — see below. +THE SHARED SESSION (`shared_boto_session`) +------------------------------------------ +The cache above serves callers that ask *this module* for a client. Two SDKs +on the first-turn agent build do not: the AgentCore Memory session manager +and Strands' `BedrockModel` each construct a fresh `boto3.Session` and build +their clients on it. A fresh session re-parses every service model it +touches (the parse is per session, not per process), so a cold first turn +paid that parse several times over — see `docs/specs/turn-path-ttft.md` +§5 P2. Both SDKs accept a session, so `shared_boto_session()` is the one +process-wide session handed to them (`agent_build_shared_session_enabled`, +default on) and to `apis/inference_api/warmup.py`, which builds its clients +at container start so a first turn finds them already parsed. Its `client()` +returns one client per configuration, so every caller that asks for the same +service with the same config shares one client and one connection pool. +Reset with everything else: a session built under one moto backend is as +stale as a client built under it. + THE MOTO TRAP, AND WHY `reset_cached_clients` EXISTS ---------------------------------------------------- `moto.mock_aws()` is entered per test (`tests/*/conftest.py`). A client built @@ -39,6 +56,9 @@ import threading from typing import Any, Dict, Optional, Tuple +import boto3 +from botocore.config import Config as _BotocoreConfig + logger = logging.getLogger(__name__) # Keyed by (service_name, region). `region=None` means "whatever the ambient @@ -112,13 +132,85 @@ def get_dynamodb_table(table_name: str, region_name: Optional[str] = None) -> An return get_resource("dynamodb", region_name).Table(table_name) +# --------------------------------------------------------------------------- +# The shared session +# --------------------------------------------------------------------------- + +# Sized for concurrent sessions sharing one pool. Async persistence writes each +# message through ``asyncio.to_thread``, so a busy container can have dozens of +# ``CreateEvent`` calls in flight on these clients at once, where each session +# manager used to have a pool of its own. Past the pool size urllib3 still +# serves the request, but discards the extra connection afterwards (and warns). +SHARED_SESSION_MAX_POOL_CONNECTIONS = 50 + + +def _shared_client_key(service_name: str, region_name: Optional[str], config: Any) -> Tuple[str, Optional[str], str]: + # ``Config`` is not hashable; its user-provided options are what make two + # configs different (the SDKs differ from each other only in user agent). + options = getattr(config, "_user_provided_options", None) or {} + return service_name, region_name, repr(sorted(options.items())) + + +class ClientReusingSession(boto3.Session): + """A boto3 session whose ``client()`` returns one client per configuration. + + boto3 clients are thread-safe; building them is not, and building one is + the expensive part, so creation happens once, under a lock. Calls carrying + anything beyond region and config (explicit credentials, an endpoint + override) are not ours to share and pass straight through; a keyword + passed as ``None`` (Strands passes ``endpoint_url=None``) is not an + override and does not defeat the sharing. + """ + + def __init__(self, **kwargs: Any) -> None: + super().__init__(**kwargs) + self._shared_clients: Dict[Tuple[str, Optional[str], str], Any] = {} + self._shared_clients_lock = threading.Lock() + + def client(self, service_name: str, region_name: Optional[str] = None, config: Any = None, **kwargs: Any) -> Any: # type: ignore[override] + overrides = {key: value for key, value in kwargs.items() if value is not None} + if overrides: + return super().client(service_name, region_name=region_name, config=config, **overrides) + key = _shared_client_key(service_name, region_name, config) + with self._shared_clients_lock: + client = self._shared_clients.get(key) + if client is None: + pooled = _BotocoreConfig(max_pool_connections=SHARED_SESSION_MAX_POOL_CONNECTIONS) + merged = pooled.merge(config) if config is not None else pooled + client = super().client(service_name, region_name=region_name, config=merged) + self._shared_clients[key] = client + return client + + +_shared_session: Optional[ClientReusingSession] = None + + +def shared_boto_session() -> ClientReusingSession: + """The process-wide session the agent build's SDK clients are built on. + + Built by ``apis/inference_api/warmup.py`` at container start, so a first + turn that reaches for it finds its service models already parsed; built + here on first use otherwise. + """ + global _shared_session + session = _shared_session + if session is not None: + return session + with _lock: + if _shared_session is None: + _shared_session = ClientReusingSession() + return _shared_session + + def reset_cached_clients() -> None: - """Drop every cached client and resource. + """Drop every cached client and resource, and the shared session. Exists for tests. `moto.mock_aws()` is entered per test, so a client built under one test's mock must never be reused by the next — see the module docstring. Production code has no reason to call this. """ + global _shared_session with _lock: _resources.clear() _clients.clear() + _shared_session = None diff --git a/backend/src/apis/shared/costs/calculator.py b/backend/src/apis/shared/costs/calculator.py index 0c26040fc..033a4dcfc 100644 --- a/backend/src/apis/shared/costs/calculator.py +++ b/backend/src/apis/shared/costs/calculator.py @@ -118,6 +118,78 @@ def calculate_message_cost( return total_cost, breakdown + @staticmethod + def calculate_voice_cost( + usage: Dict[str, int], + pricing: Dict[str, float], + ) -> Tuple[float, CostBreakdown]: + """Price a speech-to-speech (Nova Sonic) session. + + Sonic bills four buckets — speech in/out and text in/out — and the + speech rates are roughly 10x the text ones ($3.00 / $12.00 against + $0.32 / $2.65 per MTok in us-west-2, Price List API 2026-09-30). The + model's ``usageEvent`` carries the split under ``details.total``; + ``VoiceAgent`` surfaces it here as ``speechInputTokens`` / + ``textInputTokens`` / ``speechOutputTokens`` / ``textOutputTokens``. + + When the split is missing (an older Strands that only forwards + the totals, or a session that ended before the first usage event), the + totals are priced **at the speech rates**. That over-counts text + tokens rather than under-counting speech ones — a session that was + mostly listening and talking is priced about right, and one that was + mostly text history is priced high, which is the safe direction for a + quota. Never falls back to the text rates: that would be the $0-ish + bill this method exists to replace. + + ``pricing`` is the ``get_model_pricing`` dict: ``inputPricePerMtok`` / + ``outputPricePerMtok`` are the TEXT rates on a speech model, and + ``speechInputPricePerMtok`` / ``speechOutputPricePerMtok`` the speech + ones. A catalog row without the speech rates prices speech tokens at + the text rates — better than nothing, and ``UnmeteredModelCall`` no + longer fires, so the fix is to fill the row in, not to guess here. + """ + text_in_price = pricing.get("inputPricePerMtok") or 0.0 + text_out_price = pricing.get("outputPricePerMtok") or 0.0 + speech_in_price = pricing.get("speechInputPricePerMtok") + speech_out_price = pricing.get("speechOutputPricePerMtok") + if speech_in_price is None: + speech_in_price = text_in_price + if speech_out_price is None: + speech_out_price = text_out_price + + total_in = usage.get("inputTokens") or 0 + total_out = usage.get("outputTokens") or 0 + has_split = any( + usage.get(k) is not None + for k in ("speechInputTokens", "textInputTokens", "speechOutputTokens", "textOutputTokens") + ) + if has_split: + speech_in = usage.get("speechInputTokens") or 0 + text_in = usage.get("textInputTokens") or 0 + speech_out = usage.get("speechOutputTokens") or 0 + text_out = usage.get("textOutputTokens") or 0 + # The totals are authoritative; a split that does not add up to + # them (a usage event missing one bucket) puts the remainder in + # the speech bucket, for the same reason as the no-split case. + speech_in += max(0, total_in - speech_in - text_in) + speech_out += max(0, total_out - speech_out - text_out) + else: + speech_in, text_in = total_in, 0 + speech_out, text_out = total_out, 0 + + input_cost = (speech_in / 1_000_000) * speech_in_price + (text_in / 1_000_000) * text_in_price + output_cost = (speech_out / 1_000_000) * speech_out_price + (text_out / 1_000_000) * text_out_price + total_cost = input_cost + output_cost + + breakdown = CostBreakdown( + inputCost=input_cost, + outputCost=output_cost, + cacheReadCost=0.0, + cacheWriteCost=0.0, + totalCost=total_cost, + ) + return total_cost, breakdown + @staticmethod def calculate_cache_savings( cache_read_tokens: int, diff --git a/backend/src/apis/shared/costs/pricing_config.py b/backend/src/apis/shared/costs/pricing_config.py index 67194745f..e1ac096c4 100644 --- a/backend/src/apis/shared/costs/pricing_config.py +++ b/backend/src/apis/shared/costs/pricing_config.py @@ -40,6 +40,11 @@ async def get_model_by_model_id(model_id: str) -> Optional[Dict[str, any]]: return None +def _is_rate(value: object) -> bool: + """A real published rate — not None, not a bool, not a test double.""" + return isinstance(value, (int, float)) and not isinstance(value, bool) + + async def get_model_pricing(model_id: str) -> Optional[Dict[str, float]]: """ Get pricing information for a model from managed models database @@ -77,6 +82,15 @@ async def get_model_pricing(model_id: str) -> Optional[Dict[str, float]]: if model.cache_read_price_per_million_tokens is not None: pricing["cacheReadPricePerMtok"] = model.cache_read_price_per_million_tokens + # Speech-to-speech models carry a second rate card for speech tokens; + # `CostCalculator.calculate_voice_cost` reads these two. + speech_in = getattr(model, "speech_input_price_per_million_tokens", None) + speech_out = getattr(model, "speech_output_price_per_million_tokens", None) + if _is_rate(speech_in): + pricing["speechInputPricePerMtok"] = float(speech_in) + if _is_rate(speech_out): + pricing["speechOutputPricePerMtok"] = float(speech_out) + return pricing diff --git a/backend/src/apis/shared/embeddings/bedrock_embeddings.py b/backend/src/apis/shared/embeddings/bedrock_embeddings.py index 159b4e649..cb87b8378 100644 --- a/backend/src/apis/shared/embeddings/bedrock_embeddings.py +++ b/backend/src/apis/shared/embeddings/bedrock_embeddings.py @@ -12,6 +12,7 @@ import json import logging import os +import threading from typing import Any, Dict, List import boto3 @@ -33,6 +34,42 @@ logger = logging.getLogger(__name__) +# One Bedrock client for every embedding, built on first use. A client per call +# cost ~250ms the first time in a process (botocore loads the service model) and, +# every time, a fresh connection pool — a new TCP+TLS handshake on each knowledge +# base search, which sits in front of the model's first token. boto3 clients are +# thread-safe, so the executor workers below can share it. Held here rather than +# in `apis.shared.aws_clients` because the kb-sync and rag-ingestion Lambda +# images copy this package without that module. +_bedrock_runtime_client: Any = None +# Guards the first build: boto3's default session is not thread-safe, and +# parallel embeddings can reach it from separate worker threads at once. +_bedrock_runtime_client_lock = threading.Lock() + + +def _get_bedrock_runtime_client() -> Any: + global _bedrock_runtime_client + if _bedrock_runtime_client is None: + with _bedrock_runtime_client_lock: + if _bedrock_runtime_client is None: + _bedrock_runtime_client = boto3.client("bedrock-runtime", region_name=AWS_REGION) + return _bedrock_runtime_client + + +# The same, for the knowledge-base search's `QueryVectors`: one client per +# process instead of a fresh one (and its service-model parse) per search. +_s3vectors_client: Any = None +_s3vectors_client_lock = threading.Lock() + + +def _get_s3vectors_client() -> Any: + global _s3vectors_client + if _s3vectors_client is None: + with _s3vectors_client_lock: + if _s3vectors_client is None: + _s3vectors_client = boto3.client("s3vectors", region_name=AWS_REGION) + return _s3vectors_client + def _get_vector_store_bucket() -> str: """Get vector store bucket name, validating if not set""" @@ -74,18 +111,17 @@ async def generate_embeddings(chunks: List[str]) -> List[List[float]]: Raises: Exception: If Bedrock API call fails """ - bedrock_runtime = boto3.client("bedrock-runtime", region_name=AWS_REGION) - logger.info(f"Generating embeddings for {len(chunks)} chunks in parallel...") async def get_single_embedding(chunk: str, index: int) -> List[float]: """Generate embedding for a single chunk""" loop = asyncio.get_event_loop() - # Run synchronous boto3 call in thread pool to avoid blocking + # Run synchronous boto3 call in thread pool to avoid blocking; the + # client is fetched there too, so its first build stays off the loop. response = await loop.run_in_executor( None, - lambda: bedrock_runtime.invoke_model( + lambda: _get_bedrock_runtime_client().invoke_model( modelId=BEDROCK_EMBEDDING_CONFIG["model_id"], contentType="application/json", accept="application/json", @@ -148,26 +184,41 @@ async def store_embeddings_in_s3( async def search_assistant_knowledgebase(assistant_id: str, query: str): - """Search the S3 vector store for chunks relevant to the query.""" - client = boto3.client("s3vectors", region_name=AWS_REGION) + """Search the S3 vector store for chunks relevant to the query. + + Both round trips — the query embedding and ``QueryVectors`` — run on ONE + worker thread, back to back. The turn path starts this search before the + agent build and awaits it after (docs/specs/turn-path-ttft.md §5 P3b), and + a cold build holds the event loop while it runs: had the embedding been + awaited on the loop, ``QueryVectors`` could not be sent until the build + let go, and only the first hop would overlap. The ``s3vectors`` client is + cached per process, so its service-model parse is paid once. + """ + return await asyncio.to_thread(_search_assistant_knowledgebase_sync, assistant_id, query) + +def _search_assistant_knowledgebase_sync(assistant_id: str, query: str) -> Dict[str, Any]: # Generate vector for the query. Length is already capped upstream by the # facade's query clamp (10,000 chars, the Managed KB Retrieve limit). - query_embedding = await generate_embeddings([query]) + response = _get_bedrock_runtime_client().invoke_model( + modelId=BEDROCK_EMBEDDING_CONFIG["model_id"], + contentType="application/json", + accept="application/json", + body=json.dumps({"inputText": query}), + ) + query_embedding = json.loads(response["body"].read()).get("embedding") # Query the Global Index with a STRICT Filter - response = client.query_vectors( + return _get_s3vectors_client().query_vectors( vectorBucketName=_get_vector_store_bucket(), indexName=_get_vector_store_index(), - queryVector={"float32": query_embedding[0]}, + queryVector={"float32": query_embedding}, filter={"assistant_id": assistant_id}, topK=5, returnMetadata=True, returnDistance=True, ) - return response - #: The S3 Vectors ``GetVectors`` limit on ``keys`` per call. GET_VECTORS_MAX_KEYS = 100 diff --git a/backend/src/apis/shared/feature_flags.py b/backend/src/apis/shared/feature_flags.py index a4b4b03fe..5dd4ad4fb 100644 --- a/backend/src/apis/shared/feature_flags.py +++ b/backend/src/apis/shared/feature_flags.py @@ -653,3 +653,122 @@ def compaction_summary_extract_enabled() -> bool: every planted fact on the quality harness, Nova Micro 88%. """ return os.environ.get("COMPACTION_SUMMARY_EXTRACT_ENABLED", "").strip().lower() != "false" + + + +def agent_build_shared_session_enabled() -> bool: + """Whether the agent build's SDK clients are built on one process-wide boto3 session. + + Three things on a first-turn build construct a fresh ``boto3.Session`` + and parse service models on it: the AgentCore Memory session manager + (which builds a ``MemoryClient``, then a second session and clients that + replace the first pair), ``_discover_strategy_ids``, and Strands' + ``BedrockModel``. A fresh session re-parses every model it touches, and + the parse is the cost. With this on, all three are handed the session + from ``apis.shared.aws_clients.shared_boto_session``, which warm-up + builds at container start together with its clients and the strategy + ids, so the first turn finds them ready. + + **Default ON with a kill switch** (house style, mirroring + ``agent_status_enabled``): unset or empty resolves to enabled; only the + literal ``"false"`` (case-insensitive) disables. Off, every SDK builds its + own session exactly as before, and warm-up skips the shared step. + + Measured on dev, 2026-09-30, 15 first turns per arm, interleaved, every + turn a cold Runtime process (docs/specs/turn-latency-preamble.md PR-6): + ``agent_build`` median 869ms → 372ms, time to first token at the client + 4211ms → 3564ms, and the two distributions did not overlap. Nothing + reaches the prompt; the stamp on ``turn_prelude`` (``sharedSession``) + is a property, never a dimension. + """ + return os.environ.get("AGENT_BUILD_SHARED_SESSION_ENABLED", "").strip().lower() != "false" + + +def mcp_parallel_preflight_enabled() -> bool: + """Whether external MCP servers are pre-flighted concurrently at agent build. + + Each external MCP server's client is started and its tools listed before + the agent is built, and that handshake blocks its thread. Loaded one after + another, two cold Lambda-URL servers cost 9.5s + 12s on a KB agent's first + turn on dev (docs/specs/turn-path-ttft.md §5 P3). With this on, each + pre-flight runs on its own worker thread and the build waits for the + slowest, not the sum; clients are still returned in catalog order, so the + tool order in ``toolConfig`` (the prompt-cache prefix) is unchanged. + + **Default ON with a kill switch** (house style): unset or empty resolves + to enabled; only the literal ``"false"`` (case-insensitive) disables. Off, + servers load one at a time on the build's loop, exactly as before. A build with a + single external server always loads it inline. + """ + return os.environ.get("MCP_PARALLEL_PREFLIGHT_ENABLED", "").strip().lower() != "false" + + +def kb_search_ahead_enabled() -> bool: + """Whether an agent turn's knowledge-base search overlaps the agent build. + + The search used to be awaited in the route, before the stream opened, and + the (deferred) agent build only started after it. The build reads neither + the retrieved chunks nor the augmented message — only the stream does — so + with this on the search starts where it always did and is awaited after + the build, ahead of the citation frames. The augmented message and the + citations are byte-identical either way; only when the round trip happens + moves (docs/specs/turn-path-ttft.md §5 P3b). + + **Default ON with a kill switch** (house style): unset or empty resolves + to enabled; only the literal ``"false"`` (case-insensitive) disables. Off, + the search is awaited in the route, exactly as before. + """ + return os.environ.get("KB_SEARCH_AHEAD_ENABLED", "").strip().lower() != "false" + + +def memory_retrieval_prefetch_enabled() -> bool: + """Whether the long-term-memory lookup starts as soon as the user's message + is added, overlapping the two Memory writes the SDK awaits first. + + On ``MessageAddedEvent`` the SDK awaits ``append_message`` and + ``sync_agent`` (two ``CreateEvent`` calls) and only then runs the lookup — + three network calls in series before the model is called, measured at + 450ms of a 1687ms warm first token on dev (docs/specs/turn-path-ttft.md + P1b). With this on, ``TurnBasedSessionManager`` starts the lookup first and + applies its result at the same point as before, so persisted and live + message bytes are unchanged; only the round trip overlaps the writes. + + **Default ON with a kill switch** (house style): unset or empty resolves + to enabled; only the literal ``"false"`` (case-insensitive) disables. Off, + the lookup runs inline after the writes, exactly as before. + """ + return os.environ.get("MEMORY_RETRIEVAL_PREFETCH_ENABLED", "").strip().lower() != "false" + + +def history_count_prefetch_enabled() -> bool: + """Whether the per-turn history count runs off the critical path. + + Every turn counts the session's stored messages (a paginated + ``ListEvents`` with payloads) to key its per-message metadata. Read at the + head of the turn it cost ~62ms + 2.25ms per stored event before the model + was called, ~390ms at 100+ events (docs/specs/turn-path-ttft.md §5 P4). + With this on, the stream coordinator starts the read at the head of the + turn and awaits it only where the index is used, counting messages created + before the turn began so the number is the one the serial read gave. + + **Default ON with a kill switch** (house style): unset or empty resolves + to enabled; only the literal ``"false"`` (case-insensitive) disables. Off, + the count is read at the head of the turn, exactly as before. + """ + return os.environ.get("HISTORY_COUNT_PREFETCH_ENABLED", "").strip().lower() != "false" + + +def inline_attachment_persist_enabled() -> bool: + """Whether inline ``files`` bytes of a diverted class (spreadsheets, decks) + are written to S3 and registered as session files before the turn runs. + + Covers ``apis.shared.files.inline_persist`` as called from the inference + API's attachment phase. The SPA uploads first and sends ``file_upload_ids``, + so this only ever fires for headless callers posting base64 ``files`` — + without it their diverted attachments are dropped while the guidance note + says the spreadsheet / PowerPoint tools can reach them. **Default ON with + a kill switch** (house style): unset or empty resolves to enabled; only the + literal ``"false"`` (case-insensitive) disables. Off, the pre-fix behaviour + returns exactly: nothing is written and the note is unchanged. + """ + return os.environ.get("INLINE_ATTACHMENT_PERSIST_ENABLED", "").strip().lower() != "false" diff --git a/backend/src/apis/shared/files/document_digest.py b/backend/src/apis/shared/files/document_digest.py index 5abea1059..79e4cad36 100644 --- a/backend/src/apis/shared/files/document_digest.py +++ b/backend/src/apis/shared/files/document_digest.py @@ -45,6 +45,8 @@ from pydantic import BaseModel, Field +from apis.shared.aws_clients import get_client + from .document_read import _open_pdf, _pdf_page_text, docx_paragraphs, document_format_for logger = logging.getLogger(__name__) @@ -257,18 +259,24 @@ def _abstract_prompt(outline: DocumentDigest, sample: str) -> str: return "\n".join(lines) +def _converse(region: str, **kwargs: Any) -> Dict[str, Any]: + """Run in a worker thread, so a first-use client build stays off the event loop. + + The client is process-cached: one per upload paid botocore's service-model + load (~250ms the first time in a process) and, every time, a fresh + connection pool — a new TCP+TLS handshake per abstract. + """ + return get_client("bedrock-runtime", region).converse(**kwargs) + + async def generate_abstract(outline: DocumentDigest, sample: str, model_id: str = DOCUMENT_DIGEST_MODEL_ID) -> Optional[str]: """3–5 sentences from the cheap model, or ``None`` (never raises).""" if not sample.strip(): return None try: - import boto3 - except ImportError: # pragma: no cover - return None - try: - client = boto3.client("bedrock-runtime", region_name=os.environ.get("AWS_REGION", "us-west-2")) response = await asyncio.to_thread( - client.converse, + _converse, + os.environ.get("AWS_REGION", "us-west-2"), modelId=model_id, messages=[{"role": "user", "content": [{"text": _abstract_prompt(outline, sample)}]}], system=[{"text": _ABSTRACT_SYSTEM_PROMPT}], @@ -277,9 +285,15 @@ async def generate_abstract(outline: DocumentDigest, sample: str, model_id: str # silently drop every abstract. inferenceConfig={"temperature": 0.2, "maxTokens": _ABSTRACT_MAX_OUTPUT_TOKENS}, ) - if response.get("stopReason") == "max_tokens": + stop_reason = response.get("stopReason") + if stop_reason == "max_tokens": logger.debug("Document abstract hit the token ceiling; discarding") return None + # A guardrail or content-filter stop may carry the refusal as its + # text; only a finished generation is an abstract. + if stop_reason != "end_turn": + logger.debug("Document abstract stopped with stopReason=%s; discarding", stop_reason) + return None text = response["output"]["message"]["content"][0]["text"].strip() return re.sub(r"\s+", " ", text) or None except Exception: # noqa: BLE001 - an abstract is never worth a failed upload diff --git a/backend/src/apis/shared/files/inline_persist.py b/backend/src/apis/shared/files/inline_persist.py new file mode 100644 index 000000000..b9606476b --- /dev/null +++ b/backend/src/apis/shared/files/inline_persist.py @@ -0,0 +1,251 @@ +"""Persist inline attachment bytes as session files. + +The SPA never sends file bytes on a turn: it uploads first (presigned PUT via +app-api) and sends ``file_upload_ids``, so every attachment it makes already +has an S3 object and a ``FileMetadata`` row. Headless callers — the harness, +scheduled runs, API clients, the smoke matrix — post base64 ``files`` instead. + +That is fine for anything the turn sends inline to Bedrock. It is a silent +hole for the classes the turn *diverts*: spreadsheets and decks are dropped +from the prompt with a note saying the Spreadsheet Analysis / PowerPoint tools +can reach them, but those tools look files up through +``FileUploadRepository.list_session_files`` — and an inline file was never +written anywhere. The model then reports the file "is not accessible", and the +note that promised otherwise is already in the conversation. + +This module closes the hole by giving an inline diverted file exactly what an +upload would have: the same S3 key layout, the same metadata row, ``READY`` +status, the quota increment. Validation mirrors app-api's upload service +(allowlisted MIME, per-class size cap, user quota) so an inline caller cannot +bypass limits the SPA path enforces. ``source`` distinguishes the provenance +for display; it is never an access input. + +Cost on the turn path: one S3 put per diverted inline file, run off the event +loop and in parallel across files, inside the ``preamble.files`` stage. It +only runs when the request carries inline bytes of a diverted class, which the +SPA never does — attended turns pay nothing. +""" + +from __future__ import annotations + +import asyncio +import base64 +import binascii +import logging +import os +import uuid +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Any, List, Optional, Protocol, Sequence, Tuple + +import boto3 + +from apis.shared.files.models import ( + FileMetadata, + FileStatus, + is_allowed_mime_type, + is_presentation_file, +) +from apis.shared.files.repository import FileUploadRepository, get_file_upload_repository + +logger = logging.getLogger(__name__) + +#: Provenance recorded on rows this module writes (``FileMetadata.source``). +INLINE_SOURCE = "inline" + + +class InlineFile(Protocol): + """What the turn hands us: the three fields of ``FileContent``.""" + + filename: str + content_type: str + bytes: str # base64 + + +@dataclass(frozen=True) +class PersistFailure: + """One inline file that was not persisted, and why (user-facing).""" + + file: Any + reason: str + + +# Same env names and defaults as ``apis.app_api.files.service.FileUploadService``. +# The inference container cannot import app-api (``tests/architecture``), so the +# limits are read here under the same contract rather than imported. +def _max_size_for(filename: str, mime_type: str) -> int: + if is_presentation_file(filename, mime_type): + return int(os.environ.get("FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION", 25 * 1024 * 1024)) + return int(os.environ.get("FILE_UPLOAD_MAX_SIZE_BYTES", 4 * 1024 * 1024)) + + +def _user_quota_bytes() -> int: + return int(os.environ.get("FILE_UPLOAD_USER_QUOTA_BYTES", 1024 * 1024 * 1024)) + + +def _bucket_name() -> str: + return os.environ.get("S3_USER_FILES_BUCKET_NAME", "user-files") + + +def new_upload_id() -> str: + """Timestamp-prefixed id, byte-compatible with app-api's upload ids.""" + timestamp_hex = format(int(datetime.now(timezone.utc).timestamp() * 1000), "x") + return f"{timestamp_hex}_{uuid.uuid4().hex[:16]}" + + +def s3_key_for(user_id: str, session_id: str, upload_id: str, filename: str) -> str: + """The upload service's key layout, so tools and lifecycle rules see one shape.""" + return f"user-files/{user_id}/{session_id}/{upload_id}/{filename}" + + +def _decode(file: InlineFile) -> Optional[bytes]: + try: + return base64.b64decode(file.bytes or "", validate=True) + except (binascii.Error, ValueError): + return None + + +class InlineAttachmentPersister: + """Write inline attachment bytes to S3 and register them for the session.""" + + def __init__( + self, + *, + repository: Optional[FileUploadRepository] = None, + s3_client: Any = None, + bucket_name: Optional[str] = None, + ) -> None: + self._repository = repository or get_file_upload_repository() + self._bucket = bucket_name or _bucket_name() + self._s3 = s3_client or boto3.client("s3", region_name=os.environ.get("AWS_REGION", "us-west-2")) + + async def persist( + self, + *, + user_id: str, + session_id: str, + files: Sequence[InlineFile], + source: str = INLINE_SOURCE, + ) -> Tuple[List[FileMetadata], List[PersistFailure]]: + """Persist ``files`` for ``session_id``. Never raises. + + Returns ``(persisted, failures)`` in attachment order. A failure is + per-file: one bad file does not take the others down, and nothing + here fails the turn — the caller decides what to tell the user. + """ + if not files: + return [], [] + + # One quota read for the batch, charged cumulatively so five files + # that each fit cannot together overshoot the cap. + try: + quota = await self._repository.get_user_quota(user_id) + used = int(quota.total_bytes) + except Exception: # noqa: BLE001 — a quota read failure must not block the turn + logger.warning("inline persist: quota lookup failed; proceeding without the cap", exc_info=True) + used = 0 + cap = _user_quota_bytes() + + plans: List[Tuple[Any, Optional[bytes], Optional[str]]] = [] + for f in files: + content_type = (f.content_type or "").lower() + if not is_allowed_mime_type(content_type): + plans.append((f, None, f"its type `{content_type or 'unknown'}` is not accepted")) + continue + raw = _decode(f) + if raw is None: + plans.append((f, None, "its contents were not valid base64")) + continue + limit = _max_size_for(f.filename, content_type) + if len(raw) > limit: + plans.append((f, None, f"it is larger than the {limit // (1024 * 1024)} MB limit")) + continue + if used + len(raw) > cap: + plans.append((f, None, "your file storage quota is full")) + continue + used += len(raw) + plans.append((f, raw, None)) + + results = await asyncio.gather( + *(self._store_one(user_id, session_id, f, raw, source) for f, raw, why in plans if raw is not None), + return_exceptions=False, + ) + stored = iter(results) + + persisted: List[FileMetadata] = [] + failures: List[PersistFailure] = [] + for f, raw, why in plans: + if raw is None: + failures.append(PersistFailure(file=f, reason=why or "it could not be stored")) + continue + meta = next(stored) + if meta is None: + failures.append(PersistFailure(file=f, reason="it could not be stored")) + else: + persisted.append(meta) + if persisted: + logger.info("inline persist: stored %d file(s) for session %s as %s", len(persisted), session_id, source) + if failures: + logger.warning( + "inline persist: %d file(s) not stored: %s", + len(failures), + [(x.file.filename, x.reason) for x in failures], + ) + return persisted, failures + + async def _store_one(self, user_id: str, session_id: str, f: InlineFile, raw: bytes, source: str) -> Optional[FileMetadata]: + upload_id = new_upload_id() + key = s3_key_for(user_id, session_id, upload_id, f.filename) + content_type = (f.content_type or "").lower() + try: + # boto3 is synchronous; keep the S3 round trip off the event loop so + # the stream's head-of-turn work is not blocked behind it. + await asyncio.to_thread( + self._s3.put_object, + Bucket=self._bucket, + Key=key, + Body=raw, + ContentType=content_type, + ) + except Exception: # noqa: BLE001 + logger.error("inline persist: S3 put failed for %s", f.filename, exc_info=True) + return None + + meta = FileMetadata( + upload_id=upload_id, + user_id=user_id, + session_id=session_id, + filename=f.filename, + mime_type=content_type, + size_bytes=len(raw), + s3_key=key, + s3_bucket=self._bucket, + status=FileStatus.READY, + source=source, + ) + try: + await self._repository.create_file(meta) + except Exception: # noqa: BLE001 + logger.error("inline persist: metadata write failed for %s", f.filename, exc_info=True) + # Do not leave an orphan object that no row points at. + try: + await asyncio.to_thread(self._s3.delete_object, Bucket=self._bucket, Key=key) + except Exception: # noqa: BLE001 + logger.warning("inline persist: orphan cleanup failed for %s", key, exc_info=True) + return None + try: + await self._repository.increment_quota(user_id, len(raw)) + except Exception: # noqa: BLE001 — the file is usable; the counter can drift by one file + logger.warning("inline persist: quota increment failed for %s", upload_id, exc_info=True) + return meta + + +_persister: Optional[InlineAttachmentPersister] = None + + +def get_inline_attachment_persister() -> InlineAttachmentPersister: + """Process-wide persister (one S3 client), overridable in tests.""" + global _persister + if _persister is None: + _persister = InlineAttachmentPersister() + return _persister diff --git a/backend/src/apis/shared/kb_backend/byte_cap.py b/backend/src/apis/shared/kb_backend/byte_cap.py index 92df7c895..3474f77f1 100644 --- a/backend/src/apis/shared/kb_backend/byte_cap.py +++ b/backend/src/apis/shared/kb_backend/byte_cap.py @@ -60,6 +60,17 @@ reservation instead of committing it. Both ``ADD`` writes commute, so a refund landing before its commit still nets to zero. +Re-ingesting a changed source +----------------------------- +A KB sync that finds its source changed overwrites the settled document's S3 +object in place, so the new version's size has to replace the old one in the +ledger. Only the difference moves: growth is reserved against the cap *before* +the new version is submitted (:func:`claim_reingest`), so a file that grew past +the cap is refused while its previous version is still being served, and on +completion ``committedBytes`` is re-stamped and the difference committed or +refunded (:func:`settle_reingest`). A delete mid-flight returns the growth +reservation through :func:`release_reingest_once`. + Guards against driving a counter negative ----------------------------------------- :func:`release` refuses to take ``reservedBytes`` below zero, and :func:`refund` @@ -88,7 +99,7 @@ import logging import os from decimal import Decimal -from typing import Optional +from typing import Any, Dict, Optional, Tuple from apis.shared.kb_backend.metrics import emit_count @@ -196,6 +207,32 @@ def object_size_bytes(bucket: str, key: str) -> int: return int(response["ContentLength"]) +def reserved_at_request(size_bytes: object, source_adapter_key: Optional[str]) -> int: + """The bytes a document reserved when it was requested, from its ``DOC#`` row. + + Only the interactive upload (``POST /upload-url``) reserves at request time, + and it reserves the ``sizeBytes`` it writes. Every other producer — a + file-source import, a web crawl, a KB sync — reserves nothing up front and is + the only kind of row that carries a ``sourceAdapterKey``. Those producers + still write ``sizeBytes``, though: an import backfills the downloaded size + before its ``PUT``, a crawl after it. So ``sizeBytes`` alone says nothing + about what was reserved. + + Reading it as a reservation is what corrupted prod: the ingestion consumer + committed an import's size out of ``reservedBytes`` it had never added to, so + the import never counted against the cap and the corpus reservation beside + it shrank by the same amount. On a knowledge base with nothing else + reserved, the counter goes negative. The failure paths had the mirror image, + releasing the size of an import that failed or was deleted in flight. + """ + if source_adapter_key: + return 0 + try: + return max(int(size_bytes or 0), 0) + except (TypeError, ValueError): + return 0 + + def reserve( assistant_id: str, app_kb_id: str, @@ -603,6 +640,153 @@ def refund_once(assistant_id: str, document_id: str) -> int: return int(response.get("Attributes", {}).get("committedBytes") or 0) +#: The ``DOC#`` attributes one in-flight re-ingest holds (:func:`claim_reingest`). +REINGEST_CLAIM_ATTRIBUTES = ( + "reingestHash", + "reingestBytes", + "reingestReservedBytes", + "reingestSubmittedAt", +) + + +def claim_reingest( + assistant_id: str, + document_id: str, + content_hash: str, + new_bytes: int, + reserved_bytes: int, +) -> Tuple[bool, int]: + """Claim the re-ingest of one staged version of an already-settled document. + + A KB sync that finds its source changed overwrites the document's S3 object in + place. The document already settled its bytes, so :func:`settle_once` cannot + account for the new version; this claim does. It stamps the version + (``reingestHash``, the ``stagedContentHash`` the sync wrote), its S3 size and + the growth the caller has just reserved for it, conditioned on no claim for + this version existing. That makes the reservation exactly-once per version + under redelivery: the caller reserves *before* claiming, and releases its own + reservation again when the claim is refused. + + Returns ``(claimed, superseded)``. ``superseded`` is the reservation of an + older version's claim this one replaced — a re-ingest that never completed — + and the caller must release it: nothing else ever will. Only one claim can + replace it, so it is released once. + + Refused for a row that is gone, ``deleting``, or not terminal: an upload still + in flight is its first ingestion's to settle. + """ + from botocore.exceptions import ClientError + + from apis.shared.kb_backend.records import RECORD_EXISTS + + try: + response = _table().update_item( + Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + UpdateExpression=( + "SET reingestHash = :h, reingestBytes = :b, reingestReservedBytes = :r " + "REMOVE reingestSubmittedAt" + ), + ConditionExpression=( + f"{RECORD_EXISTS} AND #status IN (:complete, :failed) " + f"AND (attribute_not_exists(reingestHash) OR reingestHash <> :h)" + ), + ExpressionAttributeNames={"#status": "status"}, + ExpressionAttributeValues={ + ":h": content_hash, + ":b": Decimal(new_bytes), + ":r": Decimal(reserved_bytes), + ":complete": "complete", + ":failed": "failed", + }, + ReturnValues="UPDATED_OLD", + ) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") == "ConditionalCheckFailedException": + return False, 0 + raise + old = response.get("Attributes") or {} + if not old.get("reingestHash"): + return True, 0 + return True, int(old.get("reingestReservedBytes") or 0) + + +def release_reingest_once( + assistant_id: str, + document_id: str, + content_hash: Optional[str] = None, +) -> int: + """Drop a document's re-ingest claim and return the bytes it had reserved. + + Returns the reservation for exactly one caller, 0 for everyone else. For a + re-ingest that ends without committing — Bedrock failed it, or the owner + deleted the document mid-flight — so its growth reservation does not leak. + ``content_hash`` restricts the drop to that version's claim. + + The completing path removes the claim in the same conditional write that + stamps the new ``committedBytes``, so a delete racing a completion either + finds the claim (and releases it) or finds the new stamp (and refunds it). + """ + from botocore.exceptions import ClientError + + from apis.shared.kb_backend.records import RECORD_EXISTS + + condition = f"{RECORD_EXISTS} AND attribute_exists(reingestHash)" + values: Dict[str, Any] = {} + if content_hash is not None: + condition += " AND reingestHash = :h" + values[":h"] = content_hash + kwargs: Dict[str, Any] = { + "Key": {"PK": f"AST#{assistant_id}", "SK": f"DOC#{document_id}"}, + "UpdateExpression": "REMOVE " + ", ".join(REINGEST_CLAIM_ATTRIBUTES), + "ConditionExpression": condition, + "ReturnValues": "ALL_OLD", + } + if values: + kwargs["ExpressionAttributeValues"] = values + try: + response = _table().update_item(**kwargs) + except ClientError as exc: + if exc.response.get("Error", {}).get("Code") == "ConditionalCheckFailedException": + return 0 + raise + return int((response.get("Attributes") or {}).get("reingestReservedBytes") or 0) + + +def settle_reingest( + assistant_id: str, + app_kb_id: str, + previous_bytes: int, + new_bytes: int, + reserved_bytes: int, +) -> None: + """Move the ledger from a document's old committed size to its re-ingested one. + + Called after the ``DOC#`` row's ``committedBytes`` has been re-stamped to + ``new_bytes`` — the stamp first, for the same reason :func:`record_commit` + comes before :func:`commit`. Growth was reserved when the re-ingest was + claimed, so it is committed out of that reservation; shrinkage is refunded. + + The growth can only differ from the reservation if ``committedBytes`` moved + between the claim's read and its write — an older version's re-ingest + completing in that window. Any reservation left over is released, and growth + beyond it is counted as stored without a cap check (:func:`add_stored`): the + bytes are already indexed, and refusing to count them would only hide them. + """ + growth = new_bytes - previous_bytes + committed = min(reserved_bytes, max(growth, 0)) + commit(assistant_id, app_kb_id, committed) + if reserved_bytes > committed: + release(assistant_id, app_kb_id, reserved_bytes - committed) + if growth > committed: + logger.warning( + f"re-ingest of a document in {assistant_id} grew {growth} bytes but reserved " + f"only {reserved_bytes}; counting the difference as stored" + ) + add_stored(assistant_id, app_kb_id, growth - committed) + elif growth < 0: + refund(assistant_id, app_kb_id, -growth) + + def release_snapshot(assistant_id: str, app_kb_id: str, n_bytes: int) -> bool: """Return a migration's whole-corpus reservation, once. diff --git a/backend/src/apis/shared/kb_backend/managed_backend.py b/backend/src/apis/shared/kb_backend/managed_backend.py index 5d7f6f39b..e07b56712 100644 --- a/backend/src/apis/shared/kb_backend/managed_backend.py +++ b/backend/src/apis/shared/kb_backend/managed_backend.py @@ -428,6 +428,8 @@ async def search( query: str, top_k: int = DEFAULT_TOP_K, retrieval_filter: Optional[Mapping[str, Any]] = None, + *, + record: Optional[Mapping[str, Any]] = None, ) -> List[Chunk]: """Retrieve up to ``top_k`` chunks, best first. @@ -439,8 +441,15 @@ async def search( The synchronous ``retrieve`` call runs off the event loop (Requirement 20.7): it was measured at 662–695 ms p50, which is long enough to matter to every other coroutine sharing the loop. + + ``record`` is the KB_Record the facade read moments ago to choose this + backend. Its ``awsKbId`` is used as-is rather than read a second time (the + same row, ~50ms on a turn's critical path); without one, or without an + ``awsKbId`` on it, the record is read here as before. """ - aws_kb_id, _ = await self._locate_async(kb_ref) + aws_kb_id = (record or {}).get("awsKbId") + if not aws_kb_id: + aws_kb_id, _ = await self._locate_async(kb_ref) client = self._runtime() payload = { diff --git a/backend/src/apis/shared/kb_backend/protocol.py b/backend/src/apis/shared/kb_backend/protocol.py index 9f2955c32..c24201266 100644 --- a/backend/src/apis/shared/kb_backend/protocol.py +++ b/backend/src/apis/shared/kb_backend/protocol.py @@ -42,7 +42,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Dict, List, Optional, Protocol, runtime_checkable +from typing import Any, Dict, List, Mapping, Optional, Protocol, runtime_checkable #: Parity contract, Requirement 3.1: both backends are asked for five chunks. #: Named here, above the seam, so neither adapter can drift from the other. @@ -136,13 +136,24 @@ class KnowledgeBaseBackend(Protocol): guard against a missing method, not a substitute for reading the protocol. """ - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: + async def search( + self, + kb_ref: str, + query: str, + top_k: int = DEFAULT_TOP_K, + *, + record: Optional[Mapping[str, Any]] = None, + ) -> List[Chunk]: """Return up to ``top_k`` chunks, best first, scored by relevance. Ordering is the backend's: both underlying APIs return results ranked best-first, and an adapter re-sorting them would be inventing a ranking rather than reporting one. What an adapter *must* guarantee is that its ``relevance`` values agree with the order it returns. + + ``record`` is the KB_Record the caller has already read for this search, + if any. An adapter that needs it may use it instead of reading it again; + one that does not ignores it. """ ... diff --git a/backend/src/apis/shared/kb_backend/records.py b/backend/src/apis/shared/kb_backend/records.py index adb397eb7..4ef76eb56 100644 --- a/backend/src/apis/shared/kb_backend/records.py +++ b/backend/src/apis/shared/kb_backend/records.py @@ -162,9 +162,20 @@ def work_pk(state: str) -> str: def _table(): - import boto3 + """The assistants table, on the process-cached DynamoDB resource. + + A fresh ``boto3.resource`` per call cost ~45–55ms on its first request in the + Runtime (a new connection pool), against ~4ms on a reused one — and a turn's + knowledge base search reads this table on its critical path. ``AWS_REGION`` is + passed explicitly (it is what boto3 would resolve anyway) so this shares its + cache entry, and its warm connection, with the facade's document-status + lookups on the same table. + """ + from apis.shared.aws_clients import get_dynamodb_table - return boto3.resource("dynamodb").Table(os.environ["DYNAMODB_ASSISTANTS_TABLE_NAME"]) + return get_dynamodb_table( + os.environ["DYNAMODB_ASSISTANTS_TABLE_NAME"], os.environ.get("AWS_REGION") + ) # ── Model ──────────────────────────────────────────────────────────────────── diff --git a/backend/src/apis/shared/kb_backend/s3vectors_backend.py b/backend/src/apis/shared/kb_backend/s3vectors_backend.py index d34e5b73f..c175995ba 100644 --- a/backend/src/apis/shared/kb_backend/s3vectors_backend.py +++ b/backend/src/apis/shared/kb_backend/s3vectors_backend.py @@ -46,7 +46,7 @@ from __future__ import annotations import logging -from typing import Any, Dict, List +from typing import Any, Dict, List, Mapping, Optional from apis.shared.kb_backend.protocol import ( DEFAULT_TOP_K, @@ -68,7 +68,14 @@ class S3VectorsBackend: Stateless, so a shared instance is safe and no client is held across calls. """ - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: + async def search( + self, + kb_ref: str, + query: str, + top_k: int = DEFAULT_TOP_K, + *, + record: Optional[Mapping[str, Any]] = None, + ) -> List[Chunk]: """Query the index and return chunks scored by relevance, best first. ``top_k`` is accepted to satisfy the protocol but the underlying query @@ -78,6 +85,9 @@ async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> L incomplete document from silently shrinking a five-chunk answer to four. Slicing here instead would change that, so this returns what the index returned. + + ``record`` is accepted to satisfy the protocol and ignored: the global index + is addressed by ``kb_ref`` alone. """ from apis.shared.embeddings.bedrock_embeddings import search_assistant_knowledgebase diff --git a/backend/src/apis/shared/models/context_window.py b/backend/src/apis/shared/models/context_window.py index 0d45f52ce..278b08107 100644 --- a/backend/src/apis/shared/models/context_window.py +++ b/backend/src/apis/shared/models/context_window.py @@ -48,10 +48,14 @@ def sdk_context_window(model_id: Optional[str]) -> Optional[int]: """Strands' built-in context-window table for ``model_id``, or ``None``. Covers Anthropic-on-Bedrock and handles the cross-region prefix strip - (``us.anthropic.claude-sonnet-4-6`` resolves via ``anthropic.…``). It is - **empty for every non-Anthropic id we curate** — Astra, the GPT-5.6 - family, Kimi K3 and DeepSeek all return ``None`` — so this is a partial - safety net, not a second source of truth. + (``us.anthropic.claude-sonnet-4-6`` resolves via ``anthropic.…``). Since + strands-agents 1.56 the strip also walks nested prefixes, so the hosted + OpenAI ids resolve too: ``us.openai.gpt-6-astra`` and the GPT-5.6 family + return their full 1,050,000 window. That is the real window but **not** + our pricing cap — the curated rows' ``maxInputTokens: 272_000`` still wins + in :func:`resolve_context_window`, and a row with the field *absent* would + now fall through to 1,050,000. Kimi K3 and DeepSeek still return ``None``, + so this remains a partial safety net, not a second source of truth. ⚠️ ``strands.models._defaults`` is a private module on an experimental surface. Import failure is treated as "no fallback available", never as diff --git a/backend/src/apis/shared/models/managed_models.py b/backend/src/apis/shared/models/managed_models.py index 490555883..4fe8e93ff 100644 --- a/backend/src/apis/shared/models/managed_models.py +++ b/backend/src/apis/shared/models/managed_models.py @@ -298,6 +298,8 @@ async def _create_managed_model_cloud(model_data: ManagedModelCreate, table_name output_price_per_million_tokens=model_data.output_price_per_million_tokens, cache_write_price_per_million_tokens=model_data.cache_write_price_per_million_tokens, cache_read_price_per_million_tokens=model_data.cache_read_price_per_million_tokens, + speech_input_price_per_million_tokens=model_data.speech_input_price_per_million_tokens, + speech_output_price_per_million_tokens=model_data.speech_output_price_per_million_tokens, knowledge_cutoff_date=model_data.knowledge_cutoff_date, supports_caching=_resolve_supports_caching(model_data.supports_caching, model_data.provider), is_default=model_data.is_default, @@ -346,6 +348,10 @@ async def _create_managed_model_cloud(model_data: ManagedModelCreate, table_name item['cacheWritePricePerMillionTokens'] = model_data.cache_write_price_per_million_tokens if model_data.cache_read_price_per_million_tokens is not None: item['cacheReadPricePerMillionTokens'] = model_data.cache_read_price_per_million_tokens + if model_data.speech_input_price_per_million_tokens is not None: + item['speechInputPricePerMillionTokens'] = model_data.speech_input_price_per_million_tokens + if model_data.speech_output_price_per_million_tokens is not None: + item['speechOutputPricePerMillionTokens'] = model_data.speech_output_price_per_million_tokens if model_data.knowledge_cutoff_date is not None: item['knowledgeCutoffDate'] = model_data.knowledge_cutoff_date if model_data.short_description: diff --git a/backend/src/apis/shared/models/modalities.py b/backend/src/apis/shared/models/modalities.py new file mode 100644 index 000000000..500f4da9b --- /dev/null +++ b/backend/src/apis/shared/models/modalities.py @@ -0,0 +1,21 @@ +"""Modality predicates over catalog rows. + +A speech-to-speech model (Nova 2 Sonic) lives in the catalog so voice +sessions price, but it answers only ``InvokeModelWithBidirectionalStream``: +offered as a chat model it would fail every turn. Every surface that lists +models *for chat* filters with this predicate; cost and admin surfaces do not. +""" + +from __future__ import annotations + +from typing import Iterable, Optional, Protocol + + +class _HasOutputModalities(Protocol): + output_modalities: Optional[Iterable[str]] + + +def is_speech_model(model: _HasOutputModalities) -> bool: + """Whether the row's output modalities include speech (case-insensitive).""" + modalities = getattr(model, "output_modalities", None) or [] + return any(str(m).upper() == "SPEECH" for m in modalities) diff --git a/backend/src/apis/shared/models/models.py b/backend/src/apis/shared/models/models.py index cf7b82435..9623567b3 100644 --- a/backend/src/apis/shared/models/models.py +++ b/backend/src/apis/shared/models/models.py @@ -261,6 +261,22 @@ def _validate_icon_slug(cls, value: Optional[str]) -> Optional[str]: ge=0, description="Price per million tokens read from cache (Bedrock only, ~90% discount)" ) + # Speech-to-speech models (Nova 2 Sonic) bill speech tokens on their own + # rate card, an order of magnitude above their text rates. `input`/`output` + # above are the TEXT rates for such a model; these two carry the speech + # rates. Absent on every text model. + speech_input_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechInputPricePerMillionTokens", + ge=0, + description="Price per million SPEECH input tokens (speech-to-speech models only)" + ) + speech_output_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechOutputPricePerMillionTokens", + ge=0, + description="Price per million SPEECH output tokens (speech-to-speech models only)" + ) knowledge_cutoff_date: Optional[str] = Field(None, alias="knowledgeCutoffDate") supports_caching: Optional[bool] = Field( None, @@ -418,6 +434,22 @@ def _validate_icon_slug(cls, value: Optional[str]) -> Optional[str]: ge=0, description="Price per million tokens read from cache (Bedrock only, ~90% discount)" ) + # Speech-to-speech models (Nova 2 Sonic) bill speech tokens on their own + # rate card, an order of magnitude above their text rates. `input`/`output` + # above are the TEXT rates for such a model; these two carry the speech + # rates. Absent on every text model. + speech_input_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechInputPricePerMillionTokens", + ge=0, + description="Price per million SPEECH input tokens (speech-to-speech models only)" + ) + speech_output_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechOutputPricePerMillionTokens", + ge=0, + description="Price per million SPEECH output tokens (speech-to-speech models only)" + ) knowledge_cutoff_date: Optional[str] = Field(None, alias="knowledgeCutoffDate") supports_caching: Optional[bool] = Field( None, @@ -565,6 +597,16 @@ class ManagedModel(BaseModel): alias="cacheReadPricePerMillionTokens", description="Price per million tokens read from cache (Bedrock only, ~90% discount)" ) + speech_input_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechInputPricePerMillionTokens", + description="Price per million SPEECH input tokens (speech-to-speech models only)" + ) + speech_output_price_per_million_tokens: Optional[float] = Field( + None, + alias="speechOutputPricePerMillionTokens", + description="Price per million SPEECH output tokens (speech-to-speech models only)" + ) knowledge_cutoff_date: Optional[str] = Field(None, alias="knowledgeCutoffDate") supports_caching: bool = Field( True, diff --git a/backend/src/apis/shared/models/usage_normalization.py b/backend/src/apis/shared/models/usage_normalization.py index 8a4b766f1..0b55f9562 100644 --- a/backend/src/apis/shared/models/usage_normalization.py +++ b/backend/src/apis/shared/models/usage_normalization.py @@ -31,12 +31,15 @@ plain Converse-shaped usage dicts and need no provider awareness. The wrapper is also where ``cache_write_tokens`` re-enters the pipeline. -Strands never reads it off the Responses usage object, so -``cacheWriteInputTokens`` is structurally 0 for GPT-5.6 — which pins -``wastedUsd`` at $0 and makes the 1.25x write premium invisible, the same -blind spot that let the compaction spiral run unnoticed. An upstream patch is -in flight; until it lands (and on any older pin) this mapping is the only -source of the field. +Through strands-agents 1.57.1 the SDK never read it off the Responses usage +object, so ``cacheWriteInputTokens`` was structurally 0 for GPT-5.6 — which +pinned ``wastedUsd`` at $0 and made the 1.25x write premium invisible, the +same blind spot that let the compaction spiral run unnoticed. Both OpenAI +paths now report it upstream (Chat Completions in 1.57 via #4361, Responses +in 1.57.2 via #4193), each beside an ``inputTokens`` that is still inclusive, +so :func:`normalize_usage` still subtracts it once like any other write +bucket. The recovery here now writes the value the SDK already set; it is +redundant on the current pin, and deleting it is a separate cleanup. ⚠️ :func:`normalize_usage` is **not idempotent** for the OpenAI family — it subtracts. Apply it exactly once per usage payload, at the model seam. Do not @@ -165,9 +168,10 @@ def _normalize_metadata_chunk(event: Mapping[str, Any], chunk: Any) -> Any: if not isinstance(usage, dict): return chunk - # Recover the field Strands drops, before disjointness is computed — the - # written tokens are part of the inclusive `input_tokens` and have to come - # out of it too, or they are billed at input + 1.25x write. + # Recover the write bucket before disjointness is computed — the written + # tokens are part of the inclusive `input_tokens` and have to come out of + # it too, or they are billed at input + 1.25x write. Strands >=1.57.2 sets + # the same value itself; this assignment is then a same-value overwrite. cache_write = openai_cache_write_tokens(event.get("data")) if cache_write: usage["cacheWriteInputTokens"] = cache_write @@ -231,7 +235,7 @@ def _formatter(self: Any, event: Dict[str, Any], **kwargs: Any) -> Any: f"{base_cls.__name__} that reports disjoint token buckets.\n\n" "See apis/shared/models/usage_normalization.py — OpenAI's " "inclusive `input_tokens` is double-billed by our cost paths " - "otherwise, and Strands drops `cache_write_tokens` entirely." + "otherwise." ), }, ) diff --git a/backend/src/apis/shared/observability/build_stages.py b/backend/src/apis/shared/observability/build_stages.py index eedb04b80..9370714b6 100644 --- a/backend/src/apis/shared/observability/build_stages.py +++ b/backend/src/apis/shared/observability/build_stages.py @@ -45,18 +45,30 @@ import contextvars import logging -from typing import Callable, Optional +from typing import Any, Callable, Optional, Tuple logger = logging.getLogger(__name__) -_recorder: contextvars.ContextVar[Optional[Callable[[str], None]]] = contextvars.ContextVar( +_Recorders = Tuple[Optional[Callable[[str], None]], Optional[Callable[[str, Any], None]]] + +# One var holding both callables, so a single token installs and restores them +# together — a stage recorder from one turn can never pair with another turn's +# detail recorder. +_recorder: contextvars.ContextVar[Optional[_Recorders]] = contextvars.ContextVar( "agent_build_stage_recorder", default=None ) -def set_stage_recorder(recorder: Optional[Callable[[str], None]]) -> contextvars.Token: - """Install the recorder for the current context. Returns a reset token.""" - return _recorder.set(recorder) +def set_stage_recorder( + recorder: Optional[Callable[[str], None]], + detail_recorder: Optional[Callable[[str, Any], None]] = None, +) -> contextvars.Token: + """Install the recorders for the current context. Returns a reset token. + + ``detail_recorder`` receives log-only properties (``record_detail``), such + as the per-server timings behind ``tools.mcp``. + """ + return _recorder.set((recorder, detail_recorder)) def reset_stage_recorder(token: contextvars.Token) -> None: @@ -76,8 +88,21 @@ def mark_stage(stage: str) -> None: the thing it measures is worse than no measurement. """ try: - recorder = _recorder.get() - if recorder is not None: - recorder(stage) + recorders = _recorder.get() + if recorders is not None and recorders[0] is not None: + recorders[0](stage) except Exception: # noqa: BLE001 logger.debug("Build stage mark skipped", exc_info=True) + + +def record_detail(key: str, value: Any) -> None: + """Attach a log-only property to the turn's ``turn_prelude`` line. + + Same contract as ``mark_stage``: a no-op without a recorder, never raises. + """ + try: + recorders = _recorder.get() + if recorders is not None and recorders[1] is not None: + recorders[1](key, value) + except Exception: # noqa: BLE001 + logger.debug("Build detail skipped", exc_info=True) diff --git a/backend/src/apis/shared/sessions/messages.py b/backend/src/apis/shared/sessions/messages.py index 13a2d785c..2572fa053 100644 --- a/backend/src/apis/shared/sessions/messages.py +++ b/backend/src/apis/shared/sessions/messages.py @@ -569,6 +569,71 @@ async def fetch_tool_summaries(): raise +# The Memory SDK keeps its own records in a session's event stream beside the +# messages: a ``SESSION`` event when the session is created and ``AGENT`` events +# for agent state, each tagged with this metadata key. Message events carry no +# such tag, so "is there a message?" is "is there an event without it?". +_SDK_STATE_METADATA_KEY = "stateType" + + +def _first_message_event(memory_id: str, actor_id: str, session_id: str) -> bool: + """Whether the session's event stream holds any message event. Blocking.""" + from apis.shared.aws_clients import get_client + + client = get_client("bedrock-agentcore", region_name=os.environ.get("AWS_REGION", "us-west-2")) + params: Dict[str, Any] = { + "memoryId": memory_id, + "actorId": actor_id, + "sessionId": session_id, + "maxResults": 1, + "includePayloads": False, + "filter": { + "eventMetadata": [ + {"left": {"metadataKey": _SDK_STATE_METADATA_KEY}, "operator": "NOT_EXISTS"} + ] + }, + } + # The filter is applied server-side, but a page may still come back empty + # with a continuation token; only an empty page WITHOUT one means "none". + while True: + response = client.list_events(**params) + if response.get("events"): + return True + token = response.get("nextToken") + if not token: + return False + params["nextToken"] = token + + +async def session_has_messages(session_id: str, user_id: str) -> bool: + """True when the session has at least one persisted message, from any agent. + + An existence check, and priced like one: one ``ListEvents`` with + ``maxResults=1``, no payloads, filtered server-side to events that are not + the SDK's own state records. It replaces ``get_messages(limit=1)``, which + read the WHOLE history plus its metadata, interrupts, UI resources and + tool summaries to return one message — ~540ms on a new conversation's first + agent turn on dev, growing with history — and, by constructing an + ``AgentCoreMemorySessionManager``, wrote a ``SESSION`` event into a session + that did not have one yet. The agent's own session manager creates that + event when the turn's agent is built, as it does on every plain turn. + + Fails open: if the read itself fails this logs and returns False — the + same answer the old path gave, because the SDK swallowed a failed + ``ListEvents`` into an empty history. + """ + import asyncio + + memory_id = os.environ.get("AGENTCORE_MEMORY_ID") + if not memory_id: + raise ValueError("AGENTCORE_MEMORY_ID environment variable not set") + try: + return await asyncio.to_thread(_first_message_event, memory_id, user_id, session_id) + except Exception as e: # noqa: BLE001 - an unanswerable check must not fail the turn + logger.warning(f"Could not check session {session_id} for messages: {type(e).__name__}: {e}") + return False + + async def get_messages(session_id: str, user_id: str, limit: Optional[int] = None, next_token: Optional[str] = None) -> MessagesListResponse: """ Retrieve messages for a session and user with pagination support. diff --git a/backend/src/apis/shared/sessions/metadata.py b/backend/src/apis/shared/sessions/metadata.py index 1ef7d990f..268f97ff9 100644 --- a/backend/src/apis/shared/sessions/metadata.py +++ b/backend/src/apis/shared/sessions/metadata.py @@ -1322,8 +1322,13 @@ async def update_session_title(session_id: str, user_id: str, title: str) -> Non Uses a targeted ``UpdateExpression`` so it can run concurrently with ``store_session_metadata`` (which does a full-row merge) without racing - on other fields like ``messageCount`` or ``lastMessageAt``. Looks up the - current SK via the GSI because the SK contains a timestamp. + on other fields like ``messageCount`` or ``lastMessageAt``. + + One round trip in the common case: every row is born with the static + ``S#{session_id}`` SK (issue #175), and this runs on a session's first + turn, so the key is known without a lookup. A row still on a legacy + timestamped SK fails the ``attribute_exists`` guard and falls back to + resolving the SK through the GSI. No-op when the session row doesn't exist (preview sessions, sessions deleted mid-turn). @@ -1336,9 +1341,23 @@ async def update_session_title(session_id: str, user_id: str, title: str) -> Non raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") try: + from botocore.exceptions import ClientError table = get_dynamodb_table(sessions_metadata_table) + try: + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": _static_session_sk(session_id)}, + UpdateExpression="SET title = :t", + ConditionExpression="attribute_exists(PK)", + ExpressionAttributeValues={":t": title}, + ) + logger.info(f"💾 Updated title for session {session_id}") + return + except ClientError as e: + if e.response.get("Error", {}).get("Code") != "ConditionalCheckFailedException": + raise + existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: logger.info(f"update_session_title: session {session_id} not found, skipping") diff --git a/backend/src/apis/shared/tool_summaries/summarizer.py b/backend/src/apis/shared/tool_summaries/summarizer.py index e677dbe71..777c1bd6c 100644 --- a/backend/src/apis/shared/tool_summaries/summarizer.py +++ b/backend/src/apis/shared/tool_summaries/summarizer.py @@ -35,6 +35,8 @@ import os from typing import Any, Dict, List, Optional +from apis.shared.aws_clients import get_client + logger = logging.getLogger(__name__) # Nova Micro: the cheapest Bedrock model that reliably follows a one-line @@ -140,6 +142,16 @@ def _clean(text: str) -> str: return summary.rstrip(".").strip() +def _converse(region: str, **kwargs: Any) -> Dict[str, Any]: + """Run in a worker thread, so a first-use client build stays off the event loop. + + The client is process-cached: one per batch paid botocore's service-model + load (~250ms the first time in a process) and, every time, a fresh + connection pool — a new TCP+TLS handshake per summary. + """ + return get_client("bedrock-runtime", region).converse(**kwargs) + + async def summarize_tool_batch(calls: List[Dict[str, Any]]) -> Optional[str]: """Summarize one finished tool batch, or return ``None``. @@ -157,19 +169,12 @@ async def summarize_tool_batch(calls: List[Dict[str, Any]]) -> Optional[str]: return None try: - import boto3 - except ImportError: # pragma: no cover - dev without boto3 - return None - - try: - region = os.environ.get("AWS_REGION", "us-west-2") - client = boto3.client("bedrock-runtime", region_name=region) - # boto3's converse() is synchronous — awaited inline it would block # the event loop for the whole Nova round trip, stalling the agent # stream this task runs concurrently with. response = await asyncio.to_thread( - client.converse, + _converse, + os.environ.get("AWS_REGION", "us-west-2"), modelId=_MODEL_ID, messages=[{"role": "user", "content": [{"text": _build_prompt(calls)}]}], system=[{"text": _SUMMARY_SYSTEM_PROMPT}], @@ -187,9 +192,16 @@ async def summarize_tool_batch(calls: List[Dict[str, Any]]) -> Optional[str]: # deterministic formatter speak. (Defensive: the dangling quotes seen # on dev turned out to be `_unwrap_quotes`, not truncation, but # nothing guarded this boundary and a fragment must never persist.) - if response.get("stopReason") == "max_tokens": + # Any other non-`end_turn` stop (a guardrail or content-filter + # refusal, an unknown reason) is dropped too: its text may be the + # refusal itself, which is no summary of the batch. + stop_reason = response.get("stopReason") + if stop_reason == "max_tokens": logger.debug("Tool-batch summary hit the token ceiling; discarding") return None + if stop_reason != "end_turn": + logger.debug("Tool-batch summary stopped with stopReason=%s; discarding", stop_reason) + return None summary = _clean(response["output"]["message"]["content"][0]["text"]) if not summary: diff --git a/backend/src/apis/shared/user_settings/models.py b/backend/src/apis/shared/user_settings/models.py index 08f6b87f8..678be5666 100644 --- a/backend/src/apis/shared/user_settings/models.py +++ b/backend/src/apis/shared/user_settings/models.py @@ -1,9 +1,11 @@ """Domain models for user settings.""" -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, field_validator from typing import Annotated, Optional from pydantic import Field +from apis.shared.voice_catalog import get_voice + # Personal instructions ride in the system prompt of every conversation the user has, # so they are kept short: about a thousand tokens, paid (cached) on every turn. MAX_PERSONAL_INSTRUCTIONS_CHARS = 4_000 @@ -15,6 +17,9 @@ class UserSettings(BaseModel): default_model_id: Annotated[Optional[str], Field(alias="defaultModelId")] = None personal_instructions: Annotated[Optional[str], Field(alias="personalInstructions")] = None + # The Nova 2 Sonic voice the user hears in voice mode. `None` means the + # platform default (`apis.shared.voice_catalog.DEFAULT_VOICE_ID`). + voice_id: Annotated[Optional[str], Field(alias="voiceId")] = None class UserSettingsUpdate(BaseModel): @@ -26,3 +31,20 @@ class UserSettingsUpdate(BaseModel): personal_instructions: Annotated[ Optional[str], Field(alias="personalInstructions", max_length=MAX_PERSONAL_INSTRUCTIONS_CHARS) ] = None + # Blank clears it (back to the platform default). Anything else must name + # a voice in the catalog — a stored id Bedrock rejects would break every + # later voice session, so it is refused at the door with a 422. + voice_id: Annotated[Optional[str], Field(alias="voiceId")] = None + + @field_validator("voice_id") + @classmethod + def _validate_voice_id(cls, value: Optional[str]) -> Optional[str]: + if value is None: + return None + stripped = value.strip() + if not stripped: + return "" + voice = get_voice(stripped) + if voice is None: + raise ValueError(f"Unknown voice '{stripped}'") + return voice.voice_id diff --git a/backend/src/apis/shared/user_settings/repository.py b/backend/src/apis/shared/user_settings/repository.py index ffcaba5cb..48988aa68 100644 --- a/backend/src/apis/shared/user_settings/repository.py +++ b/backend/src/apis/shared/user_settings/repository.py @@ -12,6 +12,7 @@ DEFAULT_SETTINGS = { "defaultModelId": None, "personalInstructions": None, + "voiceId": None, } @@ -82,6 +83,7 @@ async def get_settings(self, user_id: str) -> dict: return { "defaultModelId": item.get("defaultModelId"), "personalInstructions": item.get("personalInstructions"), + "voiceId": item.get("voiceId"), } except ClientError as e: logger.error(f"Error getting settings for user {user_id}: {e}") diff --git a/backend/src/apis/shared/voice_catalog.py b/backend/src/apis/shared/voice_catalog.py new file mode 100644 index 000000000..94211fb8f --- /dev/null +++ b/backend/src/apis/shared/voice_catalog.py @@ -0,0 +1,93 @@ +"""The Nova 2 Sonic voice catalog. + +One list, shared by every consumer that has to agree on what a valid voice is: +the voice agent (which sends the id to Bedrock), the voice WebSocket route +(which accepts one from the SPA's config frame) and the user-settings route +(which persists a preference). Lives in ``apis.shared`` because ``app_api`` +and ``agents`` may not import from each other. + +Source: the Nova 2 user guide, "Language support and multilingual +capabilities" (verified 2026-09-30). Every language has a feminine-sounding +and, where Amazon ships one, a masculine-sounding voice. ``tiffany`` and +``matthew`` are the two **polyglot** voices: they speak every supported +language, which is why ``tiffany`` is the default — a user who switches +language mid-conversation keeps the same voice instead of getting a reply in +an accent the language does not have. + +The ``gender`` field is *presentation*, not identity: some languages (Hindi, +Portuguese, French, Italian, Spanish) conjugate the speaker's own gender, and +Amazon's prompt guidance is to tell the model which form matches the voice so +"I am tired" comes out as ``cansada`` for ``carolina`` and ``cansado`` for +``leo``. The voice agent folds that into its system prompt. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Dict, List, Literal, Optional + +VoiceGender = Literal["feminine", "masculine"] + + +@dataclass(frozen=True) +class NovaSonicVoice: + """One selectable Nova 2 Sonic voice.""" + + voice_id: str + name: str + language: str + locale: str + gender: VoiceGender + polyglot: bool = False + + +NOVA_SONIC_VOICES: List[NovaSonicVoice] = [ + NovaSonicVoice("tiffany", "Tiffany", "English (US)", "en-US", "feminine", polyglot=True), + NovaSonicVoice("matthew", "Matthew", "English (US)", "en-US", "masculine", polyglot=True), + NovaSonicVoice("amy", "Amy", "English (UK)", "en-GB", "feminine"), + NovaSonicVoice("olivia", "Olivia", "English (Australia)", "en-AU", "feminine"), + NovaSonicVoice("kiara", "Kiara", "English (India) / Hindi", "en-IN", "feminine"), + NovaSonicVoice("arjun", "Arjun", "English (India) / Hindi", "en-IN", "masculine"), + NovaSonicVoice("ambre", "Ambre", "French", "fr-FR", "feminine"), + NovaSonicVoice("florian", "Florian", "French", "fr-FR", "masculine"), + NovaSonicVoice("beatrice", "Beatrice", "Italian", "it-IT", "feminine"), + NovaSonicVoice("lorenzo", "Lorenzo", "Italian", "it-IT", "masculine"), + NovaSonicVoice("tina", "Tina", "German", "de-DE", "feminine"), + NovaSonicVoice("lennart", "Lennart", "German", "de-DE", "masculine"), + NovaSonicVoice("lupe", "Lupe", "Spanish (US)", "es-US", "feminine"), + NovaSonicVoice("carlos", "Carlos", "Spanish (US)", "es-US", "masculine"), + NovaSonicVoice("carolina", "Carolina", "Portuguese (Brazil)", "pt-BR", "feminine"), + NovaSonicVoice("leo", "Leo", "Portuguese (Brazil)", "pt-BR", "masculine"), +] + +DEFAULT_VOICE_ID = "tiffany" + +VOICES_BY_ID: Dict[str, NovaSonicVoice] = {v.voice_id: v for v in NOVA_SONIC_VOICES} +VOICE_IDS = frozenset(VOICES_BY_ID) + + +def get_voice(voice_id: Optional[str]) -> Optional[NovaSonicVoice]: + """Look a voice up by id, case-insensitively. ``None`` for anything unknown.""" + if not voice_id or not isinstance(voice_id, str): + return None + return VOICES_BY_ID.get(voice_id.strip().lower()) + + +def is_valid_voice_id(voice_id: Optional[str]) -> bool: + """Whether ``voice_id`` names a voice Nova 2 Sonic will accept.""" + return get_voice(voice_id) is not None + + +def resolve_voice_id(requested: Optional[str], fallback: str = DEFAULT_VOICE_ID) -> str: + """Normalise a requested voice id, falling back to a known-good one. + + A bad id must never reach ``promptStart``: Bedrock rejects the session and + the user gets a dead voice button with no explanation. Unknown or blank + input resolves to ``fallback``, which is itself validated so a misconfigured + ``NOVA_SONIC_VOICE`` env var cannot smuggle a bad id in through the back. + """ + voice = get_voice(requested) + if voice is not None: + return voice.voice_id + fallback_voice = get_voice(fallback) + return fallback_voice.voice_id if fallback_voice else DEFAULT_VOICE_ID diff --git a/backend/src/lambdas/scheduled_runs_dispatcher/requirements.txt b/backend/src/lambdas/scheduled_runs_dispatcher/requirements.txt index 7d1d18e39..cf261eef9 100644 --- a/backend/src/lambdas/scheduled_runs_dispatcher/requirements.txt +++ b/backend/src/lambdas/scheduled_runs_dispatcher/requirements.txt @@ -1,9 +1,9 @@ # scheduled-runs Lambda image dependencies (backend/Dockerfile.scheduled-runs). # Exact pins per repo policy; versions match backend/uv.lock. -boto3==1.43.68 +boto3==1.43.103 pydantic==2.12.5 httpx==0.28.1 -bedrock-agentcore==1.21.0 +bedrock-agentcore==1.24.0 # Worker-only transitive deps (both handlers share ONE image, and the # Dockerfile installs THIS file): the worker imports apis.shared.harness -> # apis.shared.sessions_bff, whose cookie.py needs cryptography (AESGCM) and diff --git a/backend/src/lambdas/scheduled_runs_worker/requirements.txt b/backend/src/lambdas/scheduled_runs_worker/requirements.txt index ffd05d98c..71ce9baeb 100644 --- a/backend/src/lambdas/scheduled_runs_worker/requirements.txt +++ b/backend/src/lambdas/scheduled_runs_worker/requirements.txt @@ -1,9 +1,9 @@ # scheduled-runs Lambda image dependencies (backend/Dockerfile.scheduled-runs). # Exact pins per repo policy; versions match backend/uv.lock. -boto3==1.43.68 +boto3==1.43.103 pydantic==2.12.5 httpx==0.28.1 -bedrock-agentcore==1.21.0 +bedrock-agentcore==1.24.0 # Worker transitive deps: apis.shared.harness -> apis.shared.sessions_bff, # whose cookie.py needs cryptography (AESGCM) and cache.py needs cachetools # (TTLCache). Kept in sync with the dispatcher requirements.txt, which is the diff --git a/backend/tests/agents/main_agent/core/test_bedrock_model_shared_session.py b/backend/tests/agents/main_agent/core/test_bedrock_model_shared_session.py new file mode 100644 index 000000000..f26f4132c --- /dev/null +++ b/backend/tests/agents/main_agent/core/test_bedrock_model_shared_session.py @@ -0,0 +1,80 @@ +"""With `AGENT_BUILD_SHARED_SESSION_ENABLED` on (the default), Strands' BedrockModel +is built on the process-wide boto3 session instead of the fresh `boto3.Session()` +it would construct itself. + +`BedrockModel.__init__` raises when `region_name` and `boto_session` are both +given (strands-agents 1.55.0), so the config must never send both. Real objects, +no network: constructing a client opens no socket. +""" + +import pytest + +from agents.main_agent.core.agent_factory import AgentFactory +from agents.main_agent.core.model_config import ModelConfig +from apis.shared import aws_clients + +MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0" + + +@pytest.fixture(autouse=True) +def _region_and_fresh_session(monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + +class TestDefaultOn: + @pytest.fixture(autouse=True) + def _default(self, monkeypatch): + monkeypatch.delenv("AGENT_BUILD_SHARED_SESSION_ENABLED", raising=False) + + def test_config_carries_the_shared_session_and_no_region(self): + config = ModelConfig(model_id=MODEL_ID).to_bedrock_config() + + assert config["boto_session"] is aws_clients.shared_boto_session() + assert "region_name" not in config + + def test_the_model_client_is_the_shared_sessions_client(self): + first = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + second = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + + assert first.client is second.client, "two models, one bedrock-runtime client" + assert first.client.meta.config.max_pool_connections == aws_clients.SHARED_SESSION_MAX_POOL_CONNECTIONS + + def test_the_shared_session_resolves_the_same_region_as_a_fresh_one(self, monkeypatch): + """Strands resolves the region from the session it is given, and a + fresh `boto3.Session()` reads the same environment the shared one + does, so switching the sharing off must not move the region.""" + shared = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + own = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + + assert shared.client.meta.region_name == own.client.meta.region_name + + def test_a_deliberate_region_next_to_the_session_is_still_refused(self): + """Pins the Strands contract the config is written around.""" + from strands.models import BedrockModel + + with pytest.raises(ValueError, match="both"): + BedrockModel(model_id=MODEL_ID, region_name="us-west-2", boto_session=aws_clients.shared_boto_session()) + + +class TestKillSwitch: + @pytest.mark.parametrize("value", ["false", "FALSE", " False "]) + def test_config_carries_no_session(self, monkeypatch, value): + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", value) + + config = ModelConfig(model_id=MODEL_ID).to_bedrock_config() + + assert "boto_session" not in config + assert "region_name" not in config + + def test_the_model_builds_its_own_client(self, monkeypatch): + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + + first = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + second = AgentFactory._create_bedrock_model(ModelConfig(model_id=MODEL_ID)) + + assert first.client is not second.client + assert aws_clients._shared_session is None, "switched off, nothing touches the shared session" diff --git a/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py b/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py index 360bd3001..0ae3ddeb0 100644 --- a/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py +++ b/backend/tests/agents/main_agent/integrations/test_external_mcp_client.py @@ -902,3 +902,235 @@ def test_take_pending_consents_drains_and_is_per_user(self): self.PROVIDER: "https://consent.example/b" } assert integration.take_pending_consents("carol") == {} + + +class TestLoadExternalToolsTimings: + """Per-server timings behind `agent_build.tools.mcp` (turn-path P1a): one + entry per server so one slow server is distinguishable from many.""" + + @pytest.mark.asyncio + async def test_one_entry_per_server_with_its_outcome(self): + integration = ExternalMCPIntegration() + when = datetime(2025, 1, 1, tzinfo=timezone.utc) + repo = SimpleNamespace( + get_tool=AsyncMock( + side_effect=[ + _fake_tool(when, tool_id="calendar"), + _fake_tool(when, tool_id="gmail"), + None, + ] + ) + ) + bad_client = SimpleNamespace( + load_tools=AsyncMock(side_effect=RuntimeError("connection refused")) + ) + good_client = SimpleNamespace(load_tools=AsyncMock(return_value=[])) + timings: list = [] + + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), patch( + "agents.main_agent.integrations.external_mcp_client.create_external_mcp_client", + side_effect=[bad_client, good_client], + ): + await integration.load_external_tools( + ["calendar", "gmail", "retired"], timings=timings + ) + + assert [(t["id"], t["outcome"]) for t in timings] == [ + ("calendar", "dropped"), + ("gmail", "loaded"), + ("retired", "skipped"), + ] + for entry in timings[:2]: + assert {"catalogMs", "preflightMs", "totalMs"} <= set(entry) + assert entry["totalMs"] >= entry["preflightMs"] + # Never reached a pre-flight, so it must not claim one. + assert "preflightMs" not in timings[2] + + @pytest.mark.asyncio + async def test_a_cached_client_reports_no_preflight(self): + integration = ExternalMCPIntegration() + tool = _fake_tool(datetime(2025, 1, 1, tzinfo=timezone.utc)) + repo = SimpleNamespace(get_tool=AsyncMock(return_value=tool)) + client = SimpleNamespace(load_tools=AsyncMock(return_value=[])) + timings: list = [] + + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ), patch( + "agents.main_agent.integrations.external_mcp_client.create_external_mcp_client", + return_value=client, + ): + await integration.load_external_tools(["gmail"]) + await integration.load_external_tools(["gmail"], timings=timings) + + assert timings[0]["outcome"] == "cached" + assert "preflightMs" not in timings[0] + + @pytest.mark.asyncio + async def test_a_catalog_error_is_recorded_not_raised(self): + integration = ExternalMCPIntegration() + repo = SimpleNamespace(get_tool=AsyncMock(side_effect=RuntimeError("ddb"))) + timings: list = [] + + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ): + result = await integration.load_external_tools(["gmail"], timings=timings) + + assert result == [] + assert timings[0]["outcome"] == "error" + assert "totalMs" in timings[0] + + +class TestLoadExternalToolsConcurrentPreflight: + """Several servers pre-flight together (turn-path spec §5 P3). + + Strands' `load_tools` blocks its thread for the whole `initialize` + handshake, so the fakes here block synchronously too — an `await + asyncio.sleep` fake would overlap under `gather` and prove nothing. On dev + two cold Lambda-URL servers cost 9.5s + 12s in series. What must hold: + the pre-flights overlap, the client list (and so the tool order in the + cacheable `toolConfig`) stays in catalog order, a lone server keeps the + inline call, and `MCP_PARALLEL_PREFLIGHT_ENABLED=false` is the old loop. + """ + + @staticmethod + def _repo(*tool_ids): + tools = { + tid: _fake_tool(datetime(2025, 1, 1, tzinfo=timezone.utc), tool_id=tid) + for tid in tool_ids + } + return SimpleNamespace(get_tool=AsyncMock(side_effect=lambda tid: tools[tid])) + + @staticmethod + def _blocking_client(on_load): + async def _load_tools(): + on_load() + return [] + + return SimpleNamespace(load_tools=_load_tools) + + async def _load(self, integration, tool_ids, clients_by_id, **kwargs): + with patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=self._repo(*tool_ids), + ), patch( + "agents.main_agent.integrations.external_mcp_client.create_external_mcp_client", + side_effect=lambda config, tool_definition, **kw: clients_by_id[tool_definition.tool_id], + ): + return await integration.load_external_tools(tool_ids, **kwargs) + + @pytest.mark.asyncio + async def test_preflights_overlap(self, monkeypatch): + import threading + + monkeypatch.delenv("MCP_PARALLEL_PREFLIGHT_ENABLED", raising=False) + # Each pre-flight blocks until BOTH are in flight: serial loading + # would break the barrier instead of passing it. + barrier = threading.Barrier(2, timeout=5) + clients = { + "canvas": self._blocking_client(barrier.wait), + "class_search": self._blocking_client(barrier.wait), + } + + result = await self._load(ExternalMCPIntegration(), ["canvas", "class_search"], clients) + + assert result == [clients["canvas"], clients["class_search"]] + + @pytest.mark.asyncio + async def test_catalog_order_survives_a_slower_first_server(self, monkeypatch): + import threading + import time as _time + + monkeypatch.delenv("MCP_PARALLEL_PREFLIGHT_ENABLED", raising=False) + finished = [] + lock = threading.Lock() + + def _after(name, delay): + def _run(): + _time.sleep(delay) + with lock: + finished.append(name) + return _run + + clients = { + "slow": self._blocking_client(_after("slow", 0.2)), + "fast": self._blocking_client(_after("fast", 0.0)), + } + timings: list = [] + + result = await self._load( + ExternalMCPIntegration(), ["slow", "fast"], clients, timings=timings + ) + + assert finished == ["fast", "slow"] + assert result == [clients["slow"], clients["fast"]] + assert [t["id"] for t in timings] == ["slow", "fast"] + assert all(t["outcome"] == "loaded" for t in timings) + + @pytest.mark.asyncio + async def test_a_lone_server_preflights_on_the_calling_thread(self, monkeypatch): + import threading + + monkeypatch.delenv("MCP_PARALLEL_PREFLIGHT_ENABLED", raising=False) + seen = [] + clients = {"canvas": self._blocking_client(lambda: seen.append(threading.current_thread()))} + + await self._load(ExternalMCPIntegration(), ["canvas"], clients) + + assert seen == [threading.current_thread()] + + @pytest.mark.asyncio + async def test_context_reaches_the_preflight_thread(self, monkeypatch): + """`BedrockAgentCoreContext` (workload token, OAuth callback URL) is + contextvar-backed; the pre-flight must see what the build set.""" + import contextvars + + monkeypatch.delenv("MCP_PARALLEL_PREFLIGHT_ENABLED", raising=False) + marker = contextvars.ContextVar("marker", default=None) + seen = [] + clients = { + "a": self._blocking_client(lambda: seen.append(marker.get())), + "b": self._blocking_client(lambda: seen.append(marker.get())), + } + + marker.set("from-the-build") + await self._load(ExternalMCPIntegration(), ["a", "b"], clients) + + assert seen == ["from-the-build", "from-the-build"] + + @pytest.mark.asyncio + async def test_kill_switch_loads_one_at_a_time(self, monkeypatch): + import threading + + monkeypatch.setenv("MCP_PARALLEL_PREFLIGHT_ENABLED", "false") + active = 0 + peak = 0 + lock = threading.Lock() + + def _track(): + nonlocal active, peak + with lock: + active += 1 + peak = max(peak, active) + with lock: + active -= 1 + + seen_threads = [] + clients = { + tid: self._blocking_client( + lambda: (_track(), seen_threads.append(threading.current_thread())) + ) + for tid in ("a", "b", "c") + } + + result = await self._load(ExternalMCPIntegration(), ["a", "b", "c"], clients) + + assert result == [clients["a"], clients["b"], clients["c"]] + assert peak == 1 + assert set(seen_threads) == {threading.current_thread()} diff --git a/backend/tests/agents/main_agent/session/test_agent_status_hook.py b/backend/tests/agents/main_agent/session/test_agent_status_hook.py index 56894d9bc..c563fbbd6 100644 --- a/backend/tests/agents/main_agent/session/test_agent_status_hook.py +++ b/backend/tests/agents/main_agent/session/test_agent_status_hook.py @@ -294,3 +294,34 @@ def test_status_queue_is_capped(hook): hook._on_before_model_call(MagicMock()) assert len(hook.drain_statuses()) == _MAX_QUEUED_STATUSES + + +# -- 6. first-model-call stamp (turn-path P1b) ------------------------------ + + +def test_first_model_call_is_stamped_once_per_turn(hook, monkeypatch): + clock = iter([10.0, 11.0, 12.0]) + monkeypatch.setattr( + "agents.main_agent.session.hooks.agent_status.time.perf_counter", + lambda: next(clock), + ) + hook._on_turn_start(MagicMock()) + hook._on_before_model_call(MagicMock()) + hook._on_before_model_call(MagicMock()) # cycle 2 must not move it + + assert hook.first_model_call_at == 10.0 + + hook._on_turn_start(MagicMock()) + assert hook.first_model_call_at is None + hook._on_before_model_call(MagicMock()) + assert hook.first_model_call_at == 11.0 + + +def test_first_model_call_is_stamped_with_narration_off(hook, monkeypatch): + """A measurement, not narration: the kill switch silences the status line, + not the first-token clock.""" + monkeypatch.setenv("AGENT_STATUS_ENABLED", "false") + hook._on_before_model_call(MagicMock()) + + assert hook.first_model_call_at is not None + assert hook.drain_statuses() == [] diff --git a/backend/tests/agents/main_agent/session/test_bidi_agent_guards.py b/backend/tests/agents/main_agent/session/test_bidi_agent_guards.py new file mode 100644 index 000000000..a5b7aaea8 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_bidi_agent_guards.py @@ -0,0 +1,148 @@ +"""TurnBasedSessionManager's text-only work must not run for the voice BidiAgent. + +Since strands-agents 1.56 a ``BidiAgent`` drives the ordinary session hooks +(``AgentInitializedEvent`` -> ``initialize``, ``MessageAddedEvent`` -> +``retrieve_customer_context``) rather than the dedicated Bidi callbacks it used +through 1.55. Without these guards the voice agent's history would be sliced by +the text agent's session-level compaction checkpoint, and every voice transcript +would trigger a long-term-memory retrieval spliced into the live Bidi history. +""" + +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from strands.bidi import BidiAgent + +from agents.main_agent.session import turn_based_session_manager as tbsm + + +def _bidi_agent(messages): + agent = BidiAgent.__new__(BidiAgent) + agent.messages = messages + agent.agent_id = "voice" + return agent + + +def test_bidi_detection(): + assert tbsm._is_bidi_agent(_bidi_agent([])) is True + assert tbsm._is_bidi_agent(SimpleNamespace(messages=[], agent_id="default")) is False + + +def test_retrieve_customer_context_skips_a_bidi_agent(make_session_manager): + mgr = make_session_manager() + mgr.config.retrieval_config = {"/ns": SimpleNamespace(top_k=5, relevance_score=0.1, strategy_id=None)} + client = MagicMock() + mgr._retrieval_client = client + message = {"role": "user", "content": [{"text": "what's my name?"}]} + + mgr.retrieve_customer_context(SimpleNamespace(agent=_bidi_agent([message]))) + + client.retrieve_memory_records.assert_not_called() + assert message == {"role": "user", "content": [{"text": "what's my name?"}]} + + +def test_prefetch_skips_a_bidi_agent(make_session_manager, monkeypatch): + """The prefetch (#1400) rides the same MessageAddedEvent the voice agent fires. + + retrieve_customer_context never consumes a lookup for a BidiAgent, so a + prefetch started for one is a paid Memory retrieval per transcript, discarded. + """ + monkeypatch.delenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", raising=False) + pool = MagicMock() + monkeypatch.setattr(tbsm, "_ltm_prefetch_pool", lambda: pool) + mgr = make_session_manager() + mgr.config.retrieval_config = {"/ns": SimpleNamespace(top_k=5, relevance_score=0.1, strategy_id=None)} + message = {"role": "user", "content": [{"text": "what's my name?"}]} + + mgr._prefetch_customer_context(SimpleNamespace(agent=_bidi_agent([message]), message=message)) + pool.submit.assert_not_called() + + # The same message from a text agent is prefetched, so the guard is what stopped it. + mgr._prefetch_customer_context(SimpleNamespace(agent=SimpleNamespace(messages=[message]), message=message)) + pool.submit.assert_called_once() + + +def test_initialize_restores_a_bidi_agent_without_text_processing(make_session_manager): + mgr = make_session_manager() + mgr.read_agent = MagicMock(return_value=SimpleNamespace()) + restored = [ + {"role": "user", "content": [{"text": "hi"}]}, + {"role": "assistant", "content": [{"text": "hello"}]}, + ] + mgr.list_messages = MagicMock(return_value=[MagicMock(to_message=lambda m=m: m) for m in restored]) + mgr._apply_compaction = MagicMock(side_effect=AssertionError("text compaction ran for voice")) + mgr._repair_restored_history = MagicMock(side_effect=AssertionError("text repair ran for voice")) + agent = _bidi_agent([]) + + mgr.initialize(agent) + + assert [m["role"] for m in agent.messages] == ["user", "assistant"] + mgr._apply_compaction.assert_not_called() + mgr._repair_restored_history.assert_not_called() + + +async def _drive_transcript(mgr, agent, persisted): + """Replay strands 1.57's transcript lifecycle through a real HookRegistry.""" + from strands.hooks import HookRegistry, MessageAddedEvent, MessageUpdatedEvent + + registry = HookRegistry() + mgr.register_hooks(registry) + + shell = {"role": "assistant", "content": [], "tracking_id": "t-1"} + agent.messages.append(shell) + await registry.invoke_callbacks_async(MessageAddedEvent(agent=agent, message=shell)) + + filled = {"role": "assistant", "content": [{"text": "The capital of France is Paris."}], "tracking_id": "t-1"} + await registry.invoke_callbacks_async(MessageUpdatedEvent(agent, "t-1", filled)) + + +def _persisting_manager(make_session_manager, monkeypatch, async_mode): + from bedrock_agentcore.memory.integrations.strands.session_manager import AgentCoreMemorySessionManager + + persisted = [] + monkeypatch.setattr( + AgentCoreMemorySessionManager, + "append_message", + lambda self, message, agent, **kw: persisted.append(message), + ) + mgr = make_session_manager() + mgr.config.async_mode = async_mode + mgr.config.batch_size = 1 + mgr.sync_agent = MagicMock() + return mgr, persisted + + + +@pytest.mark.asyncio +@pytest.mark.parametrize("async_mode", [True, False]) +async def test_voice_transcript_is_persisted_once_with_its_text(make_session_manager, monkeypatch, async_mode): + """1.57 appends a transcript empty and fills it via MessageUpdatedEvent. + + The empty shell is skipped by append_message; without the update handler + nothing was ever saved, so voice worked live and the conversation came + back empty on reload. + """ + mgr, persisted = _persisting_manager(make_session_manager, monkeypatch, async_mode) + agent = _bidi_agent([]) + + await _drive_transcript(mgr, agent, persisted) + + assert [m["content"] for m in persisted] == [[{"text": "The capital of France is Paris."}]] + + +@pytest.mark.asyncio +async def test_message_update_from_a_text_agent_is_not_persisted(make_session_manager, monkeypatch): + from strands.hooks import HookRegistry, MessageUpdatedEvent + + mgr, persisted = _persisting_manager(make_session_manager, monkeypatch, async_mode=True) + registry = HookRegistry() + mgr.register_hooks(registry) + text_agent = SimpleNamespace(messages=[], agent_id="default") + + await registry.invoke_callbacks_async( + MessageUpdatedEvent(text_agent, "t-1", {"role": "assistant", "content": [{"text": "x"}]}) + ) + + assert persisted == [] diff --git a/backend/tests/agents/main_agent/session/test_compaction_summary.py b/backend/tests/agents/main_agent/session/test_compaction_summary.py index 242ba08b1..12c278f3f 100644 --- a/backend/tests/agents/main_agent/session/test_compaction_summary.py +++ b/backend/tests/agents/main_agent/session/test_compaction_summary.py @@ -16,6 +16,7 @@ compress_with_model, truncate_records_newest_first, ) +from apis.shared import aws_clients from .conftest import make_conversation @@ -23,6 +24,14 @@ BUDGET = 100 # tokens → 400 chars +@pytest.fixture(autouse=True) +def _fresh_bedrock_client(): + """The Bedrock client is cached per process; each test builds its own.""" + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + @pytest.fixture def bedrock(monkeypatch): """Patch boto3 so no test reaches Bedrock; returns the converse mock.""" @@ -90,6 +99,34 @@ async def test_model_ceiling_hit_with_no_complete_line_falls_back(self, bedrock) assert result.outcome == "truncated_after_model" assert approx_tokens(result.text) <= BUDGET + @pytest.mark.asyncio + @pytest.mark.parametrize("stop", ["guardrail_intervened", "content_filtered", "tool_use", "something_new", None]) + async def test_a_refused_generation_falls_back_and_is_never_the_summary(self, bedrock, stop): + """Only a completed (or salvageable) generation becomes the checkpoint. + + A refusal persisted as the summary would sit in the cacheable history + until the next cut. + """ + bedrock.return_value = _model_reply("Sorry, the model cannot answer this.", stop=stop) + records = ["old " * 100, "new " * 50] + result = await bound_summary(records, BUDGET, model_enabled=True, model_id="m") + assert result.outcome == "truncated_after_model" + assert "Sorry" not in result.text and result.text.startswith("new") + assert await compress_with_model(records, BUDGET, model_id="m") is None + + @pytest.mark.asyncio + async def test_calls_reuse_one_bedrock_client_per_region(self, bedrock): + """A client per call paid botocore's model load and a fresh TLS handshake.""" + bedrock.return_value = _model_reply("summary") + factory = sys.modules["boto3"].client + + await compress_with_model(["r" * 900], BUDGET, model_id="m", region="us-west-2") + await compress_with_model(["r" * 900], BUDGET, model_id="m", region="us-west-2") + await compress_with_model(["r" * 900], BUDGET, model_id="m", region="us-east-1") + + assert [c.kwargs["region_name"] for c in factory.call_args_list] == ["us-west-2", "us-east-1"] + assert bedrock.call_count == 3 + @pytest.mark.asyncio async def test_model_overshoot_is_tail_trimmed(self, bedrock): bedrock.return_value = _model_reply("y" * 2000 + "END") @@ -390,6 +427,27 @@ async def test_extraction_ceiling_hit_keeps_the_complete_lines(self, bedrock): assert result.outcome == "extract_then_compress" assert "- PRJ-4417" in result.text and "- PRJ-44\n" not in result.text + @pytest.mark.asyncio + async def test_refused_extraction_keeps_the_narrative_without_a_third_call(self, bedrock): + bedrock.side_effect = _route( + _model_reply("Sorry, the model cannot answer this.", stop="guardrail_intervened"), + _model_reply("plain compressed summary"), + ) + result = await _extract(["r" * 2000]) + assert result.outcome == "model" + assert result.text == "plain compressed summary" + assert bedrock.call_count == 2 + + @pytest.mark.asyncio + async def test_refused_narrative_keeps_the_pinned_block_and_truncates(self, bedrock): + bedrock.side_effect = _route( + _model_reply(PINNED), + _model_reply("Sorry, the model cannot answer this.", stop="guardrail_intervened"), + ) + result = await _extract(["old " * 300, "newest record"]) + assert result.outcome == "extract_then_truncate" + assert "Sorry" not in result.text and result.text.endswith("newest record") + @pytest.mark.asyncio async def test_narrative_ceiling_hit_keeps_its_complete_lines(self, bedrock): """A cut-off narrative is salvaged, not swapped for raw records (extract_then_truncate).""" diff --git a/backend/tests/agents/main_agent/session/test_display_text_hook.py b/backend/tests/agents/main_agent/session/test_display_text_hook.py index 8a5d6dc30..05c125324 100644 --- a/backend/tests/agents/main_agent/session/test_display_text_hook.py +++ b/backend/tests/agents/main_agent/session/test_display_text_hook.py @@ -224,3 +224,37 @@ def test_registers_for_message_added(self): registered = {call.args[0] for call in registry.add_callback.call_args_list} assert MessageAddedEvent in registered + + +class TestLazyIndex: + """The coordinator arms with ``HistoryCount.resolve`` so the history read + behind the index runs alongside the turn; the hook awaits it at the write.""" + + @pytest.mark.asyncio + async def test_resolves_an_async_index_at_the_write(self, store): + resolver = AsyncMock(return_value=7) + hook = DisplayTextHook() + hook.arm(session_id="s1", user_id="u1", message_index=resolver, display_text="typed") + + resolver.assert_not_awaited() + await hook.write_display_text(_event(_user_message())) + + store.assert_awaited_once_with( + session_id="s1", user_id="u1", message_id=7, display_text="typed" + ) + assert hook.wrote_this_turn is True + + @pytest.mark.asyncio + async def test_a_failed_resolve_is_a_failed_write_not_a_failed_turn(self, store): + hook = DisplayTextHook() + hook.arm( + session_id="s1", + user_id="u1", + message_index=AsyncMock(side_effect=RuntimeError("boom")), + display_text="typed", + ) + + await hook.write_display_text(_event(_user_message())) + + store.assert_not_awaited() + assert hook.wrote_this_turn is False diff --git a/backend/tests/agents/main_agent/session/test_memory_retrieval_prefetch.py b/backend/tests/agents/main_agent/session/test_memory_retrieval_prefetch.py new file mode 100644 index 000000000..db943fea8 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_memory_retrieval_prefetch.py @@ -0,0 +1,276 @@ +"""The long-term-memory prefetch (docs/specs/turn-path-ttft.md P4a). + +On ``MessageAddedEvent`` the SDK awaits two Memory writes and only then runs +the lookup: three network calls in series before the model is called, 450ms +of a 1687ms warm first token on dev. The prefetch starts the lookup first and +applies it where it always was. + +What is expensive to get wrong is not the speed-up but the bytes. The lookup +prepends a context block to the LIVE user message after the SDK has persisted +it, so the persisted message has no block. Restore rebuilds history from the +persisted copy, and restored history must be byte-stable for the prompt cache. +These tests drive the SDK's own async-mode wiring through a real Strands +``HookRegistry`` and pin both copies, byte for byte, against the switch-off +path. +""" + +import asyncio +import json +import threading +from types import SimpleNamespace + +import pytest +from strands.hooks import HookRegistry, MessageAddedEvent + +from agents.main_agent.session import turn_based_session_manager as tbsm + +PREFS = "/strategies/pref-1/actors/{actorId}" +FACTS = "/strategies/sem-1/actors/{actorId}" + + +class _GatedClient: + """Signals when a lookup starts; holds it until the writes are done. + + That makes overlap a yes/no question rather than a timing threshold: if + the lookup only starts after the writes, ``append_message`` never sees it + start. + """ + + def __init__(self, writes_done: threading.Event) -> None: + self.started = threading.Event() + self.writes_done = writes_done + self.calls = 0 + + def retrieve_memory_records(self, **kwargs): + self.calls += 1 + self.started.set() + self.writes_done.wait(timeout=5) + return {"memoryRecordSummaries": [{"content": {"text": f"fact from {kwargs['namespacePath']}"}, "score": 0.9}]} + + +def _manager(make_session_manager, client): + mgr = make_session_manager() + mgr.config.async_mode = True + mgr.config.batch_size = 1 + mgr.config.context_tag = "user_context" + # One namespace: two would make the block's line order depend on which + # finished first, a pre-existing property that is not what this pins. + mgr.config.retrieval_config = { + FACTS: SimpleNamespace(top_k=5, relevance_score=0.5, strategy_id=None), + } + mgr._retrieval_client = client + return mgr + + +async def _run_turn(mgr, client, *, writes_done): + """One user message through the real registry. Returns what was persisted, + whether the lookup had started by the time ``append_message`` returned, + and the live message afterwards.""" + persisted = {} + + def append_message(message, agent, **kwargs): + # Serialize at write time — what reaches Memory is this string. + persisted["message"] = json.dumps(message, sort_keys=True) + persisted["lookup_started_during_write"] = client.started.wait(timeout=2) + + def sync_agent(agent, **kwargs): + writes_done.set() + + mgr.append_message = append_message + mgr.sync_agent = sync_agent + + registry = HookRegistry() + mgr.register_hooks(registry) + + message = {"role": "user", "content": [{"text": "what is my name?"}]} + agent = SimpleNamespace(agent_id="default", messages=[message]) + await registry.invoke_callbacks_async(MessageAddedEvent(agent=agent, message=message)) + return persisted, json.dumps(message, sort_keys=True) + + +@pytest.fixture +def prefetch_on(monkeypatch): + monkeypatch.delenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", raising=False) + + +class TestThroughTheSdkWiring: + @pytest.mark.asyncio + async def test_the_lookup_overlaps_the_writes(self, make_session_manager, prefetch_on): + writes_done = threading.Event() + client = _GatedClient(writes_done) + mgr = _manager(make_session_manager, client) + + persisted, _ = await _run_turn(mgr, client, writes_done=writes_done) + + assert persisted["lookup_started_during_write"] is True + assert client.calls == 1 # prefetched, not fetched again inline + + @pytest.mark.asyncio + async def test_persisted_message_has_no_context_block_and_live_one_does( + self, make_session_manager, prefetch_on + ): + writes_done = threading.Event() + client = _GatedClient(writes_done) + mgr = _manager(make_session_manager, client) + + persisted, live = await _run_turn(mgr, client, writes_done=writes_done) + + assert "user_context" not in persisted["message"] + assert json.loads(live)["content"][0]["text"].startswith("") + assert json.loads(live)["content"][-1] == {"text": "what is my name?"} + + @pytest.mark.asyncio + async def test_bytes_match_the_switch_off_path(self, make_session_manager, monkeypatch): + """The whole contract in one assertion: with the prefetch on, both + copies are byte-identical to what the inline path produces.""" + results = {} + for flag in ("true", "false"): + monkeypatch.setenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", flag) + writes_done = threading.Event() + client = _GatedClient(writes_done) + mgr = _manager(make_session_manager, client) + persisted, live = await _run_turn(mgr, client, writes_done=writes_done) + results[flag] = (persisted["message"], live) + + assert results["true"] == results["false"] + + @pytest.mark.asyncio + async def test_switch_off_fetches_inline_after_the_writes(self, make_session_manager, monkeypatch): + monkeypatch.setenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", "false") + writes_done = threading.Event() + client = _GatedClient(writes_done) + mgr = _manager(make_session_manager, client) + # Without overlap the write waits out its full 2s for a lookup that + # cannot start yet; shorten that wait for this case only. + client.started.wait = lambda timeout=None: client.started.is_set() + + persisted, _ = await _run_turn(mgr, client, writes_done=writes_done) + + assert persisted["lookup_started_during_write"] is False + assert client.calls == 1 + + +class _RecordingPool: + def __init__(self): + self.submitted = [] + + def submit(self, fn, *args): + from concurrent.futures import Future + + self.submitted.append(args) + future = Future() + future.set_result(fn(*args)) + return future + + +def _event(message): + return SimpleNamespace(agent=SimpleNamespace(messages=[message]), message=message) + + +class TestPrefetchEligibility: + @pytest.fixture + def pool(self, monkeypatch, prefetch_on): + pool = _RecordingPool() + monkeypatch.setattr(tbsm, "_ltm_prefetch_pool", lambda: pool) + return pool + + def _mgr(self, make_session_manager): + client = _GatedClient(threading.Event()) + client.writes_done.set() + return _manager(make_session_manager, client) + + @pytest.mark.parametrize( + "message", + [ + {"role": "assistant", "content": [{"text": "hello"}]}, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t1", "content": []}}]}, + {"role": "user", "content": []}, + ], + ) + def test_only_a_text_user_message_is_prefetched(self, make_session_manager, pool, message): + mgr = self._mgr(make_session_manager) + + mgr._prefetch_customer_context(_event(message)) + + assert pool.submitted == [] + + def test_a_cancelled_session_is_not_prefetched(self, make_session_manager, pool): + mgr = self._mgr(make_session_manager) + mgr.cancelled = True + + mgr._prefetch_customer_context(_event({"role": "user", "content": [{"text": "hi"}]})) + + assert pool.submitted == [] + + def test_the_query_is_the_capped_user_text(self, make_session_manager, pool): + mgr = self._mgr(make_session_manager) + text = "x" * (tbsm.MEMORY_RETRIEVAL_QUERY_MAX_CHARS + 50) + + mgr._prefetch_customer_context(_event({"role": "user", "content": [{"text": text}]})) + + assert pool.submitted == [(text[: tbsm.MEMORY_RETRIEVAL_QUERY_MAX_CHARS],)] + + +class TestTakingThePrefetch: + def _mgr(self, make_session_manager): + client = _GatedClient(threading.Event()) + client.writes_done.set() + return _manager(make_session_manager, client), client + + def test_a_prefetch_for_another_message_is_never_applied(self, make_session_manager, prefetch_on): + """A turn that died between the two callbacks leaves a prefetch behind. + It must not land on the next message; that one fetches inline.""" + mgr, client = self._mgr(make_session_manager) + from concurrent.futures import Future + + stale = Future() + stale.set_result(["a fact for a different question"]) + mgr._ltm_prefetch = ({"role": "user", "content": [{"text": "old"}]}, stale, 0.0) + message = {"role": "user", "content": [{"text": "new"}]} + + mgr.retrieve_customer_context(_event(message)) + + assert "different question" not in message["content"][0]["text"] + assert client.calls == 1 + assert mgr._ltm_prefetch is None + + def test_a_failed_prefetch_leaves_the_turn_without_context(self, make_session_manager, prefetch_on, caplog): + mgr, client = self._mgr(make_session_manager) + from concurrent.futures import Future + + failed = Future() + failed.set_exception(RuntimeError("pool down")) + message = {"role": "user", "content": [{"text": "hi"}]} + mgr._ltm_prefetch = (message, failed, 0.0) + + mgr.retrieve_customer_context(_event(message)) # must not raise + + assert message["content"] == [{"text": "hi"}] + assert "memory retrieval prefetch failed" in caplog.text + + def test_the_wait_is_logged_without_content(self, make_session_manager, prefetch_on, caplog): + import logging + + caplog.set_level(logging.INFO, logger=tbsm.__name__) + mgr, _ = self._mgr(make_session_manager) + message = {"role": "user", "content": [{"text": "a private question"}]} + mgr._prefetch_customer_context(_event(message)) + + mgr.retrieve_customer_context(_event(message)) + + lines = [r.getMessage() for r in caplog.records if "prefetch waitedMs" in r.getMessage()] + assert len(lines) == 1 + assert "private" not in lines[0] + + +def test_register_hooks_puts_the_prefetch_first(make_session_manager): + """Strands runs a hook's callbacks in registration order; the prefetch + only overlaps the writes if it runs before the SDK's persist callback.""" + mgr = _manager(make_session_manager, _GatedClient(threading.Event())) + registry = HookRegistry() + + mgr.register_hooks(registry) + + callbacks = list(registry.get_callbacks_for(MessageAddedEvent(agent=SimpleNamespace(), message={}))) + assert callbacks[0] == mgr._prefetch_customer_context + assert len(callbacks) == 3 # prefetch, the SDK's persist, the SDK's lookup diff --git a/backend/tests/agents/main_agent/session/test_session_factory_shared_clients.py b/backend/tests/agents/main_agent/session/test_session_factory_shared_clients.py new file mode 100644 index 000000000..4b45140f9 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_session_factory_shared_clients.py @@ -0,0 +1,215 @@ +"""AgentCore Memory session managers share one set of boto3 clients. + +The SDK's ``AgentCoreMemorySessionManager.__init__`` built a ``MemoryClient`` +(fresh session, two clients) and then a second fresh session and two more +clients, on every construction. That was ~360ms of CPU per session manager on +the event loop, plus a cold connection pool. These tests build REAL +``TurnBasedSessionManager`` instances through the factory, with only the SDK's +network calls stubbed, so they fail if an SDK upgrade stops going through the +seams the factory relies on. The session itself lives in +``apis.shared.aws_clients`` (``shared_boto_session``), where warm-up builds it. +""" + +import threading +from concurrent.futures import ThreadPoolExecutor +from typing import Any, List +from unittest.mock import MagicMock, patch + +import boto3 +import pytest + +from agents.main_agent.session import session_factory as factory +from apis.shared import aws_clients +from bedrock_agentcore.memory.integrations.strands import session_manager as sdk + + +@pytest.fixture +def memory_env(monkeypatch): + """Real construction, no network: the SDK's session read/create are stubbed.""" + monkeypatch.setenv("AGENTCORE_MEMORY_ID", "mem-test") + monkeypatch.setenv("AWS_REGION", "us-west-2") + monkeypatch.delenv("AGENT_BUILD_SHARED_SESSION_ENABLED", raising=False) # default on + monkeypatch.setattr(factory, "_discover_strategy_ids", lambda memory_id, region, **kwargs: (None, None, None)) + monkeypatch.setattr(sdk.AgentCoreMemorySessionManager, "read_session", lambda self, session_id, **k: None) + monkeypatch.setattr(sdk.AgentCoreMemorySessionManager, "create_session", lambda self, session, **k: session) + # A fresh shared session per test, so client counts start from zero. + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + +@pytest.fixture +def client_builds(monkeypatch) -> List[str]: + """Every boto3 client actually constructed, by service name.""" + built: List[str] = [] + original = boto3.Session.client + + def counting_client(self, service_name, *args, **kwargs): + built.append(service_name) + return original(self, service_name, *args, **kwargs) + + monkeypatch.setattr(boto3.Session, "client", counting_client) + return built + + +def _build(session_id: str) -> Any: + return factory.SessionFactory.create_session_manager(session_id=session_id, user_id="user-1") + + +class TestSharedClients: + def test_the_sdk_builds_its_memory_client_through_the_shared_seam(self): + assert sdk.MemoryClient is factory._SharedSessionMemoryClient + + def test_later_session_managers_build_no_clients(self, memory_env, client_builds): + first = _build("session-a") + built_by_first = len(client_builds) + + second = _build("session-b") + + assert built_by_first > 0 + assert len(client_builds) == built_by_first + assert second.memory_client.gmdp_client is first.memory_client.gmdp_client + assert second.memory_client.gmcp_client is first.memory_client.gmcp_client + + def test_each_manager_keeps_its_own_session_state(self, memory_env, client_builds): + """Only the clients are shared; the session identity is per manager.""" + first = _build("session-a") + second = _build("session-b") + + assert first.session_id == "session-a" + assert second.session_id == "session-b" + assert first.config is not second.config + + def test_shared_clients_get_a_bigger_pool_and_keep_the_sdk_user_agent(self, memory_env): + manager = _build("session-a") + config = manager.memory_client.gmdp_client.meta.config + + assert config.max_pool_connections == aws_clients.SHARED_SESSION_MAX_POOL_CONNECTIONS + assert "strands-agents" in (config.user_agent_extra or "") + + def test_concurrent_first_builds_share_one_client(self, memory_env, client_builds): + """Two builds racing for the first client must still end up with one.""" + barrier = threading.Barrier(4) + + def build(i: int) -> Any: + barrier.wait() + return _build(f"session-{i}") + + with ThreadPoolExecutor(max_workers=4) as pool: + managers = list(pool.map(build, range(4))) + + assert len({id(m.memory_client.gmdp_client) for m in managers}) == 1 + + +class TestKillSwitch: + def test_off_keeps_the_sdks_per_manager_clients(self, memory_env, client_builds, monkeypatch): + """`AGENT_BUILD_SHARED_SESSION_ENABLED=false` restores the SDK's own sessions.""" + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + + first = _build("session-a") + built_by_first = len(client_builds) + second = _build("session-b") + + assert len(client_builds) == 2 * built_by_first + assert second.memory_client.gmdp_client is not first.memory_client.gmdp_client + + def test_the_memory_client_seam_is_inert_outside_a_shared_construction(self, memory_env): + """The rebinding in the SDK module must not change a MemoryClient built + anywhere else (e.g. `_discover_strategy_ids`).""" + client = sdk.MemoryClient(region_name="us-west-2") + assert client.gmdp_client is not aws_clients.shared_boto_session().client( + "bedrock-agentcore", region_name="us-west-2" + ) + + +class TestClientReusingSession: + def test_explicit_credentials_are_never_shared(self): + session = aws_clients.ClientReusingSession() + kwargs = dict(region_name="us-west-2", aws_access_key_id="a", aws_secret_access_key="b") + + assert session.client("sts", **kwargs) is not session.client("sts", **kwargs) + + def test_different_configs_get_different_clients(self): + from botocore.config import Config + + session = aws_clients.ClientReusingSession() + a = session.client("sts", region_name="us-west-2", config=Config(user_agent_extra="a")) + b = session.client("sts", region_name="us-west-2", config=Config(user_agent_extra="b")) + + assert a is not b + assert session.client("sts", region_name="us-west-2", config=Config(user_agent_extra="a")) is a + + +class TestWhatTheFactoryHandsTheSdk: + """The seam the shared session rides on: the SDK's constructor takes ``boto_session``, + and ``MemoryClient`` takes ``boto3_session``. On (the default) both get the + shared session; off, nothing — the SDKs build their own, as before.""" + + def test_the_sdk_constructor_gets_the_shared_session_by_default(self, memory_env, monkeypatch): + captured = {} + original = sdk.AgentCoreMemorySessionManager.__init__ + + def spy(self, *args, **kwargs): + captured["boto_session"] = kwargs.get("boto_session") + original(self, *args, **kwargs) + + monkeypatch.setattr(sdk.AgentCoreMemorySessionManager, "__init__", spy) + + _build("session-a") + + assert captured["boto_session"] is aws_clients.shared_boto_session() + + def test_the_sdk_constructor_gets_nothing_with_the_kill_switch(self, memory_env, monkeypatch): + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + captured = {} + original = sdk.AgentCoreMemorySessionManager.__init__ + + def spy(self, *args, **kwargs): + captured["boto_session"] = kwargs.get("boto_session") + original(self, *args, **kwargs) + + monkeypatch.setattr(sdk.AgentCoreMemorySessionManager, "__init__", spy) + + _build("session-a") + + assert captured["boto_session"] is None + + def test_strategy_discovery_builds_its_client_on_the_shared_session(self): + fetch = factory._discover_strategy_ids.__wrapped__ + with patch.object(factory, "MemoryClient") as memory_client: + memory_client.return_value.get_memory_strategies.return_value = [] + fetch("mem-test", "us-west-2", shared_session=True) + + memory_client.assert_called_once_with( + region_name="us-west-2", boto3_session=aws_clients.shared_boto_session() + ) + + def test_strategy_discovery_builds_a_fresh_client_when_asked_not_to_share(self): + fetch = factory._discover_strategy_ids.__wrapped__ + with patch.object(factory, "MemoryClient") as memory_client: + memory_client.return_value.get_memory_strategies.return_value = [] + fetch("mem-test", "us-west-2", shared_session=False) + + memory_client.assert_called_once_with(region_name="us-west-2", boto3_session=None) + + def test_the_factory_asks_for_the_shared_entry_unless_switched_off(self, memory_env, monkeypatch): + asked: List[bool] = [] + monkeypatch.setattr( + factory, + "_discover_strategy_ids", + lambda memory_id, region, **kwargs: (asked.append(kwargs["shared_session"]), (None, None, None))[1], + ) + + _build("session-a") + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + _build("session-b") + + assert asked == [True, False] + + def test_warm_strategy_ids_primes_the_shared_entry(self, memory_env, monkeypatch): + """Warm-up's call and the first turn must hit the same cache key, + or the first turn pays the call warm-up already made.""" + with patch.object(factory, "_discover_strategy_ids", return_value=(None, None, None)) as discover: + factory.warm_strategy_ids() + + discover.assert_called_once_with("mem-test", "us-west-2", shared_session=True) diff --git a/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py b/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py index 1700295db..a3fe26cf0 100644 --- a/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py +++ b/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py @@ -206,6 +206,98 @@ def drain_statuses(self): assert any(f.startswith("event: done") for f, _ in frames) +_TITLE_FRAME = 'event: session_title\ndata: {"type": "session_title", "title": "T"}\n\n' + + +class _OneShotTitle: + """The routes' `_session_title_sse` contract: ready once, emitted once. + + It becomes ready when the agent stream goes silent (the hook's first + record), which models a title that lands during a tool call or the + model's time-to-first-token. + """ + + def __init__(self, hook: _Hook) -> None: + self.ready = False + self.emitted = 0 + record = hook.record + + def _record(status: dict) -> None: + self.ready = True + record(status) + + hook.record = _record + + def __call__(self) -> Optional[str]: + if self.emitted or not self.ready: + return None + self.emitted += 1 + return _TITLE_FRAME + + +async def _collect_with_side_frame(agent, wrapper, poll) -> List[tuple]: + coordinator = StreamCoordinator() + seen: List[tuple] = [] + async for sse in coordinator.stream_response( + agent=agent, + prompt="hi", + session_manager=_SessionManager(), + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=wrapper, + poll_side_frame=poll, + ): + seen.append((sse, agent.spoke_again)) + return seen + + +class TestSideFrame: + """The first turn's `session_title` rides the same poll as the statuses.""" + + @pytest.mark.asyncio + async def test_a_ready_title_goes_out_during_the_silence(self, monkeypatch): + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + agent = _StallingAgent(hook) + poll = _OneShotTitle(hook) + + frames = await _collect_with_side_frame(agent, _Wrapper(hook), poll) + + arrivals = [spoke for sse, spoke in frames if sse == _TITLE_FRAME] + assert arrivals == [False] + assert poll.emitted == 1 + + @pytest.mark.asyncio + async def test_it_is_polled_even_without_a_status_hook(self, monkeypatch): + """No hook means no statuses, not no title.""" + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + agent = _StallingAgent(_Hook(), silence=0.25) + emitted: List[str] = [] + + def poll() -> Optional[str]: + if emitted: + return None + emitted.append(_TITLE_FRAME) + return _TITLE_FRAME + + frames = await _collect_with_side_frame(agent, _Wrapper(None), poll) + + assert [sse for sse, _ in frames].count(_TITLE_FRAME) == 1 + + @pytest.mark.asyncio + async def test_a_failing_poll_does_not_break_the_turn(self, monkeypatch): + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + agent = _StallingAgent(hook, silence=0.05) + + def poll() -> Optional[str]: + raise RuntimeError("title task exploded") + + frames = await _collect_with_side_frame(agent, _Wrapper(hook), poll) + + assert any(f.startswith("event: done") for f, _ in frames) + + class TestEarlyExit: @pytest.mark.asyncio async def test_a_cooperative_stop_mid_silence_ends_cleanly(self, monkeypatch): diff --git a/backend/tests/agents/main_agent/streaming/test_display_text_write.py b/backend/tests/agents/main_agent/streaming/test_display_text_write.py index 0177d547d..40c1d4753 100644 --- a/backend/tests/agents/main_agent/streaming/test_display_text_write.py +++ b/backend/tests/agents/main_agent/streaming/test_display_text_write.py @@ -58,6 +58,15 @@ def __init__(self, wrote: bool = False) -> None: def arm(self, **kwargs) -> None: self.arms.append(kwargs) + async def resolved_arms(self) -> List[dict]: + """The arms with the index resolved: the coordinator arms with + ``HistoryCount.resolve`` so the history read stays off the turn path.""" + out = [] + for arm in self.arms: + index = arm["message_index"] + out.append({**arm, "message_index": await index() if callable(index) else index}) + return out + @property def wrote_this_turn(self) -> bool: return self._wrote @@ -98,7 +107,7 @@ async def test_arms_the_hook_with_this_turns_text_and_index(self, store): await _run(_Wrapper(hook), original_message="what the user typed") - assert hook.arms == [ + assert await hook.resolved_arms() == [ { "session_id": "sess-1", "user_id": "user-1", @@ -115,7 +124,7 @@ async def test_arms_to_none_when_the_prompt_was_not_modified(self, store): await _run(_Wrapper(hook), original_message=None) - assert hook.arms == [ + assert await hook.resolved_arms() == [ { "session_id": "sess-1", "user_id": "user-1", diff --git a/backend/tests/agents/main_agent/streaming/test_first_token_clock.py b/backend/tests/agents/main_agent/streaming/test_first_token_clock.py new file mode 100644 index 000000000..bc5bee02a --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_first_token_clock.py @@ -0,0 +1,200 @@ +"""The coordinator's half of the first-token clock (turn-path spec P1b). + +`turn_prelude` stops when the agent is ready. Everything after it — the head +of turn, LTM retrieval and the hooks before the model call, the model's own +time to first token — was unmeasured, and a first token that moved because of +any of it was invisible. These pin three things: the coordinator closes those +stages in order on the clock it was handed; the boundary between `pre_model` +and `model` comes from the status hook's stamp; and the `turn_first_token` +line is written only AFTER the first token has reached the consumer, because a +log write in front of the first token is exactly the latency this measures. +""" + +from typing import Any, AsyncIterator, Dict, List, Optional + +import pytest + +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +class _Clock: + """Records each call with how many frames the consumer had received.""" + + def __init__(self, received: List[str]) -> None: + self.calls: List[tuple] = [] + self._received = received + + def mark(self, stage: str) -> None: + self.calls.append(("mark", stage, len(self._received))) + + def mark_at(self, stage: str, at: float) -> None: + self.calls.append(("mark_at", stage, at)) + + def emit_first_token(self) -> None: + self.calls.append(("emit_first_token", None, len(self._received))) + + def stages(self) -> List[str]: + return [c[1] for c in self.calls if c[0] in ("mark", "mark_at")] + + +class _Hook: + def __init__(self, stamp: Optional[float]) -> None: + self.first_model_call_at = stamp + + def drain_statuses(self) -> List[dict]: + return [] + + def drain_batches(self) -> List[dict]: + return [] + + +class _Wrapper: + def __init__(self, hook: Optional[_Hook]) -> None: + if hook is not None: + self.agent_status_hook = hook + + +class _SessionManager: + cancelled = False + + async def update_after_turn(self, input_tokens, current_messages=None): + return None + + +class _Agent: + def __init__(self, events: List[Dict[str, Any]]) -> None: + self.messages = [{"role": "user", "content": [{"text": "hi"}]}] + self._events = events + + def stream_async(self, prompt: Any) -> AsyncIterator[Dict[str, Any]]: + async def _gen() -> AsyncIterator[Dict[str, Any]]: + for event in self._events: + yield event + + return _gen() + + +_TEXT_TURN = [ + {"event": {"messageStart": {"role": "assistant"}}}, + {"event": {"contentBlockDelta": {"contentBlockIndex": 0, "delta": {"text": "Hello"}}}}, + {"event": {"contentBlockDelta": {"contentBlockIndex": 0, "delta": {"text": " there"}}}}, + {"event": {"contentBlockStop": {"contentBlockIndex": 0}}}, + {"event": {"messageStop": {"stopReason": "end_turn"}}}, +] + +_HEAD_OF_TURN = [ + "head_of_turn.handoff", + "head_of_turn.compaction", + "head_of_turn.history_count", + "head_of_turn.rest", +] + + +async def _run(events, wrapper, *, with_clock=True): + received: List[str] = [] + clock = _Clock(received) if with_clock else None + async for sse in StreamCoordinator().stream_response( + agent=_Agent(events), + prompt="hi", + session_manager=_SessionManager(), + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=wrapper, + turn_clock=clock, + ): + received.append(sse) + return clock, received + + +def _index_of_first_delta(received: List[str], text: str) -> int: + """The typed frame the SPA renders — not the raw `event: event` + passthrough the processor also yields ahead of it.""" + return next( + i + for i, sse in enumerate(received) + if sse.startswith("event: content_block_delta") and text in sse + ) + + +@pytest.mark.asyncio +async def test_stages_close_in_order_and_split_at_the_hook_stamp(): + clock, _ = await _run(_TEXT_TURN, _Wrapper(_Hook(stamp=123.0))) + + assert clock.stages() == [*_HEAD_OF_TURN, "pre_model", "model"] + assert ("mark_at", "pre_model", 123.0) in clock.calls + assert [c[0] for c in clock.calls].count("emit_first_token") == 1 + + +@pytest.mark.asyncio +async def test_the_line_is_written_after_the_first_token_reached_the_consumer(): + """The emit must never sit in front of the token it measures.""" + clock, received = await _run(_TEXT_TURN, _Wrapper(_Hook(stamp=1.0))) + + first_token_frame = _index_of_first_delta(received, "Hello") + model_mark = next(c for c in clock.calls if c[1] == "model") + emit = next(c for c in clock.calls if c[0] == "emit_first_token") + + # Marked on arrival — before that frame was handed over... + assert model_mark[2] <= first_token_frame + # ...and emitted only once the consumer already held it. + assert emit[2] > first_token_frame + + +@pytest.mark.asyncio +async def test_without_a_hook_stamp_the_gap_is_not_split(): + clock, _ = await _run(_TEXT_TURN, _Wrapper(_Hook(stamp=None))) + + assert clock.stages() == [*_HEAD_OF_TURN, "pre_model_and_model"] + + +@pytest.mark.asyncio +async def test_a_tool_first_turn_counts_the_tool_call_as_the_first_token(): + events = [ + {"event": {"messageStart": {"role": "assistant"}}}, + { + "event": { + "contentBlockStart": { + "contentBlockIndex": 0, + "start": {"toolUse": {"toolUseId": "t1", "name": "calculator"}}, + } + } + }, + {"event": {"messageStop": {"stopReason": "end_turn"}}}, + ] + + clock, _ = await _run(events, _Wrapper(_Hook(stamp=1.0))) + + assert clock.stages()[-2:] == ["pre_model", "model"] + assert [c[0] for c in clock.calls].count("emit_first_token") == 1 + + +@pytest.mark.asyncio +async def test_a_turn_with_no_model_output_emits_nothing(): + """No first token, no line — not a line with a made-up number.""" + events = [{"event": {"messageStart": {"role": "assistant"}}}] + + clock, _ = await _run(events, _Wrapper(_Hook(stamp=1.0))) + + assert clock.stages() == _HEAD_OF_TURN + assert not any(c[0] == "emit_first_token" for c in clock.calls) + + +@pytest.mark.asyncio +async def test_a_failing_clock_never_breaks_the_turn(): + class _Broken: + def __getattr__(self, _name): + raise RuntimeError("clock down") + + received: List[str] = [] + async for sse in StreamCoordinator().stream_response( + agent=_Agent(_TEXT_TURN), + prompt="hi", + session_manager=_SessionManager(), + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=_Wrapper(_Hook(stamp=1.0)), + turn_clock=_Broken(), + ): + received.append(sse) + + assert any("Hello" in sse for sse in received) diff --git a/backend/tests/agents/main_agent/streaming/test_history_count.py b/backend/tests/agents/main_agent/streaming/test_history_count.py new file mode 100644 index 000000000..1b07f0040 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_history_count.py @@ -0,0 +1,240 @@ +"""The per-turn history count, off the critical path (turn-path spec §5 P4, F5). + +Every per-message metadata row (cost, usage, displayText, the artifact anchor) +is keyed by ``count of messages before this turn + offset``, and the messages +endpoint re-derives the same index on reload. So the number must be exactly +what a serial read at the head of the turn gave — a wrong one silently puts +one turn's cost or clean text on another turn's bubble. What is under test, +most expensive first: + +1. **The count excludes this turn's own messages** even when the read lands + after they were written (the cutoff on ``created_at``). +2. **The read does not block the stream**: the agent starts while it runs. +3. **A failure falls back to the count as it was at the head of the turn**, + not to a maintained count that has since moved. +4. ``HISTORY_COUNT_PREFETCH_ENABLED=false`` restores the inline read. +""" + +import asyncio +import threading +from datetime import datetime, timedelta, timezone +from types import SimpleNamespace +from typing import Any, AsyncIterator, Dict, List, Optional +from unittest.mock import AsyncMock, patch + +import pytest + +from agents.main_agent.streaming.history_count import HistoryCount, count_created_before +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + +_HEAD = datetime(2026, 10, 2, 12, 0, 0, tzinfo=timezone.utc) + + +def _msg(created_at: Optional[str]) -> SimpleNamespace: + return SimpleNamespace(created_at=created_at) + + +def _old(n: int) -> List[SimpleNamespace]: + return [_msg((_HEAD - timedelta(minutes=10 - i)).isoformat()) for i in range(n)] + + +class TestCountCreatedBefore: + def test_no_cutoff_counts_everything(self): + assert count_created_before(_old(3) + [_msg(None)], None) == 4 + + def test_messages_from_this_turn_are_excluded(self): + this_turn = _msg((_HEAD + timedelta(milliseconds=1)).isoformat()) + assert count_created_before(_old(3) + [this_turn], _HEAD) == 3 + + def test_a_message_stamped_exactly_at_the_cutoff_is_this_turns(self): + assert count_created_before([_msg(_HEAD.isoformat())], _HEAD) == 0 + + def test_z_suffix_and_naive_stamps_read_as_utc(self): + z = _msg((_HEAD - timedelta(seconds=1)).strftime("%Y-%m-%dT%H:%M:%S.%fZ")) + naive = _msg((_HEAD + timedelta(seconds=1)).replace(tzinfo=None).isoformat()) + assert count_created_before([z, naive], _HEAD) == 1 + + def test_missing_or_unparseable_stamps_count_as_history(self): + assert count_created_before([_msg(None), _msg(""), _msg("not a date")], _HEAD) == 3 + + def test_empty(self): + assert count_created_before(None, _HEAD) == 0 + assert count_created_before([], None) == 0 + + +class TestHistoryCount: + @pytest.mark.asyncio + async def test_start_returns_before_the_read_finishes(self): + release = threading.Event() + + def _count(before: datetime) -> int: + assert release.wait(5) + return 9 + + count = HistoryCount.start(_count, fallback=0) + assert count.value is None + release.set() + assert await count.resolve() == 9 + + @pytest.mark.asyncio + async def test_the_cutoff_is_taken_at_start(self): + seen: List[datetime] = [] + count = HistoryCount.start(lambda before: seen.append(before) or 0, fallback=0, now=_HEAD) + await count.resolve() + assert seen == [_HEAD] + + @pytest.mark.asyncio + async def test_reads_once_however_many_consumers_await(self): + calls: List[int] = [] + count = HistoryCount.start(lambda before: calls.append(1) or 4, fallback=0) + assert await asyncio.gather(count.resolve(), count.resolve(), count.resolve()) == [4, 4, 4] + assert await count.resolve() == 4 + assert calls == [1] + + @pytest.mark.asyncio + async def test_a_failed_read_resolves_to_the_fallback(self): + def _boom(before: datetime) -> int: + raise RuntimeError("ListEvents throttled") + + count = HistoryCount.start(_boom, fallback=6) + assert await count.resolve() == 6 + + @pytest.mark.asyncio + async def test_resolved_needs_no_read(self): + count = HistoryCount.resolved(3) + assert count.value == 3 + assert await count.resolve() == 3 + + +class _ListingManager: + """A session manager whose ``list_messages`` is the SDK's global read.""" + + def __init__(self, messages, *, message_count: int = 0, gate: Optional[threading.Event] = None): + self.config = SimpleNamespace(session_id="sess-1") + self.message_count = message_count + self._messages = messages + self._gate = gate + self.calls: List[threading.Thread] = [] + self.gate_was_open: Optional[bool] = None + self.cancelled = False + self.turn_lease = None + + def list_messages(self, session_id: str, agent_id: str) -> List[Any]: + self.calls.append(threading.current_thread()) + if self._gate is not None: + self.gate_was_open = self._gate.wait(5) + messages = self._messages() if callable(self._messages) else self._messages + return list(messages) + + async def update_after_turn(self, input_tokens, current_messages=None): + return None + + +class TestStartHistoryCount: + @pytest.mark.asyncio + async def test_counts_off_the_calling_thread_with_this_turns_messages_excluded(self, monkeypatch): + monkeypatch.delenv("HISTORY_COUNT_PREFETCH_ENABLED", raising=False) + # The read lands after this turn's user message was written: it comes + # back in the listing, stamped after the head of the turn. + manager = _ListingManager( + lambda: _old(5) + [_msg(datetime.now(timezone.utc).isoformat())] + ) + + count = StreamCoordinator()._start_history_count(manager) + + assert await count.resolve() == 5 + assert manager.calls and manager.calls[0] is not threading.current_thread() + + @pytest.mark.asyncio + async def test_kill_switch_reads_inline_and_counts_everything(self, monkeypatch): + monkeypatch.setenv("HISTORY_COUNT_PREFETCH_ENABLED", "false") + manager = _ListingManager(_old(5)) + + count = StreamCoordinator()._start_history_count(manager) + + assert count.value == 5 + assert manager.calls == [threading.current_thread()] + + @pytest.mark.asyncio + async def test_the_fallback_is_the_count_at_the_head_of_the_turn(self, monkeypatch): + monkeypatch.delenv("HISTORY_COUNT_PREFETCH_ENABLED", raising=False) + gate = threading.Event() + + def _boom(): + raise RuntimeError("ListEvents failed") + + manager = _ListingManager(_boom, message_count=4, gate=gate) + count = StreamCoordinator()._start_history_count(manager) + # This turn's user message is appended while the read is in flight. + manager.message_count = 5 + gate.set() + + assert await count.resolve() == 4 + + @pytest.mark.asyncio + async def test_a_manager_that_cannot_list_uses_its_own_count(self, monkeypatch): + monkeypatch.delenv("HISTORY_COUNT_PREFETCH_ENABLED", raising=False) + manager = SimpleNamespace(message_count=3) + + count = StreamCoordinator()._start_history_count(manager) + + assert count.value == 3 + + +class _GateOpeningAgent: + """Opens the gate the history read waits on as soon as the stream starts.""" + + def __init__(self, gate: threading.Event) -> None: + self.messages = [{"role": "user", "content": [{"text": "hi"}]}] + self._gate = gate + + def stream_async(self, prompt: Any) -> AsyncIterator[Dict[str, Any]]: + async def _gen() -> AsyncIterator[Dict[str, Any]]: + self._gate.set() + return + yield # pragma: no cover - empty stream + + return _gen() + + +class _RecordingHook: + def __init__(self) -> None: + self.arms: List[dict] = [] + + def arm(self, **kwargs) -> None: + self.arms.append(kwargs) + + @property + def wrote_this_turn(self) -> bool: + return True + + +class TestThroughStreamResponse: + @pytest.mark.asyncio + async def test_the_agent_starts_while_the_count_is_still_reading(self, monkeypatch): + """The whole point: the read used to finish before the agent was + called. Here it cannot finish until the agent has started.""" + monkeypatch.delenv("HISTORY_COUNT_PREFETCH_ENABLED", raising=False) + gate = threading.Event() + manager = _ListingManager( + lambda: _old(6) + [_msg(datetime.now(timezone.utc).isoformat())], gate=gate + ) + hook = _RecordingHook() + + with patch( + "apis.shared.sessions.metadata.store_user_display_text", new_callable=AsyncMock + ): + async for _ in StreamCoordinator().stream_response( + agent=_GateOpeningAgent(gate), + prompt="augmented prompt", + session_manager=manager, + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=SimpleNamespace(display_text_hook=hook), + original_message="typed", + ): + pass + + assert manager.gate_was_open is True + [arm] = hook.arms + assert await arm["message_index"]() == 6 diff --git a/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py b/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py index 540a42f9d..6df8196a8 100644 --- a/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py +++ b/backend/tests/agents/main_agent/streaming/test_stream_response_signature.py @@ -110,8 +110,10 @@ async def test_every_kwarg_chat_agent_forwards_is_accepted(): original_message="a message", turn_agent_id="ast-canvas", turn_lease=object(), + turn_clock=object(), ): pass + assert "turn_clock" in coordinator.captured accepted = set(inspect.signature(StreamCoordinator.stream_response).parameters) assert set(coordinator.captured) <= accepted diff --git a/backend/tests/agents/main_agent/test_base_agent_tool_stages.py b/backend/tests/agents/main_agent/test_base_agent_tool_stages.py new file mode 100644 index 000000000..bab058c14 --- /dev/null +++ b/backend/tests/agents/main_agent/test_base_agent_tool_stages.py @@ -0,0 +1,84 @@ +"""`agent_build.tools` decomposed (P1a in docs/specs/turn-path-ttft.md). + +The stage was 2039ms on a cold dev build with no owner named. These pin that +`_build_filtered_tools` closes its four sub-stages in order whatever the turn +carries — so the shape of the line never depends on the toolset — and that the +per-server MCP timings reach the recorder from the calling thread, including +when the load crosses into the executor. +""" +import pytest +from types import SimpleNamespace +from unittest.mock import patch + +from agents.main_agent.base_agent import BaseAgent +from agents.main_agent.tools.tool_filter import ToolFilterResult +from apis.shared.observability import build_stages + +_STAGES = ["tools.filter", "tools.gateway", "tools.mcp", "tools.extra"] + + +def _agent(*, external_ids=None, extra_tools=None): + result = ToolFilterResult(["local"], [], list(external_ids or [])) + return SimpleNamespace( + enabled_tools=["calculator", *(external_ids or [])], + tool_filter=SimpleNamespace(filter_tools_extended=lambda _ids: result), + extra_tools=extra_tools or [], + user_id="u", + auth_token=None, + ) + + +class _FakeIntegration: + def __init__(self): + self.client = object() + + async def load_external_tools(self, ids, *, user_id, auth_token, timings=None): + for tool_id in ids: + timings.append({"id": tool_id, "outcome": "loaded", "totalMs": 5}) + return [self.client] + + +def _build(agent): + stages, details = [], {} + token = build_stages.set_stage_recorder(stages.append, details.__setitem__) + try: + tools = BaseAgent._build_filtered_tools(agent) + finally: + build_stages.reset_stage_recorder(token) + return tools, stages, details + + +def test_a_turn_without_mcp_still_closes_every_sub_stage(): + tools, stages, details = _build(_agent(extra_tools=["spreadsheet"])) + + assert tools == ["local", "spreadsheet"] + assert stages == _STAGES + assert "mcpServers" not in details + + +def test_mcp_server_timings_reach_the_recorder(): + integration = _FakeIntegration() + with patch( + "agents.main_agent.integrations.external_mcp_client.get_external_mcp_integration", + return_value=integration, + ): + tools, stages, details = _build(_agent(external_ids=["canvas"])) + + assert integration.client in tools + assert stages == _STAGES + assert details["mcpServers"] == [{"id": "canvas", "outcome": "loaded", "totalMs": 5}] + + +@pytest.mark.asyncio +async def test_timings_cross_the_executor_hop(): + """Under a running loop the load goes to a worker thread; the list must + still come back filled and the marks still land on this thread.""" + integration = _FakeIntegration() + with patch( + "agents.main_agent.integrations.external_mcp_client.get_external_mcp_integration", + return_value=integration, + ): + _, stages, details = _build(_agent(external_ids=["canvas"])) + + assert stages == _STAGES + assert [s["id"] for s in details["mcpServers"]] == ["canvas"] diff --git a/backend/tests/agents/main_agent/test_voice_agent.py b/backend/tests/agents/main_agent/test_voice_agent.py index a67ecdf3b..9a28cadb5 100644 --- a/backend/tests/agents/main_agent/test_voice_agent.py +++ b/backend/tests/agents/main_agent/test_voice_agent.py @@ -42,7 +42,7 @@ def _provider_source(self): import importlib.util import pathlib - spec = importlib.util.find_spec("strands.experimental.bidi") + spec = importlib.util.find_spec("strands.bidi") assert spec is not None and spec.origin, "strands bidi package not found" provider = pathlib.Path(spec.origin).parent / "models" / "bedrock.py" assert provider.is_file(), ( @@ -72,26 +72,88 @@ def test_voice_agent_imports_the_current_provider_name(self): for alias in node.names } assert ( - "strands.experimental.bidi.models.bedrock.BedrockNovaSonicModel" in imported + "strands.bidi.models.bedrock.BedrockNovaSonicModel" in imported ) assert not any("BidiNovaSonicModel" in name for name in imported), ( f"stale 1.51 provider name still imported: {sorted(imported)}" ) - def test_provider_takes_flattened_audio_and_region_kwargs(self): - """1.55.0 replaced provider_config/client_config with audio/region.""" + def test_provider_takes_audio_voice_and_region_kwargs(self): + """1.57 split `voice` out of the audio config; region stays a kwarg.""" source = self._provider_source() - assert "audio: AudioConfig | None = None" in source + assert "audio: BedrockNovaSonicAudioConfig | None = None" in source + assert "voice: str =" in source assert "region: str | None = None" in source assert "provider_config" not in source - def test_audio_config_still_carries_the_five_keys_we_send(self): - from strands.experimental.bidi.types.model import AudioConfig - - assert {"voice", "input_rate", "output_rate", "channels", "format"} <= set( - AudioConfig.__annotations__ + def test_audio_config_carries_the_per_direction_rates_we_send(self): + """VoiceAgent sends {"input": {"sample_rate"}, "output": {"sample_rate"}}.""" + from strands.bidi.models.configs import ( + BedrockNovaSonicAudioConfig, + BedrockNovaSonicAudioStreamConfig, ) + assert {"input", "output"} == set(BedrockNovaSonicAudioConfig.__annotations__) + assert set(BedrockNovaSonicAudioStreamConfig.__annotations__) == {"sample_rate"} + assert Defaults.NOVA_SONIC_INPUT_RATE in (8000, 16000, 24000) + assert Defaults.NOVA_SONIC_OUTPUT_RATE in (8000, 16000, 24000) + + def test_agent_send_accepts_the_input_shapes_we_use(self): + """send_audio / send_text build `audio_delta` and `text` dicts.""" + import importlib.util + import pathlib + + spec = importlib.util.find_spec("strands.bidi") + source = (pathlib.Path(spec.origin).parent / "agent" / "agent.py").read_text() + assert 'case {"audio_delta": audio, **rest} if not rest:' in source + assert 'case {"text": text, **rest} if not rest:' in source + + def test_output_event_names_the_wire_adapter_translates(self): + """If upstream renames these again, VoiceWireAdapter must follow.""" + import importlib.util + import pathlib + + spec = importlib.util.find_spec("strands.bidi") + source = (pathlib.Path(spec.origin).parent / "types" / "events.py").read_text() + for name in ( + "bidi_response_start", + "bidi_response_stop", + "bidi_transcript_start", + "bidi_transcript_delta", + "bidi_transcript_stop", + "bidi_audio_start", + "bidi_audio_delta", + "bidi_audio_stop", + "bidi_barge_in", + "bidi_usage", + "bidi_connection_stop", + ): + assert f'"type": "{name}"' in source, name + + def test_session_manager_rides_in_as_a_hook(self): + """1.57.2 removed BidiAgent(session_manager=); persistence now depends on `hooks=`. + + A stale kwarg raises TypeError at voice start; dropping it instead would + silently stop saving voice transcripts. Pin both halves: the SDK still + fires the events the manager listens on, and VoiceAgent hands it over. + """ + import importlib.util + import inspect + import pathlib + + import agents.main_agent.voice_agent as va + + spec = importlib.util.find_spec("strands.bidi") + source = (pathlib.Path(spec.origin).parent / "agent" / "agent.py").read_text() + init = source[source.index("def __init__") : source.index('"""', source.index("def __init__"))] + assert "session_manager" not in init + assert "hooks: list[HookProvider] | None = None" in init + assert "AgentInitializedEvent" in source and "MessageAddedEvent" in source + + create = inspect.getsource(va.VoiceAgent._create_agent) + assert "hooks=[self.session_manager]" in create + assert "session_manager=self.session_manager" not in create + def test_nova_sonic_usage_is_still_cumulative(self): """VoiceAgent de-cumulates bidi_usage; a switch to deltas would double-count.""" assert "usage_is_cumulative = True" in self._provider_source() @@ -230,3 +292,401 @@ def test_pyaudio_in_sys_modules(self): # After importing voice_agent, pyaudio should be mocked import agents.main_agent.voice_agent # noqa: F401 assert "pyaudio" in sys.modules + + +class TestVoiceSelection: + """The voice a session speaks with is the user's, validated, never a bad id.""" + + def _agent(self, voice=None, monkeypatch=None): + from agents.main_agent.voice_agent import VoiceAgent + + agent = VoiceAgent.__new__(VoiceAgent) + # Only the constructor's voice resolution is under test; skip BaseAgent. + with patch.object(BaseAgent, "__init__", return_value=None): + VoiceAgent.__init__(agent, voice=voice) + return agent + + def test_defaults_to_the_platform_voice(self, monkeypatch): + monkeypatch.delenv(EnvVars.NOVA_SONIC_VOICE, raising=False) + assert self._agent().voice_id == Defaults.NOVA_SONIC_VOICE + + def test_uses_a_requested_catalog_voice(self, monkeypatch): + monkeypatch.delenv(EnvVars.NOVA_SONIC_VOICE, raising=False) + assert self._agent(voice="Carlos").voice_id == "carlos" + + def test_unknown_request_falls_back_to_the_env_default(self, monkeypatch): + monkeypatch.setenv(EnvVars.NOVA_SONIC_VOICE, "matthew") + assert self._agent(voice="siri").voice_id == "matthew" + + def test_bad_env_default_never_reaches_bedrock(self, monkeypatch): + monkeypatch.setenv(EnvVars.NOVA_SONIC_VOICE, "alexa") + assert self._agent(voice=None).voice_id == "tiffany" + + +class TestVoicePresentationInPrompt: + """Amazon's guidance: tell the model the voice's gender for languages that conjugate it.""" + + def _prompt_for(self, voice_id): + from agents.main_agent.voice_agent import VoiceAgent + + agent = VoiceAgent.__new__(VoiceAgent) + agent.system_prompt = "Base." + agent._voice = voice_id + return agent._build_voice_system_prompt() + + def test_feminine_voice(self): + prompt = self._prompt_for("carolina") + assert 'voice "Carolina"' in prompt + assert "use the feminine form" in prompt + + def test_masculine_voice(self): + prompt = self._prompt_for("leo") + assert 'voice "Leo"' in prompt + assert "use the masculine form" in prompt + + def test_guidelines_still_precede_the_voice_line(self): + prompt = self._prompt_for("tiffany") + assert prompt.index("Voice Interaction Guidelines") < prompt.index('voice "Tiffany"') + + +class TestUsageModalitySplit: + """Nova's usageEvent split survives the provider, and prices each bucket.""" + + NOVA_USAGE = { + "totalInputTokens": 1_300, + "totalOutputTokens": 2_400, + "totalTokens": 3_700, + "details": { + "delta": {"input": {"speechTokens": 10, "textTokens": 1}, "output": {"speechTokens": 20, "textTokens": 2}}, + "total": { + "input": {"speechTokens": 1_000, "textTokens": 300}, + "output": {"speechTokens": 2_000, "textTokens": 400}, + }, + }, + } + + def test_split_rows_come_from_details_total_not_delta(self): + from agents.main_agent.voice_agent import usage_modality_details + + rows = usage_modality_details(self.NOVA_USAGE) + assert rows == [ + {"modality": "audio", "input_tokens": 1_000, "output_tokens": 2_000}, + {"modality": "text", "input_tokens": 300, "output_tokens": 400}, + ] + + def test_no_details_means_no_rows(self): + from agents.main_agent.voice_agent import usage_modality_details + + assert usage_modality_details({"totalInputTokens": 5, "totalOutputTokens": 6}) == [] + assert usage_modality_details({"details": {}}) == [] + + @pytest.mark.asyncio + async def test_receive_events_accumulates_the_split_and_prices_it(self, monkeypatch): + from agents.main_agent.voice_agent import VoiceAgent, VoiceWireAdapter + + agent = VoiceAgent.__new__(VoiceAgent) + agent._wire = VoiceWireAdapter() + agent._accumulated_usage = {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0} + agent._per_turn_usage = [] + agent._turn_count = 0 + agent._response_start_count = 0 + + usage_event = { + "type": "bidi_usage", + "inputTokens": 1_300, + "outputTokens": 2_400, + "totalTokens": 3_700, + "modality_details": [ + {"modality": "audio", "input_tokens": 1_000, "output_tokens": 2_000}, + {"modality": "text", "input_tokens": 300, "output_tokens": 400}, + ], + } + + class _Ev: + def as_dict(self): + return dict(usage_event) + + class _Bidi: + async def receive(self): + yield _Ev() + + agent._bidi_agent = _Bidi() + + pricing = { + "inputPricePerMtok": 0.319, + "outputPricePerMtok": 2.651, + "speechInputPricePerMtok": 3.0, + "speechOutputPricePerMtok": 12.0, + } + + async def fake_pricing(_model_id): + return pricing + + import apis.shared.costs.pricing_config as pc + + monkeypatch.setattr(pc, "get_model_pricing", fake_pricing) + monkeypatch.setenv(EnvVars.NOVA_SONIC_MODEL_ID, "amazon.nova-2-sonic-v1:0") + + events = [e async for e in agent.receive_events()] + assert len(events) == 1 + usage = agent.accumulated_usage + assert usage["speechInputTokens"] == 1_000 + assert usage["textInputTokens"] == 300 + assert usage["speechOutputTokens"] == 2_000 + assert usage["textOutputTokens"] == 400 + + expected_in = (1_000 * 3.0 + 300 * 0.319) / 1_000_000 + expected_out = (2_000 * 12.0 + 400 * 2.651) / 1_000_000 + assert events[0]["cost"]["inputCost"] == pytest.approx(expected_in) + assert events[0]["cost"]["outputCost"] == pytest.approx(expected_out) + assert events[0]["cost"]["total"] == pytest.approx(expected_in + expected_out) + + +class TestProviderSubclass: + """The usage-detail subclass only exists with the bidi extra installed.""" + + def test_subclass_overrides_the_converter_the_provider_still_has(self): + import agents.main_agent.voice_agent as va + + if not va.BIDI_AVAILABLE: + pytest.skip("strands-agents[bidi] not installed") + assert issubclass(va.NovaSonicModelWithUsageDetails, va.BedrockNovaSonicModel) + # The hook we override must still be the provider's name for it. + assert callable(getattr(va.BedrockNovaSonicModel, "_convert_nova_event", None)) + + def test_override_signature_matches_the_providers(self): + """The provider calls the hook positionally; a stale arity raises on every event. + + 1.57 added ``response_state``. A name-only check passed straight through + that change, so pin the parameter list itself. + """ + import inspect + + import agents.main_agent.voice_agent as va + + if not va.BIDI_AVAILABLE: + pytest.skip("strands-agents[bidi] not installed") + ours = list(inspect.signature(va.NovaSonicModelWithUsageDetails._convert_nova_event).parameters) + theirs = list(inspect.signature(va.BedrockNovaSonicModel._convert_nova_event).parameters) + assert ours == theirs + + def test_converter_keeps_the_split_on_the_usage_event(self): + import agents.main_agent.voice_agent as va + + if not va.BIDI_AVAILABLE: + pytest.skip("strands-agents[bidi] not installed") + model = va.NovaSonicModelWithUsageDetails.__new__(va.NovaSonicModelWithUsageDetails) + events = model._convert_nova_event({"usageEvent": TestUsageModalitySplit.NOVA_USAGE}, MagicMock()) + assert len(events) == 1 + d = events[0].as_dict() + assert d["inputTokens"] == 1_300 and d["outputTokens"] == 2_400 + assert [r["modality"] for r in d["modality_details"]] == ["audio", "text"] + assert d["modality_details"][0]["input_tokens"] == 1_000 + + def test_converter_leaves_other_events_to_the_provider(self): + import agents.main_agent.voice_agent as va + + if not va.BIDI_AVAILABLE: + pytest.skip("strands-agents[bidi] not installed") + model = va.NovaSonicModelWithUsageDetails.__new__(va.NovaSonicModelWithUsageDetails) + state = MagicMock() + with patch.object(va.BedrockNovaSonicModel, "_convert_nova_event", return_value=["delegated"]) as sup: + assert model._convert_nova_event({"completionStart": {"completionId": "c1"}}, state) == ["delegated"] + sup.assert_called_once_with({"completionStart": {"completionId": "c1"}}, state) + + @pytest.mark.asyncio + async def test_drain_keeps_the_split_too(self): + """Usage that only arrives after stop() must price on the same buckets as the live stream.""" + from agents.main_agent.voice_agent import VoiceAgent, VoiceWireAdapter + + class _Ev: + def as_dict(self): + return { + "type": "bidi_usage", + "inputTokens": 1_300, + "outputTokens": 2_400, + "totalTokens": 3_700, + "modality_details": [ + {"modality": "audio", "input_tokens": 1_000, "output_tokens": 2_000}, + {"modality": "text", "input_tokens": 300, "output_tokens": 400}, + ], + } + + class _Bidi: + async def receive(self): + yield _Ev() + + agent = VoiceAgent.__new__(VoiceAgent) + agent._bidi_agent = _Bidi() + agent._wire = VoiceWireAdapter() + agent._accumulated_usage = {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0} + agent._per_turn_usage = [] + agent._turn_count = 0 + + await agent.drain_remaining_events(timeout=1.0) + + assert agent.accumulated_usage["speechInputTokens"] == 1_000 + assert agent.accumulated_usage["textOutputTokens"] == 400 + assert agent.accumulated_usage["totalTokens"] == 3_700 + + +class TestVoiceWireAdapter: + """Strands 1.57 Bidi events -> the 1.55 wire contract the SPA speaks.""" + + def _turn(self, *events): + from agents.main_agent.voice_agent import VoiceWireAdapter + + adapter = VoiceWireAdapter() + out = [] + for event in events: + out.extend(adapter.translate(event)) + return out + + def test_user_speech_is_flushed_before_the_assistant_starts(self): + out = self._turn( + {"type": "bidi_response_start", "response_id": "r1"}, + {"type": "bidi_transcript_start", "role": "user", "content_id": "u"}, + {"type": "bidi_transcript_delta", "delta": "hello", "role": "user", "content_id": "u"}, + {"type": "bidi_transcript_stop", "transcript": "hello", "role": "user", "content_id": "u"}, + {"type": "bidi_transcript_start", "role": "assistant", "content_id": "a"}, + {"type": "bidi_transcript_delta", "delta": "hi there", "role": "assistant", "content_id": "a"}, + {"type": "bidi_audio_start"}, + {"type": "bidi_audio_delta", "audio": "QUJD", "format": "pcm", "sample_rate": 16000, "channels": 1}, + {"type": "bidi_audio_stop"}, + {"type": "bidi_transcript_stop", "transcript": "hi there", "role": "assistant", "content_id": "a"}, + {"type": "bidi_response_stop", "response_id": "r1"}, + ) + assert [e["type"] for e in out] == [ + "bidi_transcript_stream", + "bidi_response_start", + "bidi_transcript_stream", + "bidi_audio_stream", + "bidi_response_complete", + ] + assert out[0]["role"] == "user" and out[0]["delta"] == {"text": "hello"} + assert out[1]["response_id"] == "r1" + assert out[2]["role"] == "assistant" and out[2]["is_final"] is True + assert out[3] == { + "type": "bidi_audio_stream", + "audio": "QUJD", + "format": "pcm", + "sample_rate": 16000, + "channels": 1, + } + assert out[4] == {"type": "bidi_response_complete", "response_id": "r1", "stop_reason": "complete"} + + def test_exactly_one_start_per_response(self): + out = self._turn( + {"type": "bidi_response_start", "response_id": "r1"}, + {"type": "bidi_transcript_start", "role": "assistant", "content_id": "a"}, + {"type": "bidi_audio_start"}, + {"type": "bidi_response_stop", "response_id": "r1"}, + {"type": "bidi_response_start", "response_id": "r2"}, + {"type": "bidi_audio_start"}, + {"type": "bidi_response_stop", "response_id": "r2"}, + ) + assert [e["type"] for e in out].count("bidi_response_start") == 2 + + def test_barge_in_maps_to_interruption_and_an_interrupted_complete(self): + out = self._turn( + {"type": "bidi_response_start", "response_id": "r1"}, + {"type": "bidi_audio_start"}, + {"type": "bidi_barge_in", "reason": "user_speech"}, + {"type": "bidi_response_stop", "response_id": "r1"}, + ) + assert out[1] == {"type": "bidi_interruption", "reason": "user_speech"} + assert out[-1]["stop_reason"] == "interrupted" + + def test_completed_block_events_do_not_reach_the_wire(self): + """1.57.2 repeats each finished transcript as a block event; the deltas already carried it.""" + out = self._turn( + {"type": "bidi_response_start", "response_id": "r1"}, + {"type": "bidi_transcript_delta", "delta": "hi", "role": "assistant", "content_id": "a"}, + {"type": "bidi_transcript_stop", "transcript": "hi", "role": "assistant", "content_id": "a"}, + {"type": "bidi_transcript_block", "text": "hi", "role": "assistant", "content_id": "a"}, + {"type": "bidi_response_stop", "response_id": "r1"}, + ) + assert "bidi_transcript_block" not in [e["type"] for e in out] + assert [e["type"] for e in out].count("bidi_transcript_stream") == 1 + + def test_a_response_with_no_assistant_output_still_opens_before_it_closes(self): + out = self._turn( + {"type": "bidi_response_start", "response_id": "r1"}, + {"type": "bidi_response_stop", "response_id": "r1"}, + ) + assert [e["type"] for e in out] == ["bidi_response_start", "bidi_response_complete"] + + def test_connection_stop_maps_to_close_and_the_rest_pass_through(self): + out = self._turn( + {"type": "bidi_connection_start", "connection_id": "c", "model": "m"}, + {"type": "bidi_usage", "inputTokens": 1, "outputTokens": 2, "totalTokens": 3}, + {"type": "bidi_connection_stop", "connection_id": "c", "reason": "complete"}, + ) + assert out[0]["type"] == "bidi_connection_start" + assert out[1]["type"] == "bidi_usage" and out[1]["totalTokens"] == 3 + assert out[2] == {"type": "bidi_connection_close", "connection_id": "c", "reason": "complete"} + + +class TestVoiceAgentSend: + """send_audio / send_text speak the 1.57 BidiAgent.send() input shapes.""" + + def _agent(self): + from unittest.mock import AsyncMock + + from agents.main_agent.voice_agent import VoiceAgent + + agent = VoiceAgent.__new__(VoiceAgent) + agent._bidi_agent = MagicMock() + agent._bidi_agent.send = AsyncMock() + return agent + + @pytest.mark.asyncio + async def test_send_audio_decodes_base64_into_an_audio_delta(self): + import base64 + + agent = self._agent() + await agent.send_audio(base64.b64encode(b"\x00\x01").decode(), sample_rate=16000) + agent._bidi_agent.send.assert_awaited_once_with( + {"audio_delta": {"format": "pcm", "source": {"bytes": b"\x00\x01"}}} + ) + + @pytest.mark.asyncio + async def test_send_text_sends_a_text_block(self): + agent = self._agent() + await agent.send_text("hello") + agent._bidi_agent.send.assert_awaited_once_with({"text": "hello"}) + + @pytest.mark.asyncio + async def test_receive_events_counts_turns_on_the_translated_stream(self): + from agents.main_agent.voice_agent import VoiceWireAdapter + + class _Evt(dict): + def as_dict(self): + return dict(self) + + raw = [ + _Evt(type="bidi_response_start", response_id="r1"), + _Evt(type="bidi_audio_start"), + _Evt(type="bidi_usage", inputTokens=10, outputTokens=5, totalTokens=15), + _Evt(type="bidi_response_stop", response_id="r1"), + ] + + async def _receive(): + for event in raw: + yield event + + agent = self._agent() + agent._bidi_agent.receive = _receive + agent._wire = VoiceWireAdapter() + agent._accumulated_usage = {"inputTokens": 0, "outputTokens": 0, "totalTokens": 0} + agent._per_turn_usage = [] + agent._turn_count = 0 + agent._response_start_count = 0 + + with patch("apis.shared.costs.pricing_config.get_model_pricing", side_effect=RuntimeError("no pricing")): + out = [e async for e in agent.receive_events()] + + assert [e["type"] for e in out] == ["bidi_response_start", "bidi_usage", "bidi_response_complete"] + assert agent.response_start_count == 1 + assert agent.turn_count == 1 + assert agent.per_turn_usage == [{"inputTokens": 10, "outputTokens": 5, "totalTokens": 15}] diff --git a/backend/tests/apis/app_api/web_sources/test_crawler.py b/backend/tests/apis/app_api/web_sources/test_crawler.py index 3501c45dd..fd8177e91 100644 --- a/backend/tests/apis/app_api/web_sources/test_crawler.py +++ b/backend/tests/apis/app_api/web_sources/test_crawler.py @@ -650,3 +650,41 @@ async def test_refresh_robots_disallow_is_not_seen(recorder: _Recorder): # Robots said stop indexing: the URL is deliberately NOT seen, so the # worker's miss counter starts ticking toward removal. assert state.seen_urls == set() + + +@pytest.mark.asyncio +async def test_refresh_stage_failure_emits_stage_failed( + recorder: _Recorder, monkeypatch: pytest.MonkeyPatch +): + async def failing_put(*, assistant_id, document_id, markdown, filename): + raise RuntimeError("S3 PutObject failed") + + monkeypatch.setattr(crawler, "_put_markdown", failing_put) + pages = { + ROOT: '

changed words

n', + "https://example.com/new": "

new page

", + } + state, log = _refresh_state( + recorder, {ROOT: crawler.RefreshDoc(document_id="DOC-root", content_hash="different")} + ) + + await _run_refresh(recorder, pages, state) + + # Each pre-stage emit is followed by stage_failed, so the worker can roll + # back the gate values it wrote for bytes that never reached S3. + outcomes = sorted((outcome, url) for outcome, url, _ in log.events) + assert outcomes == [ + ("changed", ROOT), + ("created", "https://example.com/new"), + ("stage_failed", ROOT), + ("stage_failed", "https://example.com/new"), + ] + assert state.stage_failed == 2 + assert recorder.failed_delta == 2 + assert recorder.metadata_updates == [] + # The existing page keeps serving its last-good version; the new page, + # which has nothing staged at all, is marked failed. + new_doc_id = recorder.created_docs[0][0] + assert (new_doc_id, "failed") in recorder.status_updates + assert ("DOC-root", "failed") not in recorder.status_updates + assert recorder.finalized_status == "complete" diff --git a/backend/tests/apis/inference_api/test_get_agent_call_sites.py b/backend/tests/apis/inference_api/test_get_agent_call_sites.py index 811a1c2e9..2dd97cffb 100644 --- a/backend/tests/apis/inference_api/test_get_agent_call_sites.py +++ b/backend/tests/apis/inference_api/test_get_agent_call_sites.py @@ -40,7 +40,7 @@ def test_the_main_turn_passes_the_binding_it_built_tools_from(): if _is_false(kw.get("is_resume")) and "extra_tools_key_described" in kw ] assert len(mains) == 1 - assert ast.unparse(mains[0]["memory_binding"]) == "memory_binding_key" + assert ast.unparse(mains[0]["memory_binding"]) == "turn_tools.memory_binding_key" def test_memory_context_is_passed_on_the_main_turn_and_replayed_on_resume(): diff --git a/backend/tests/apis/inference_api/test_inline_attachment_persist.py b/backend/tests/apis/inference_api/test_inline_attachment_persist.py new file mode 100644 index 000000000..e6b23f8e6 --- /dev/null +++ b/backend/tests/apis/inference_api/test_inline_attachment_persist.py @@ -0,0 +1,197 @@ +"""Inline spreadsheets and decks are stored as session files before the turn runs. + +The attachment phase diverts csv/xlsx/pptx out of the prompt and tells the +model the Spreadsheet Analysis / PowerPoint tools can reach them. Those tools +only see files that have a ``FileMetadata`` row, which an upload has and an +inline base64 ``files`` entry did not. These pin the wiring: + +- only the *inline* diverted files are persisted (resolved uploads already + have rows; identity decides, not filename); +- a file the persister could not store leaves the diverted set, leaves the + ``[Attached files: …]`` marker, and is named in the guidance with its reason; +- the kill switch restores the pre-fix behaviour exactly. +""" + +from __future__ import annotations + +import base64 +from dataclasses import dataclass +from typing import Any, List + +import pytest + +import apis.inference_api.chat.routes as routes +from apis.inference_api.chat.models import FileContent, InvocationRequest +from apis.shared.files.inline_persist import PersistFailure + +CSV = "text/csv" +PDF = "application/pdf" +PPTX = "application/vnd.openxmlformats-officedocument.presentationml.presentation" + + +def _b64(raw: bytes) -> str: + return base64.b64encode(raw).decode() + + +@dataclass +class _ResolvedUpload: + filename: str + content_type: str + bytes: str + + +class _FakePersister: + def __init__(self, fail_names: set | None = None) -> None: + self.calls: List[dict] = [] + self.fail_names = fail_names or set() + + async def persist(self, *, user_id: str, session_id: str, files, source: str = "inline"): + self.calls.append({"user_id": user_id, "session_id": session_id, "files": list(files)}) + persisted, failures = [], [] + for f in files: + if f.filename in self.fail_names: + failures.append(PersistFailure(file=f, reason="it could not be stored")) + else: + persisted.append(f) + return persisted, failures + + +class _FakeResolver: + def __init__(self, resolved: List[_ResolvedUpload]) -> None: + self._resolved = resolved + + async def resolve_files(self, *, user_id: str, upload_ids: List[str], max_files: Any = None): + return self._resolved + + +@pytest.fixture +def phase(monkeypatch): + """Wire the attachment phase to fakes; returns a runner + the persister.""" + persister = _FakePersister() + state = {"persister": persister, "resolved": []} + + async def _no_pending(*args, **kwargs): + return [] + + import apis.shared.sessions.metadata as metadata + + monkeypatch.setattr(metadata, "pop_pending_attachments", _no_pending) + monkeypatch.setattr(routes, "get_inline_attachment_persister", lambda: state["persister"]) + monkeypatch.setattr(routes, "get_file_resolver", lambda: _FakeResolver(state["resolved"])) + + async def run(files=None, upload_ids=None): + req = InvocationRequest( + session_id="sess-1", + message="look at this", + files=files, + file_upload_ids=upload_ids, + ) + return await routes._resolve_turn_attachments(req, "user-1", None, is_resume=False, is_continuation=False) + + return run, state + + +@pytest.mark.asyncio +async def test_inline_diverted_files_are_persisted_and_pdf_is_not(phase): + run, state = phase + csv = FileContent(filename="data.csv", content_type=CSV, bytes=_b64(b"a,b\n")) + pdf = FileContent(filename="doc.pdf", content_type=PDF, bytes=_b64(b"%PDF")) + deck = FileContent(filename="deck.pptx", content_type=PPTX, bytes=_b64(b"PK")) + + out = await run(files=[csv, pdf, deck]) + + assert len(state["persister"].calls) == 1 + call = state["persister"].calls[0] + assert call["user_id"] == "user-1" and call["session_id"] == "sess-1" + # Both diverted classes, in attachment order; the inline-for-Bedrock PDF is untouched. + assert [f.filename for f in call["files"]] == ["data.csv", "deck.pptx"] + assert [f.filename for f in out.diverted_tabular] == ["data.csv"] + assert [f.filename for f in out.diverted_presentations] == ["deck.pptx"] + assert [f.filename for f in out.files_to_send] == ["doc.pdf"] + assert out.unpersisted_inline == [] + assert out.marker_names == ["data.csv", "doc.pdf", "deck.pptx"] + + +@pytest.mark.asyncio +async def test_resolved_uploads_are_not_re_persisted(phase): + run, state = phase + state["resolved"] = [_ResolvedUpload("sales.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", _b64(b"PK"))] + + out = await run(upload_ids=["up-1"]) + + assert state["persister"].calls == [] + assert [f.filename for f in out.diverted_tabular] == ["sales.xlsx"] + + +@pytest.mark.asyncio +async def test_identity_not_filename_selects_the_inline_copy(phase): + run, state = phase + # Same name arrives both ways. Dedupe keeps the first (inline) occurrence, + # and that object is the one persisted — never the resolved upload. + inline = FileContent(filename="same.csv", content_type=CSV, bytes=_b64(b"inline")) + state["resolved"] = [_ResolvedUpload("same.csv", CSV, _b64(b"uploaded"))] + + out = await run(files=[inline], upload_ids=["up-1"]) + + assert len(state["persister"].calls) == 1 + assert state["persister"].calls[0]["files"] == [inline] + assert [f.filename for f in out.diverted_tabular] == ["same.csv"] + + +@pytest.mark.asyncio +async def test_failure_leaves_the_diverted_set_the_marker_and_names_the_reason(phase): + run, state = phase + state["persister"] = _FakePersister(fail_names={"bad.csv"}) + good = FileContent(filename="good.csv", content_type=CSV, bytes=_b64(b"1")) + bad = FileContent(filename="bad.csv", content_type=CSV, bytes=_b64(b"2")) + pdf = FileContent(filename="doc.pdf", content_type=PDF, bytes=_b64(b"%PDF")) + + out = await run(files=[good, bad, pdf]) + + assert [f.filename for f in out.diverted_tabular] == ["good.csv"] + assert [x.file.filename for x in out.unpersisted_inline] == ["bad.csv"] + # The card must not be promised for a file that is not in the session. + assert out.marker_names == ["good.csv", "doc.pdf"] + + note = routes._build_attachment_guidance( + out.diverted_tabular, + out.diverted_presentations, + out.oversized_inline, + ["list_spreadsheets", "analyze_spreadsheet"], + unpersisted=out.unpersisted_inline, + ) + assert "`good.csv` are available through the Spreadsheet Analysis tool" in note + assert "`bad.csv` (it could not be stored) could not be stored for this conversation" in note + assert "Upload them through the Files panel" in note + + +@pytest.mark.asyncio +async def test_kill_switch_restores_the_old_behaviour(phase, monkeypatch): + run, state = phase + monkeypatch.setenv("INLINE_ATTACHMENT_PERSIST_ENABLED", "false") + csv = FileContent(filename="data.csv", content_type=CSV, bytes=_b64(b"a,b\n")) + + out = await run(files=[csv]) + + assert state["persister"].calls == [] + assert [f.filename for f in out.diverted_tabular] == ["data.csv"] + assert out.unpersisted_inline == [] + + +@pytest.mark.asyncio +async def test_no_diverted_inline_means_no_call(phase): + run, state = phase + pdf = FileContent(filename="doc.pdf", content_type=PDF, bytes=_b64(b"%PDF")) + await run(files=[pdf]) + assert state["persister"].calls == [] + + +def test_flag_default_on_only_false_disables(monkeypatch): + from apis.shared.feature_flags import inline_attachment_persist_enabled + + monkeypatch.delenv("INLINE_ATTACHMENT_PERSIST_ENABLED", raising=False) + assert inline_attachment_persist_enabled() is True + monkeypatch.setenv("INLINE_ATTACHMENT_PERSIST_ENABLED", "") + assert inline_attachment_persist_enabled() is True + monkeypatch.setenv("INLINE_ATTACHMENT_PERSIST_ENABLED", "False") + assert inline_attachment_persist_enabled() is False diff --git a/backend/tests/apis/inference_api/test_turn_timing.py b/backend/tests/apis/inference_api/test_turn_timing.py index 1f4c105e1..ed2644f3e 100644 --- a/backend/tests/apis/inference_api/test_turn_timing.py +++ b/backend/tests/apis/inference_api/test_turn_timing.py @@ -226,6 +226,201 @@ def test_a_group_never_breaks_the_turn_it_measures(self, monkeypatch): prelude.emit(session_id="s", stream_kind="agent") # must not raise +class TestNestedGroups: + """P1a splits `agent_build.tools` (docs/specs/turn-path-ttft.md). Every + proper prefix is a group, so the split keeps `agent_build.tools` as a + number exactly as the first split kept `agent_build`.""" + + def test_every_prefix_level_is_a_group(self, monkeypatch): + # Binary-exact steps, so the int() truncation cannot shave a ms. + clock = iter([0.0, 0.125, 0.25, 1.25, 1.375, 1.5, 1.625]) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + + prelude = TurnPrelude() + prelude.mark("agent_build.tools.filter") # 125ms + prelude.mark("agent_build.tools.gateway") # 125ms + prelude.mark("agent_build.tools.mcp") # 1000ms + prelude.mark("agent_build.tools.extra") # 125ms + prelude.mark("agent_build.hooks") # 125ms + + assert _emitted(prelude)["groups"] == { + "agent_build": 1500, + "agent_build.tools": 1375, + } + + def test_the_pre_split_metric_survives_as_a_group(self, monkeypatch): + """`AgentBuildToolsMs` is the series the cold-build baseline is stated + in; it must keep arriving after its own decomposition.""" + record = _emf_record( + monkeypatch, + ["agent_build.tools.filter", "agent_build.tools.mcp", "agent_build.hooks"], + ) + + assert { + "AgentBuildMs", + "AgentBuildToolsMs", + "AgentBuildToolsFilterMs", + "AgentBuildToolsMcpMs", + "AgentBuildHooksMs", + } <= set(record["metrics"]) + + +class TestDetails: + """Log-only properties: which MCP server owned `tools.mcp`.""" + + def test_a_detail_rides_on_the_log_line(self): + prelude = TurnPrelude() + servers = [{"id": "canvas", "outcome": "loaded", "totalMs": 812}] + prelude.detail("mcpServers", servers) + + assert _emitted(prelude)["mcpServers"] == servers + + def test_a_detail_is_never_a_metric(self, monkeypatch): + captured = {} + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", + lambda **kw: captured.update(kw), + ) + prelude = TurnPrelude() + prelude.detail("mcpServers", [{"id": "canvas", "totalMs": 812}]) + prelude.emit(session_id="s", stream_kind="agent") + + assert "mcpServers" not in captured["metrics"] + assert "mcpServers" not in captured["properties"] + + def test_the_caller_s_extras_win_a_key_collision(self): + prelude = TurnPrelude() + prelude.detail("isResume", "from-a-detail") + + assert _emitted(prelude, extra={"isResume": True})["isResume"] is True + + +class TestFirstToken: + """`turn_first_token` (docs/specs/turn-path-ttft.md P1b): the same clock, + continued past `turn_prelude` to the first model output.""" + + def _turn(self, monkeypatch, *, readings): + clock = iter(readings) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + return TurnPrelude() + + def test_reports_only_what_came_after_the_prelude(self, monkeypatch): + prelude = self._turn( + monkeypatch, readings=[0.0, 0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 2.5] + ) + prelude.mark("agent_build.session_mgr") # 0.25 -> 250 + prelude.mark("stream_setup") # 0.5 -> 250 + _emitted(prelude, extra={"isResume": False}) # totalMs reads 0.75 + prelude.mark("head_of_turn.handoff") # 1.0 -> 500 (from 0.5) + prelude.mark("head_of_turn.history_count") # 1.5 -> 500 + prelude.mark("pre_model") # 2.0 -> 500 + + payload = _first_token(prelude) + + # At the last mark (2.0), not at emit: the remaining reading (2.5) is + # never taken, and the wait for the second event is not first-token time. + assert payload["firstTokenMs"] == 2000 + assert payload["preludeTotalMs"] == 750 + assert payload["stages"] == { + "head_of_turn.handoff": 500, + "head_of_turn.history_count": 500, + "pre_model": 500, + } + assert payload["groups"] == {"head_of_turn": 1000} + # The prelude's identity and extras carry over: one join key, one shape. + assert payload["sessionId"] == "s" + assert payload["isResume"] is False + + def test_first_token_ms_is_the_model_mark_not_the_emit(self, monkeypatch): + """The line is written on the coordinator's next pass, after the first + token was yielded. The time spent waiting for that next event is not + first-token time, and the stages must add up to `firstTokenMs`.""" + prelude = self._turn(monkeypatch, readings=[0.0, 0.5, 1.0, 1.75, 9.0]) + prelude.mark("stream_setup") # 0.5 + _emitted(prelude) # totalMs reads 1.0 + prelude.mark("pre_model") # 1.75 -> 1250 (from 0.5) + + payload = _first_token(prelude) # a now-reading of 9.0 must not be used + + assert payload["firstTokenMs"] == 1750 + assert payload["firstTokenMs"] == 500 + sum(payload["stages"].values()) + + def test_emits_once(self, caplog): + prelude = TurnPrelude() + _emitted(prelude) + caplog.set_level(logging.INFO, logger="apis.inference_api.chat.turn_timing") + + prelude.emit_first_token() + prelude.emit_first_token() + + lines = [r for r in caplog.records if r.getMessage().startswith("turn_first_token ")] + assert len(lines) == 1 + + def test_without_a_prelude_line_it_reports_every_mark(self): + prelude = TurnPrelude() + prelude.mark("head_of_turn.handoff") + + payload = _first_token(prelude) + + assert list(payload["stages"]) == ["head_of_turn.handoff"] + assert payload["preludeTotalMs"] is None + + def test_mark_at_closes_a_stage_at_an_earlier_reading(self, monkeypatch): + prelude = self._turn(monkeypatch, readings=[0.0, 1.0, 3.0, 3.5]) + prelude.mark("head_of_turn.rest") # 1.0 + prelude.mark_at("pre_model", 2.0) # stamped by a hook at 2.0 -> 1000 + prelude.mark("model") # 3.0 -> 1000 (from 2.0, not from 1.0) + + stages = _first_token(prelude)["stages"] + + assert stages["pre_model"] == 1000 + assert stages["model"] == 1000 + + def test_mark_at_never_produces_a_negative_stage(self, monkeypatch): + """A stamp older than the last mark — a hook that fired before the + coordinator's own mark — is a zero stage, not a negative one.""" + prelude = self._turn(monkeypatch, readings=[0.0, 2.0, 2.5]) + prelude.mark("head_of_turn.rest") # 2.0 + prelude.mark_at("pre_model", 1.0) + + assert _first_token(prelude)["stages"]["pre_model"] == 0 + + def test_metrics_carry_first_token_ms_not_prelude_total(self, monkeypatch): + captured = [] + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", + lambda **kw: captured.append(kw), + ) + prelude = TurnPrelude() + prelude.mark("agent_build.session_mgr") + prelude.emit(session_id="s", stream_kind="agent", extra={"isResume": True}) + prelude.mark("head_of_turn.handoff") + prelude.mark("pre_model") + prelude.mark("model") + prelude.emit_first_token() + + first_token = captured[-1] + assert {"FirstTokenMs", "HeadOfTurnMs", "HeadOfTurnHandoffMs", "PreModelMs", "ModelMs"} == set( + first_token["metrics"] + ) + assert first_token["properties"]["isResume"] is True + + def test_a_first_token_emit_never_raises(self, monkeypatch): + def _boom(**_kw): + raise RuntimeError("emf down") + + monkeypatch.setattr("apis.shared.observability.emf.emit_emf_metrics", _boom) + prelude = TurnPrelude() + prelude.mark("model") + prelude.emit_first_token() # must not raise + + class TestMetricNames: """The EMF metric name is derived from the stage name, not looked up. @@ -381,3 +576,26 @@ def emit(self, record): logger.removeHandler(handler) logger.setLevel(previous) return captured + + +def _first_token(prelude): + """The JSON payload `emit_first_token` would log, parsed back.""" + captured = {} + + class _Sink(logging.Handler): + def emit(self, record): + message = record.getMessage() + if message.startswith("turn_first_token "): + captured.update(json.loads(message[len("turn_first_token ") :])) + + logger = logging.getLogger("apis.inference_api.chat.turn_timing") + handler = _Sink() + logger.addHandler(handler) + previous = logger.level + logger.setLevel(logging.INFO) + try: + prelude.emit_first_token() + finally: + logger.removeHandler(handler) + logger.setLevel(previous) + return captured diff --git a/backend/tests/apis/inference_api/test_warmup.py b/backend/tests/apis/inference_api/test_warmup.py index c7a330517..b4f4b1f35 100644 --- a/backend/tests/apis/inference_api/test_warmup.py +++ b/backend/tests/apis/inference_api/test_warmup.py @@ -8,7 +8,7 @@ import sys import threading import types -from unittest.mock import patch +from unittest.mock import MagicMock, patch import pytest @@ -89,6 +89,101 @@ def flaky(service, **kwargs): assert calls == ["bedrock-runtime", "s3"] +class TestWarmSharedSession: + """The agent build's SDK clients are built on one process-wide session + (`apis.shared.aws_clients.shared_boto_session`). Building them here is + what lets the `shared_clients` arm's first turn skip the service-model + parses; discovering the strategy ids here is what lets it skip the one + control-plane call.""" + + def test_builds_each_client_once_on_the_shared_session(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + session = MagicMock(name="shared-session") + + with patch("apis.shared.aws_clients.shared_boto_session", return_value=session), patch( + "agents.main_agent.session.session_factory.warm_strategy_ids" + ): + warmup.warm_shared_session(["bedrock-agentcore", "bedrock-runtime"]) + + assert [c.args[0] for c in session.client.call_args_list] == ["bedrock-agentcore", "bedrock-runtime"] + assert all(c.kwargs["region_name"] == "us-west-2" for c in session.client.call_args_list) + + def test_the_default_list_covers_both_sdks(self): + # The Memory session manager (data + control plane) and Strands' + # BedrockModel (runtime) are the two SDKs handed the shared session. + assert set(warmup.WARM_SHARED_SESSION_SERVICES) == { + "bedrock-agentcore", + "bedrock-agentcore-control", + "bedrock-runtime", + } + + def test_discovers_the_strategy_ids_once_on_the_shared_session(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + monkeypatch.setenv("AGENTCORE_MEMORY_ID", "mem-warm") + from agents.main_agent.session import session_factory + + with patch("apis.shared.aws_clients.shared_boto_session", return_value=MagicMock()), patch.object( + session_factory, "_discover_strategy_ids", return_value=(None, None, None) + ) as discover: + warmup.warm_shared_session([]) + + discover.assert_called_once_with("mem-warm", "us-west-2", shared_session=True) + + def test_no_memory_configured_skips_discovery_and_does_not_raise(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + monkeypatch.delenv("AGENTCORE_MEMORY_ID", raising=False) + from agents.main_agent.session import session_factory + + with patch("apis.shared.aws_clients.shared_boto_session", return_value=MagicMock()), patch.object( + session_factory, "_discover_strategy_ids" + ) as discover: + warmup.warm_shared_session([]) + + discover.assert_not_called() + + def test_skips_without_a_region(self, monkeypatch): + monkeypatch.delenv("AWS_REGION", raising=False) + monkeypatch.delenv("AWS_DEFAULT_REGION", raising=False) + with patch("apis.shared.aws_clients.shared_boto_session") as session: + warmup.warm_shared_session() + session.assert_not_called() + + def test_the_kill_switch_skips_the_whole_step(self, monkeypatch): + """Off means the SDKs build their own sessions, so nothing here would + be used — including the one connection the strategy read opens.""" + monkeypatch.setenv("AWS_REGION", "us-west-2") + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", "false") + from agents.main_agent.session import session_factory + + with patch("apis.shared.aws_clients.shared_boto_session") as session, patch.object( + session_factory, "_discover_strategy_ids" + ) as discover: + warmup.warm_shared_session() + + session.assert_not_called() + discover.assert_not_called() + + def test_a_failing_client_does_not_stop_the_rest(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + session = MagicMock() + session.client.side_effect = [RuntimeError("no such service"), object()] + + with patch("apis.shared.aws_clients.shared_boto_session", return_value=session), patch( + "agents.main_agent.session.session_factory.warm_strategy_ids" + ) as ids: + warmup.warm_shared_session(["bedrock-agentcore-control", "bedrock-runtime"]) + + assert session.client.call_count == 2 + ids.assert_called_once() + + def test_run_warmup_includes_it(self): + with patch.object(warmup, "warm_modules"), patch.object(warmup, "warm_boto_clients"), patch.object( + warmup, "warm_shared_session" + ) as shared: + warmup.run_warmup() + shared.assert_called_once_with() + + class TestBackground: def test_runs_on_a_daemon_thread_and_returns_immediately(self, monkeypatch): monkeypatch.delenv(warmup.WARMUP_ENABLED_ENV, raising=False) diff --git a/backend/tests/apis/shared/tool_summaries/test_summarizer.py b/backend/tests/apis/shared/tool_summaries/test_summarizer.py index 4d116975b..889b00556 100644 --- a/backend/tests/apis/shared/tool_summaries/test_summarizer.py +++ b/backend/tests/apis/shared/tool_summaries/test_summarizer.py @@ -16,6 +16,7 @@ import pytest +from apis.shared import aws_clients from apis.shared.tool_summaries.summarizer import ( _build_prompt, _clean, @@ -50,6 +51,14 @@ def summaries_enabled(monkeypatch): monkeypatch.setenv("TOOL_SUMMARIES_ENABLED", "true") +@pytest.fixture(autouse=True) +def _fresh_bedrock_client(): + """The Bedrock client is cached per process; each test builds its own.""" + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + @pytest.fixture def bedrock(monkeypatch): """Patch boto3.client so no test ever reaches Bedrock.""" @@ -60,6 +69,19 @@ def bedrock(monkeypatch): return client +@pytest.mark.asyncio +async def test_batches_reuse_one_bedrock_client(bedrock): + """A client per batch paid botocore's model load and a fresh TLS handshake.""" + bedrock.converse.return_value = _response("Found the BIO 101 course") + factory = __import__("sys").modules["boto3"].client + + for _ in range(3): + assert await summarize_tool_batch(_calls()) == "Found the BIO 101 course" + + assert factory.call_count == 1 + assert bedrock.converse.call_count == 3 + + # -- the truncation regression ------------------------------------------- @@ -92,6 +114,27 @@ async def test_truncation_is_judged_by_stop_reason_not_by_length(bedrock): assert await summarize_tool_batch(_calls()) == long_but_finished + + +@pytest.mark.asyncio +@pytest.mark.parametrize("stop_reason", ["guardrail_intervened", "content_filtered", "tool_use", "something_new"]) +async def test_any_stop_other_than_end_turn_is_discarded(bedrock, stop_reason): + """A guardrail or content-filter stop can carry the refusal as its text. + + Only `end_turn` says the model finished a summary; anything else must + leave the deterministic line in place. + """ + bedrock.converse.return_value = _response("Sorry, the model cannot answer this.", stop_reason=stop_reason) + + assert await summarize_tool_batch(_calls()) is None + + +@pytest.mark.asyncio +async def test_a_missing_stop_reason_is_discarded(bedrock): + bedrock.converse.return_value = {"output": {"message": {"content": [{"text": "Found 3 active courses"}]}}} + + assert await summarize_tool_batch(_calls()) is None + # -- every other failure is also a None ----------------------------------- diff --git a/backend/tests/apis/test_personal_instructions.py b/backend/tests/apis/test_personal_instructions.py index 2de650e6a..90b43a956 100644 --- a/backend/tests/apis/test_personal_instructions.py +++ b/backend/tests/apis/test_personal_instructions.py @@ -68,11 +68,11 @@ def client(repo) -> TestClient: def test_saved_trimmed_kept_beside_the_default_model_and_cleared_by_blank(client): client.put("/users/me/settings", json={"defaultModelId": "m-1"}) saved = client.put("/users/me/settings", json={"personalInstructions": " Answer briefly. "}).json() - assert saved == {"defaultModelId": "m-1", "personalInstructions": "Answer briefly."} + assert saved == {"defaultModelId": "m-1", "personalInstructions": "Answer briefly.", "voiceId": None} assert client.get("/users/me/settings").json()["personalInstructions"] == "Answer briefly." cleared = client.put("/users/me/settings", json={"personalInstructions": " "}).json() - assert cleared == {"defaultModelId": "m-1", "personalInstructions": None} + assert cleared == {"defaultModelId": "m-1", "personalInstructions": None, "voiceId": None} def test_too_long_is_refused(client): diff --git a/backend/tests/apis/test_voice_settings.py b/backend/tests/apis/test_voice_settings.py new file mode 100644 index 000000000..7ce965e9e --- /dev/null +++ b/backend/tests/apis/test_voice_settings.py @@ -0,0 +1,84 @@ +"""The user's voice-mode voice, saved in their settings. + +``voiceId`` is the Nova 2 Sonic voice the SPA sends on every voice connect. +A stored id Bedrock rejects would break every later session, so the route +refuses unknown ids; blank clears back to the platform default. +""" + +from __future__ import annotations + +import boto3 +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from moto import mock_aws + +import apis.app_api.user_settings.routes as settings_routes +from apis.shared.auth.dependencies import get_current_user_from_session +from apis.shared.auth.models import User +from apis.shared.user_settings.repository import UserSettingsRepository + +TABLE = "test-user-settings" +USER = User(user_id="u-1", email="u1@example.edu", name="U", roles=["default"]) + + +@pytest.fixture() +def repo(monkeypatch) -> UserSettingsRepository: + for k, v in { + "AWS_DEFAULT_REGION": "us-east-1", + "AWS_ACCESS_KEY_ID": "testing", + "AWS_SECRET_ACCESS_KEY": "testing", + "AWS_SESSION_TOKEN": "testing", + "DYNAMODB_USER_SETTINGS_TABLE_NAME": TABLE, + }.items(): + monkeypatch.setenv(k, v) + with mock_aws(): + boto3.client("dynamodb", region_name="us-east-1").create_table( + TableName=TABLE, + KeySchema=[{"AttributeName": "PK", "KeyType": "HASH"}, {"AttributeName": "SK", "KeyType": "RANGE"}], + AttributeDefinitions=[{"AttributeName": n, "AttributeType": "S"} for n in ("PK", "SK")], + BillingMode="PAY_PER_REQUEST", + ) + yield UserSettingsRepository(table_name=TABLE) + + +@pytest.fixture() +def client(repo) -> TestClient: + app = FastAPI() + app.include_router(settings_routes.router) + app.dependency_overrides[get_current_user_from_session] = lambda: USER + app.dependency_overrides[settings_routes.get_user_settings_repository] = lambda: repo + return TestClient(app) + + +def test_default_is_unset(client): + assert client.get("/users/me/settings").json()["voiceId"] is None + + +def test_saves_a_catalog_voice_and_normalises_it(client): + res = client.put("/users/me/settings", json={"voiceId": " Matthew "}) + assert res.status_code == 200, res.text + assert res.json()["voiceId"] == "matthew" + assert client.get("/users/me/settings").json()["voiceId"] == "matthew" + + +def test_refuses_a_voice_bedrock_would_reject(client): + res = client.put("/users/me/settings", json={"voiceId": "siri"}) + assert res.status_code == 422 + assert client.get("/users/me/settings").json()["voiceId"] is None + + +def test_blank_clears_back_to_the_default(client): + client.put("/users/me/settings", json={"voiceId": "amy"}) + res = client.put("/users/me/settings", json={"voiceId": ""}) + assert res.status_code == 200, res.text + assert res.json()["voiceId"] is None + assert client.get("/users/me/settings").json()["voiceId"] is None + + +def test_other_settings_survive_a_voice_change(client): + client.put("/users/me/settings", json={"personalInstructions": "Be brief."}) + client.put("/users/me/settings", json={"voiceId": "lupe"}) + body = client.get("/users/me/settings").json() + assert body["personalInstructions"] == "Be brief." + assert body["voiceId"] == "lupe" diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 31194d440..f7b262340 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -133,6 +133,26 @@ def _clear_config_cache(): config_cache.get_config_cache().clear() +@pytest.fixture(autouse=True) +def _reset_cached_aws_clients(): + """Drop every process-cached boto3 client and resource between tests. + + The knowledge base's KB_Record reads and document-status lookups use + ``apis.shared.aws_clients`` handles, and they are reached from tests all over + the tree — many with their own ``mock_aws()`` or a ``patch("boto3.resource")`` + rather than the per-directory ``aws`` fixtures that already reset this cache. + A handle cached under one test's mock (or one test's MagicMock) must never + answer the next test's call. + """ + from apis.shared.aws_clients import reset_cached_clients + + reset_cached_clients() + try: + yield + finally: + reset_cached_clients() + + @pytest.fixture(autouse=True) def _clear_env_config_bleed(): saved = { diff --git a/backend/tests/costs/test_calculator.py b/backend/tests/costs/test_calculator.py index 1fecd43d7..49bf7d7aa 100644 --- a/backend/tests/costs/test_calculator.py +++ b/backend/tests/costs/test_calculator.py @@ -388,3 +388,65 @@ def test_mixed_read_write_call_prices_each_bucket_once(self): + breakdown.cache_read_cost + breakdown.cache_write_cost ) + + +# Nova 2 Sonic, us-west-2, Price List API `AmazonBedrock` offer file +# (usagetype USW2-NovaSonic2.0-*, 2026-09-30). Speech is ~10x text. +NOVA_2_SONIC_PRICING = { + "inputPricePerMtok": 0.319, + "outputPricePerMtok": 2.651, + "speechInputPricePerMtok": 3.0, + "speechOutputPricePerMtok": 12.0, +} + + +class TestCalculateVoiceCost: + """Speech-to-speech sessions bill four buckets on two rate cards.""" + + def test_prices_each_modality_on_its_own_rate(self): + usage = { + "inputTokens": 2_000_000, + "outputTokens": 2_000_000, + "speechInputTokens": 1_000_000, + "textInputTokens": 1_000_000, + "speechOutputTokens": 1_000_000, + "textOutputTokens": 1_000_000, + } + total, breakdown = CostCalculator.calculate_voice_cost(usage, NOVA_2_SONIC_PRICING) + assert breakdown.input_cost == pytest.approx(3.0 + 0.319) + assert breakdown.output_cost == pytest.approx(12.0 + 2.651) + assert total == pytest.approx(3.319 + 14.651) + assert breakdown.cache_read_cost == 0.0 + assert breakdown.cache_write_cost == 0.0 + + def test_no_split_prices_the_totals_at_the_speech_rates(self): + # The conservative direction: over-count text as speech, never the + # other way round. + usage = {"inputTokens": 1_000_000, "outputTokens": 1_000_000} + total, breakdown = CostCalculator.calculate_voice_cost(usage, NOVA_2_SONIC_PRICING) + assert breakdown.input_cost == pytest.approx(3.0) + assert breakdown.output_cost == pytest.approx(12.0) + assert total == pytest.approx(15.0) + + def test_a_split_that_undercounts_the_total_bills_the_remainder_as_speech(self): + usage = { + "inputTokens": 1_000_000, + "outputTokens": 0, + "speechInputTokens": 250_000, + "textInputTokens": 250_000, + } + _, breakdown = CostCalculator.calculate_voice_cost(usage, NOVA_2_SONIC_PRICING) + # 750k at speech ($3/M) + 250k at text ($0.319/M) + assert breakdown.input_cost == pytest.approx(0.75 * 3.0 + 0.25 * 0.319) + + def test_row_without_speech_rates_falls_back_to_its_text_rates(self): + # Better than $0, and UnmeteredModelCall stays quiet; the fix is the row. + pricing = {"inputPricePerMtok": 1.0, "outputPricePerMtok": 4.0} + usage = {"inputTokens": 1_000_000, "outputTokens": 1_000_000} + total, _ = CostCalculator.calculate_voice_cost(usage, pricing) + assert total == pytest.approx(5.0) + + def test_none_tokens_count_as_zero(self): + usage = {"inputTokens": None, "outputTokens": None, "speechInputTokens": None} + total, _ = CostCalculator.calculate_voice_cost(usage, NOVA_2_SONIC_PRICING) + assert total == 0.0 diff --git a/backend/tests/lambdas/test_kb_byte_cap_lifecycle.py b/backend/tests/lambdas/test_kb_byte_cap_lifecycle.py index 718cac094..a94f38dae 100644 --- a/backend/tests/lambdas/test_kb_byte_cap_lifecycle.py +++ b/backend/tests/lambdas/test_kb_byte_cap_lifecycle.py @@ -16,6 +16,11 @@ * **Deleting a ``failed`` document released bytes it never reserved.** Prod has unsettled ``failed`` rows from before their knowledge base migrated, with multi-megabyte ``sizeBytes``; each delete credited that much allowance back. +* **An import was committed out of a reservation it never made.** A file-source + import writes its real ``sizeBytes`` before its ``PUT`` but reserves nothing, and + the consumer read that size as the reservation. In prod two Drive imports left a + migrated knowledge base's ``reservedBytes`` and ``totalBytes`` short by exactly + their 499,835 bytes, so they never counted against the cap. """ from __future__ import annotations @@ -149,6 +154,18 @@ def _upload_and_complete(table, document_id, declared, real): ic.set_document_terminal(ASSISTANT_ID, document_id, ic.STATUS_COMPLETE) +def _import_and_complete(table, document_id, size, adapter="google-drive"): + """A file-source import (or crawl, or sync): nothing reserved at request time, + and ``sizeBytes`` already real by the time the consumer reads the row.""" + _seed_doc(table, document_id, size=size, sourceAdapterKey=adapter) + _put_object(document_id, size) + ic._reconcile_bytes_on_complete( + ASSISTANT_ID, document_id, BUCKET, _key(document_id), _kb(table), + ic._declared_bytes(_doc(table, document_id)), + ) + ic.set_document_terminal(ASSISTANT_ID, document_id, ic.STATUS_COMPLETE) + + def _delete(document_id): return asyncio.run(soft_delete_document(ASSISTANT_ID, document_id, OWNER)) @@ -244,6 +261,70 @@ def test_an_unsettled_failed_row_never_reserved(self, table, owned): assert _counters(table) == (0, 5000, 5000) +class TestImportsReserveAtIngestion: + def test_an_import_is_counted_against_the_cap(self, table): + """MUTATION GUARD: read ``sizeBytes`` as the reservation again and this ends + at (500, -500, 0) — stored, but never counted in ``totalBytes``.""" + _seed_kb(table) + + _import_and_complete(table, "DOC-imp", 500) + + assert _counters(table) == (500, 0, 500) + assert _doc(table, "DOC-imp")["committedBytes"] == 500 + + def test_the_prod_shape_does_not_recur(self, table): + """A migrated corpus still held as a reservation, then two imports. Before + the fix this ended at (499835, 1664866, 2164701).""" + corpus = 2_164_701 + _seed_kb(table, storedBytes=0, reservedBytes=corpus, totalBytes=corpus) + _seed_doc(table, "DOC-migrated", status="complete", size=corpus) + + _import_and_complete(table, "DOC-imp1", 140_270) + _import_and_complete(table, "DOC-imp2", 359_565) + + assert _counters(table) == (499_835, corpus, corpus + 499_835) + + def test_a_crawled_page_reserves_whether_or_not_its_size_landed_first(self, table): + """The crawler writes ``sizeBytes`` after its ``PUT``, so the consumer can + read either 0 or the real size. Both must end the same way.""" + _seed_kb(table) + + _import_and_complete(table, "DOC-page", 300, adapter="http") + + assert _counters(table) == (300, 0, 300) + + def test_an_import_over_the_cap_fails_instead_of_slipping_past_it(self, table): + cap = byte_cap.effective_cap() + _seed_kb(table, storedBytes=cap - 100, reservedBytes=0, totalBytes=cap - 100) + _seed_doc(table, "DOC-big", size=500, sourceAdapterKey="google-drive") + _put_object("DOC-big", 500) + + with pytest.raises(byte_cap.ByteCapExceeded): + ic._reconcile_bytes_on_complete( + ASSISTANT_ID, "DOC-big", BUCKET, _key("DOC-big"), _kb(table), + ic._declared_bytes(_doc(table, "DOC-big")), + ) + + assert _counters(table) == (cap - 100, 0, cap - 100) + + def test_an_upload_still_reserves_its_declared_size(self, table): + assert ic._declared_bytes({"sizeBytes": Decimal(700)}) == 700 + assert ic._declared_bytes({"sizeBytes": Decimal(700), "sourceAdapterKey": "google-drive"}) == 0 + assert ic._declared_bytes(None) == 0 + + def test_deleting_an_import_in_flight_releases_nothing(self, table, owned): + """MUTATION GUARD: without the provenance check the delete releases 400 + bytes out of the other document's reservation: (0, 300, 300).""" + _seed_kb(table) + _seed_doc(table, "DOC-upload", size=700) + byte_cap.reserve(ASSISTANT_ID, ASSISTANT_ID, 700, CAP) + _seed_doc(table, "DOC-imp", size=400, sourceAdapterKey="google-drive") + + _delete("DOC-imp") + + assert _counters(table) == (0, 700, 700) + + # ── Guards ─────────────────────────────────────────────────────────────────── class TestCountersNeverGoNegative: def test_a_release_larger_than_the_reservations_is_refused(self, table): diff --git a/backend/tests/lambdas/test_kb_sync_managed_reingest.py b/backend/tests/lambdas/test_kb_sync_managed_reingest.py new file mode 100644 index 000000000..76731c4a0 --- /dev/null +++ b/backend/tests/lambdas/test_kb_sync_managed_reingest.py @@ -0,0 +1,536 @@ +"""A synced source that changed must be re-ingested into its MANAGED knowledge base. + +The KB sync worker (``kb_sync/worker.py``) detects a changed Drive file and +overwrites the document's existing S3 object, relying on the bucket's +``ObjectCreated`` event to re-run ingestion. That works for the legacy pipeline. +For a managed knowledge base the event reaches the ingestion consumer, which found +a ``complete`` row with ``byteCapSettled`` and returned "already-settled" — the +same early exit that makes EventBridge redelivery idempotent. The new bytes were +never ingested, and the size change never reached the byte ledger. + +These tests drive the real path end to end against moto: the real sync worker +(Drive and the token vault stubbed at its seams) stages to a real S3 object, and +the real consumer handles the resulting event, against a fake Bedrock that models +document status the way the service reports it. +""" + +from __future__ import annotations + +import asyncio +from datetime import datetime, timezone +from decimal import Decimal +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import boto3 +import pytest + +from apis.app_api.documents.models import DocumentProvenance +from apis.app_api.documents.services.document_service import create_document, soft_delete_document +from apis.app_api.file_sources.models import DownloadedFile +from apis.app_api.kb_migration import ingestion_consumer as ic +from apis.app_api.kb_sync import worker +from apis.shared.assistants.service import create_assistant +from apis.shared.sync_policies.service import create_sync_policy + +REGION = "us-east-1" +BUCKET = "test-reingest-docs" +USER_ID = "user-reingest" +FILE_ID = "drive-file-reingest" +CONNECTOR_ID = "google-workspace" +DOCUMENT_ID = "doc-reingest" +FILENAME = "report.pdf" + + +# ── fakes ──────────────────────────────────────────────────────────────────── +class FakeBedrock: + """ManagedKbBackend's surface, modelling per-document status like the service. + + ``ingest`` reads the object as it is in S3 at submission — Bedrock ingests from + the S3 location — and re-ingesting an id replaces the document. A submitted + document reports ``IN_PROGRESS`` on the next probe, and ``INDEXED`` (or + ``FAILED``) on the one after, stamped with the time it flipped. While ``hold`` + is set it stays ``IN_PROGRESS``. ``stale_probes`` makes the probes straight + after a submit still report the PREVIOUS version's status and timestamp, which + is what a document Bedrock already holds can look like. + """ + + def __init__(self, stale_probes=0, fail=False): + self.ingests: list = [] + self.indexed_content = None + self.hold = False + self.fail = fail + self.stale_probes = stale_probes + self._status = "NOT_FOUND" + self._updated_at = None + self._pending = None + self._probes_until_flip = 0 + self._stale_left = 0 + self._agent_client = SimpleNamespace(get_knowledge_base_documents=self._get_documents) + + # -- the private surface `document_status` reuses -------------------------- + def _agent(self): + return self._agent_client + + def _locate(self, kb_ref): + return ("KB123", "DS456") + + def _get_documents(self, **kwargs): + if self._stale_left: + self._stale_left -= 1 + elif self._pending is not None: + if self.hold or self._probes_until_flip: + self._probes_until_flip = max(self._probes_until_flip - 1, 0) + self._status = "IN_PROGRESS" + self._updated_at = datetime.now(timezone.utc) + else: + self._status = "FAILED" if self.fail else "INDEXED" + self._updated_at = datetime.now(timezone.utc) + if not self.fail: + self.indexed_content = self._pending + self._pending = None + if self._status == "NOT_FOUND": + return {"documentDetails": []} + return {"documentDetails": [{"status": self._status, "updatedAt": self._updated_at}]} + + # -- the protocol surface -------------------------------------------------- + async def ingest(self, kb_ref, source): + body = boto3.client("s3", region_name=REGION).get_object(Bucket=BUCKET, Key=source.s3_key) + self._pending = body["Body"].read() + self.ingests.append(self._pending) + self._probes_until_flip = 1 + self._stale_left = self.stale_probes + + async def search(self, kb_ref, query, top_k=5, retrieval_filter=None): + if self.indexed_content is None: + return [] + chunk = MagicMock() + chunk.metadata = {"document_id": DOCUMENT_ID} + return [chunk] + + +class FakeDriveAdapter: + def __init__(self, version, content): + self.version = version + self.content = content + + async def get_file_metadata(self, access_token, file_id): + return {"version": self.version, "trashed": False} + + async def download(self, access_token, file_id): + return DownloadedFile(content=self.content, filename=FILENAME, content_type="application/pdf") + + +# ── fixtures ───────────────────────────────────────────────────────────────── +@pytest.fixture(autouse=True) +def _fast_polls(monkeypatch): + monkeypatch.setattr(ic, "INDEXED_POLL_TIMEOUT_SECONDS", 0.2) + monkeypatch.setattr(ic, "INDEXED_POLL_INTERVAL_SECONDS", 0.001) + monkeypatch.setattr(ic, "RETRIEVABLE_POLL_TIMEOUT_SECONDS", 0.05) + monkeypatch.setattr(ic, "RETRIEVABLE_POLL_INTERVAL_SECONDS", 0.001) + + +@pytest.fixture() +def drive(monkeypatch): + """The worker's seams to Google and the vault. Returns a setter for the file.""" + provider = SimpleNamespace(provider_id=CONNECTOR_ID, scopes=[], custom_parameters=None) + + async def fake_get_provider(self, provider_id): + return provider if provider_id == CONNECTOR_ID else None + + async def fake_resolve(provider, user_id): + return "test-access-token" + + monkeypatch.setattr( + "apis.shared.oauth.provider_repository.OAuthProviderRepository.get_provider", fake_get_provider + ) + monkeypatch.setattr(worker, "_resolve_access_token", fake_resolve) + + def set_file(version, content): + adapter = FakeDriveAdapter(version, content) + monkeypatch.setattr(worker.registry, "get", lambda key: adapter if key == "google-drive" else None) + + return set_file + + +@pytest.fixture() +def kb(assistants_table, monkeypatch): + """A managed agent holding one Drive-imported document with a daily sync policy.""" + monkeypatch.setenv("S3_ASSISTANTS_DOCUMENTS_BUCKET_NAME", BUCKET) + boto3.client("s3", region_name=REGION).create_bucket(Bucket=BUCKET) + + async def _build(): + assistant = await create_assistant( + owner_id=USER_ID, owner_name="U", name="A", description="d", + instructions="i", vector_index_id="assistants-index", + ) + assistant_id = assistant.assistant_id + # Imports reserve nothing at request time, so the row declares no size. + document = await create_document( + assistant_id=assistant_id, + filename=FILENAME, + content_type="application/pdf", + size_bytes=0, + s3_key=f"assistants/{assistant_id}/documents/{DOCUMENT_ID}/{FILENAME}", + document_id=DOCUMENT_ID, + provenance=DocumentProvenance( + source_connector_id=CONNECTOR_ID, + source_adapter_key="google-drive", + source_file_id=FILE_ID, + imported_by_user_id=USER_ID, + source_etag="1", + ), + ) + policy = await create_sync_policy( + assistant_id=assistant_id, source_type="drive_file", source_ref=DOCUMENT_ID, + interval="daily", created_by_user_id=USER_ID, + ) + return assistant_id, document, policy + + assistant_id, document, policy = asyncio.run(_build()) + return SimpleNamespace( + table=assistants_table, assistant_id=assistant_id, key=document.s3_key, policy=policy + ) + + +def _seed_kb_record(kb, engine="managed", **counters): + item = { + "PK": f"AST#{kb.assistant_id}", + "SK": f"KB#{kb.assistant_id}", + "appKbId": kb.assistant_id, + "ownerUserId": USER_ID, + "awsKbId": "KB123", + "awsDataSourceId": "DS456", + } + if engine: + item["retrievalEngine"] = engine + item.update({k: Decimal(v) for k, v in counters.items()}) + kb.table.put_item(Item=item) + + +def _put(kb, content): + boto3.client("s3", region_name=REGION).put_object(Bucket=BUCKET, Key=kb.key, Body=content) + + +def _event(kb, bedrock): + """The ObjectCreated event the S3 overwrite produces, handled by the consumer.""" + with patch("apis.shared.kb_backend.managed_backend.ManagedKbBackend", return_value=bedrock): + return ic.handle_object(BUCKET, kb.key) + + +def _sync(kb): + return asyncio.run( + worker.run_sync({ + "policyId": kb.policy.policy_id, + "assistantId": kb.assistant_id, + "sourceType": "drive_file", + "sourceRef": DOCUMENT_ID, + }) + ) + + +def _doc(kb): + return kb.table.get_item(Key={"PK": f"AST#{kb.assistant_id}", "SK": f"DOC#{DOCUMENT_ID}"}).get("Item") + + +def _counters(kb): + item = kb.table.get_item(Key={"PK": f"AST#{kb.assistant_id}", "SK": f"KB#{kb.assistant_id}"}).get("Item") + return tuple(int((item or {}).get(k) or 0) for k in ("storedBytes", "reservedBytes", "totalBytes")) + + +def _delete(kb): + with patch( + "apis.shared.assistants.service.get_assistant", + new_callable=AsyncMock, + return_value=SimpleNamespace(assistant_id=kb.assistant_id, owner_id=USER_ID), + ): + asyncio.run(soft_delete_document(kb.assistant_id, DOCUMENT_ID, USER_ID)) + + +def _imported_and_complete(kb, bedrock, content=b"a" * 1000): + """The import's own first ingestion, to ``complete`` with its bytes committed.""" + _seed_kb_record(kb) + _put(kb, content) + result = _event(kb, bedrock) + assert result["ingested"] is True + assert _doc(kb)["status"] == "complete" + assert _counters(kb) == (len(content), 0, len(content)) + + +# ── the defect ─────────────────────────────────────────────────────────────── +class TestAChangedFileIsReingested: + def test_the_new_bytes_reach_the_knowledge_base(self, kb, drive): + """The repro. MUTATION GUARD: without the re-ingest branch the consumer + returns "already-settled", Bedrock is never asked, and it keeps serving the + version the file had at import.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + + drive("2", b"b" * 1500) + assert _sync(kb)["result"] == "changed" + result = _event(kb, bedrock) + + assert result.get("note") != "already-settled" + assert bedrock.ingests == [b"a" * 1000, b"b" * 1500] + assert bedrock.indexed_content == b"b" * 1500 + assert _doc(kb)["status"] == "complete" + + def test_the_ledger_grows_by_the_size_delta(self, kb, drive): + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + + drive("2", b"b" * 1500) + _sync(kb) + _event(kb, bedrock) + + assert _counters(kb) == (1500, 0, 1500) + assert _doc(kb)["committedBytes"] == 1500 + + def test_the_ledger_shrinks_by_the_size_delta(self, kb, drive): + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + + drive("2", b"c" * 400) + _sync(kb) + _event(kb, bedrock) + + assert bedrock.indexed_content == b"c" * 400 + assert _counters(kb) == (400, 0, 400) + assert _doc(kb)["committedBytes"] == 400 + + def test_a_delete_after_the_reingest_refunds_the_new_size(self, kb, drive): + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + _event(kb, bedrock) + + _delete(kb) + + assert _counters(kb) == (0, 0, 0) + + +# ── idempotency under EventBridge redelivery ───────────────────────────────── +class TestRedeliveryIsIdempotent: + def test_a_redelivery_after_the_reingest_does_nothing(self, kb, drive): + """The early exit the defect came from still does its job: once the new + version is recorded as ingested, its event is a plain redelivery.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + _event(kb, bedrock) + + result = _event(kb, bedrock) + + assert result["note"] == "already-settled" + assert len(bedrock.ingests) == 2 + assert _counters(kb) == (1500, 0, 1500) + + def test_a_redelivery_while_indexing_neither_resubmits_nor_rereserves(self, kb, drive): + """Lambda's async retry is capped at 2, so a slow re-ingest spans + deliveries. Re-submitting would restart Bedrock's work (the dev failure + the first-ingest path documents); re-reserving would leak the growth.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + + bedrock.hold = True + for _ in range(2): + with pytest.raises(ic.IngestionRoutingError): + _event(kb, bedrock) + assert _counters(kb) == (1000, 500, 1500) + assert _doc(kb)["status"] == "complete", "the previous version stops being served" + + bedrock.hold = False + result = _event(kb, bedrock) + + assert result["note"] == "reingested" + assert len(bedrock.ingests) == 2 + assert _counters(kb) == (1500, 0, 1500) + assert not any(k.startswith("reingest") for k in _doc(kb)) + + def test_a_status_from_before_the_submit_is_not_the_new_version(self, kb, drive): + """Bedrock already holds this id, so the probes straight after a + re-submit can still describe the previous version. MUTATION GUARD: drop + ``not_before`` and the consumer records the new version as ingested while + Bedrock is still serving the old one.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + + bedrock.stale_probes = 3 + _event(kb, bedrock) + + assert bedrock.indexed_content == b"b" * 1500 + assert _doc(kb)["ingestedContentHash"] == worker._sha256(b"b" * 1500) + + def test_a_newer_change_supersedes_an_unfinished_reingest(self, kb, drive): + """Two changes, the first never finishing: its growth reservation is + returned, the newer version is submitted, and only its size is counted.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + bedrock.hold = True + with pytest.raises(ic.IngestionRoutingError): + _event(kb, bedrock) + assert _counters(kb) == (1000, 500, 1500) + + drive("3", b"c" * 1200) + _sync(kb) + bedrock.hold = False + result = _event(kb, bedrock) + + assert result["note"] == "reingested" + assert bedrock.ingests[-1] == b"c" * 1200 + assert bedrock.indexed_content == b"c" * 1200 + assert _counters(kb) == (1200, 0, 1200) + assert _event(kb, bedrock)["note"] == "already-settled" + + +# ── the byte cap ───────────────────────────────────────────────────────────── +class TestTheByteCapGatesTheNewVersion: + @pytest.fixture() + def cap_1200(self, monkeypatch): + monkeypatch.setenv("MANAGED_KB_PER_OWNER_DEFAULT_BYTES", "1200") + monkeypatch.setenv("MANAGED_KB_PER_KB_CEILING_BYTES", "1200") + + def test_growth_past_the_cap_keeps_the_previous_version(self, kb, drive, cap_1200): + """Checked BEFORE submitting: once Bedrock has the new bytes the old + version is gone, and failing then would leave the owner with nothing.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + + result = _event(kb, bedrock) + + assert result["note"] == "byte-cap-exceeded" + assert bedrock.ingests == [b"a" * 1000] + assert bedrock.indexed_content == b"a" * 1000 + doc = _doc(kb) + assert doc["status"] == "complete" + assert doc["committedBytes"] == 1000 + assert "storage limit" in doc["ingestionError"] + assert _counters(kb) == (1000, 0, 1000) + + def test_the_next_sync_retries_once_there_is_room(self, kb, drive, cap_1200, monkeypatch): + """The refused version is re-staged by the next sync run (its gates were + cleared) and goes through once the owner has room.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + _event(kb, bedrock) + + assert _sync(kb)["result"] == "changed", "the refused version is never retried" + + monkeypatch.setenv("MANAGED_KB_PER_OWNER_DEFAULT_BYTES", "2000") + monkeypatch.setenv("MANAGED_KB_PER_KB_CEILING_BYTES", "2000") + result = _event(kb, bedrock) + + assert result["note"] == "reingested" + assert _counters(kb) == (1500, 0, 1500) + assert "ingestionError" not in _doc(kb) + + def test_shrinking_is_never_refused(self, kb, drive, cap_1200): + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock, content=b"a" * 1200) + + drive("2", b"b" * 900) + _sync(kb) + + assert _event(kb, bedrock)["note"] == "reingested" + assert _counters(kb) == (900, 0, 900) + + +# ── terminal outcomes ──────────────────────────────────────────────────────── +class TestTerminalOutcomes: + def test_a_delete_mid_reingest_returns_every_byte(self, kb, drive): + """The delete refunds the committed old size and releases the growth + reserved for the new one; the consumer, finding the row deleting, touches + neither.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + bedrock.hold = True + with pytest.raises(ic.IngestionRoutingError): + _event(kb, bedrock) + + _delete(kb) + assert _counters(kb) == (0, 0, 0) + + bedrock.hold = False + assert _event(kb, bedrock)["note"] == "document-deleted" + assert _counters(kb) == (0, 0, 0) + + def test_bedrock_failing_the_new_version_releases_its_growth(self, kb, drive): + """The document is failed like any other Bedrock failure. Its old size + stays committed — the object is still stored — so a delete refunds it.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + drive("2", b"b" * 1500) + _sync(kb) + + bedrock.fail = True + result = _event(kb, bedrock) + + assert result["status"] == "FAILED" + assert _doc(kb)["status"] == "failed" + assert _counters(kb) == (1000, 0, 1000) + + _delete(kb) + assert _counters(kb) == (0, 0, 0) + + def test_a_failed_document_recovers_on_a_changed_source(self, kb, drive): + """New bytes deserve a fresh attempt. A failed import reserved and + committed nothing, so its whole new size is reserved and committed.""" + bedrock = FakeBedrock(fail=True) + _seed_kb_record(kb) + _put(kb, b"a" * 1000) + _event(kb, bedrock) + assert _doc(kb)["status"] == "failed" + assert _counters(kb) == (0, 0, 0) + + bedrock.fail = False + drive("2", b"b" * 1500) + _sync(kb) + result = _event(kb, bedrock) + + assert result["note"] == "reingested" + assert _doc(kb)["status"] == "complete" + assert _doc(kb)["committedBytes"] == 1500 + assert _counters(kb) == (1500, 0, 1500) + + def test_an_unsynced_redelivery_keeps_the_settled_early_exit(self, kb): + """No staged version, no change in behaviour.""" + bedrock = FakeBedrock() + _imported_and_complete(kb, bedrock) + + assert _event(kb, bedrock)["note"] == "already-settled" + assert len(bedrock.ingests) == 1 + + +# ── legacy ─────────────────────────────────────────────────────────────────── +class TestTheLegacyPathIsUnaffected: + def test_a_legacy_kb_is_left_to_its_own_pipeline(self, kb, drive): + """The marker is written regardless of engine; the consumer still routes + a legacy document away before reading it, and counts nothing.""" + _seed_kb_record(kb, engine=None) + _put(kb, b"a" * 1000) + bedrock = FakeBedrock() + + drive("2", b"b" * 1500) + assert _sync(kb)["result"] == "changed" + result = _event(kb, bedrock) + + assert result["routed"] == "legacy" + assert bedrock.ingests == [] + doc = _doc(kb) + assert doc["stagedContentHash"] == worker._sha256(b"b" * 1500) + assert doc["previousChunkCount"] == 0, "the legacy shrinkage stash is unchanged" + assert _counters(kb) == (0, 0, 0) diff --git a/backend/tests/lambdas/test_kb_sync_worker.py b/backend/tests/lambdas/test_kb_sync_worker.py index 86197b801..6086ba139 100644 --- a/backend/tests/lambdas/test_kb_sync_worker.py +++ b/backend/tests/lambdas/test_kb_sync_worker.py @@ -166,6 +166,7 @@ async def test_same_bytes_advances_etag_without_staging( assert staged == [] doc = assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-1"})["Item"] assert doc["sourceEtag"] == "42" # gate 1 passes next run + assert "stagedContentHash" not in doc # nothing staged, nothing to re-ingest async def test_changed_bytes_staged_with_stash(self, assistants_table, staged, token_ok, provider_ok, monkeypatch): adapter = FakeDriveAdapter(metadata={"version": "42", "trashed": False}, content=b"new bytes") @@ -183,6 +184,8 @@ async def test_changed_bytes_staged_with_stash(self, assistants_table, staged, t assert item["sourceEtag"] == "42" assert item["contentHash"] == worker._sha256(b"new bytes") assert item["previousChunkCount"] == 7 + # Marks the version a managed KB's consumer still has to re-ingest. + assert item["stagedContentHash"] == worker._sha256(b"new bytes") assert "lastSyncedAt" in item updated_policy = await get_sync_policy(assistant_id, policy.policy_id) assert updated_policy.last_result == "changed" @@ -199,6 +202,120 @@ async def test_trashed_is_grace_skip(self, assistants_table, staged, token_ok, p assert adapter.download_calls == 0 +class TestStageFailure: + """A failed S3 stage must not leave the change-detection gates advanced — + otherwise the next run matches them and the change is never staged.""" + + @pytest.fixture() + def flaky_stage(self, monkeypatch): + """First stage raises (S3 put failure), later ones succeed.""" + calls = [] + + def stage(*args): + calls.append(args) + if len(calls) == 1: + raise RuntimeError("S3 PutObject failed") + + monkeypatch.setattr(worker, "_stage_to_s3", stage) + return calls + + def _doc(self, assistants_table, assistant_id): + return assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-1"})["Item"] + + async def test_failed_stage_is_restaged_next_run( + self, assistants_table, flaky_stage, token_ok, provider_ok, monkeypatch + ): + adapter = FakeDriveAdapter(metadata={"version": "42", "trashed": False}, content=b"new bytes") + _use_adapter(monkeypatch, adapter) + assistant_id, _, policy = await _setup(assistants_table, etag="41", chunk_count=7) + + first = await worker.run_sync(_payload(assistant_id, policy)) + + assert first["result"] == "failed" + item = self._doc(assistants_table, assistant_id) + assert item["sourceEtag"] == "41" + assert "contentHash" not in item + assert "stagedContentHash" not in item + assert "lastSyncedAt" not in item + + # Same Drive version, same bytes: both gates must still miss. + second = await worker.run_sync(_payload(assistant_id, policy)) + + assert second["result"] == "changed" + assert len(flaky_stage) == 2 + assert flaky_stage[1][1] == b"new bytes" + item = self._doc(assistants_table, assistant_id) + assert item["sourceEtag"] == "42" + assert item["contentHash"] == worker._sha256(b"new bytes") + assert item["stagedContentHash"] == worker._sha256(b"new bytes") + + async def test_failed_stage_restores_previous_sync_values( + self, assistants_table, flaky_stage, token_ok, provider_ok, monkeypatch + ): + adapter = FakeDriveAdapter(metadata={"version": "42", "trashed": False}, content=b"new bytes") + _use_adapter(monkeypatch, adapter) + assistant_id, _, policy = await _setup(assistants_table, etag="41", content_hash="old-hash") + assistants_table.update_item( + Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-1"}, + UpdateExpression="SET stagedContentHash = :h, lastSyncedAt = :t", + ExpressionAttributeValues={":h": "old-hash", ":t": "2026-01-01T00:00:00Z"}, + ) + + await worker.run_sync(_payload(assistant_id, policy)) + + item = self._doc(assistants_table, assistant_id) + assert item["sourceEtag"] == "41" + assert item["contentHash"] == "old-hash" + assert item["stagedContentHash"] == "old-hash" + assert item["lastSyncedAt"] == "2026-01-01T00:00:00Z" + + async def test_rollback_leaves_a_newer_write_alone(self, assistants_table): + from apis.app_api.kb_sync import records + + assistant_id, _, _ = await _setup(assistants_table, etag="41") + written = {"sourceEtag": "42", "contentHash": "h2", "stagedContentHash": "h2"} + records.update_document_sync_fields( + assistant_id, "doc-1", source_etag="42", content_hash="h2", staged_content_hash="h2" + ) + # A later run staged a newer version before this rollback landed. + records.update_document_sync_fields( + assistant_id, "doc-1", source_etag="43", content_hash="h3", staged_content_hash="h3" + ) + + rolled_back = records.rollback_document_sync_fields( + assistant_id, "doc-1", written=written, previous={"sourceEtag": "41"} + ) + + assert rolled_back is False + item = self._doc(assistants_table, assistant_id) + assert (item["sourceEtag"], item["contentHash"], item["stagedContentHash"]) == ("43", "h3", "h3") + + async def test_rollback_does_not_resurrect_gates_the_consumer_cleared(self, assistants_table): + """The managed consumer's over-cap refusal REMOVEs the gates to force a + re-stage; a late rollback must not write them back.""" + from apis.app_api.kb_sync import records + + assistant_id, _, _ = await _setup(assistants_table, etag="41", content_hash="h1") + records.update_document_sync_fields( + assistant_id, "doc-1", source_etag="42", content_hash="h2", staged_content_hash="h2" + ) + assistants_table.update_item( + Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-1"}, + UpdateExpression="REMOVE sourceEtag, contentHash", + ) + + rolled_back = records.rollback_document_sync_fields( + assistant_id, + "doc-1", + written={"sourceEtag": "42", "contentHash": "h2", "stagedContentHash": "h2"}, + previous={"sourceEtag": "41", "contentHash": "h1"}, + ) + + assert rolled_back is False + item = self._doc(assistants_table, assistant_id) + assert "sourceEtag" not in item and "contentHash" not in item + + class TestFailureModes: async def test_not_found_strikes_counter(self, assistants_table, staged, token_ok, provider_ok, monkeypatch): adapter = FakeDriveAdapter(metadata_error=FileSourceNotFoundError("gone or unshared")) @@ -465,6 +582,8 @@ async def test_changed_page_stashes_and_records_changed(self, assistants_table, assert item["previousChunkCount"] == 4 assert item["sourceEtag"] == '"e2"' assert item["contentHash"] == "hash2" + # Marks the version a managed KB's consumer still has to re-ingest. + assert item["stagedContentHash"] == "hash2" # crawler invoked in refresh mode without TTL finalization assert fake_crawl["captured"]["finalize_with_ttl"] is False assert fake_crawl["captured"]["settings"].max_pages == 10 @@ -566,3 +685,141 @@ async def test_missing_root_doc_is_recreated(self, assistants_table, fake_crawl) item = assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": f"DOC#{root_doc_id}"})["Item"] assert item["sourceFileId"] == self.ROOT assert item["sourceConnectorId"] == "web" + + async def test_failed_stage_rolls_back_page_gates(self, assistants_table, fake_crawl): + assistant_id, job, policy = await self._setup_crawl(assistants_table) + assistants_table.update_item( + Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-web-0"}, + UpdateExpression="SET sourceEtag = :e, contentHash = :h", + ExpressionAttributeValues={":e": '"e1"', ":h": "hash1"}, + ) + fake_crawl["seen"] = [self.ROOT] + fake_crawl["emit"] = [ + (self.ROOT, "changed", '"e2"', "hash2"), + (self.ROOT, "stage_failed", '"e2"', "hash2"), + ] + + result = await worker.run_sync(self._payload(assistant_id, policy, job)) + + # Nothing reached the knowledge base, so the run changed nothing. + assert result["result"] == "unchanged" + item = assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-web-0"})["Item"] + assert item["sourceEtag"] == '"e1"' + assert item["contentHash"] == "hash1" + assert "stagedContentHash" not in item + + # The next re-crawl still sees the old gates, so it will re-stage. + fake_crawl["emit"] = [] + await worker.run_sync(self._payload(assistant_id, policy, job)) + refresh_doc = fake_crawl["captured"]["refresh"].docs[self.ROOT] + assert (refresh_doc.source_etag, refresh_doc.content_hash) == ('"e1"', "hash1") + + +class TestWebCrawlStageFailureEndToEnd: + """Worker + the REAL crawler (httpx on a MockTransport, S3 put stubbed): + a page whose stage fails is staged again by the next re-crawl, for both + a changed page and a page new to the crawl.""" + + ROOT = "https://example.com/" + NEW = "https://example.com/new" + + @pytest.fixture() + def site(self, monkeypatch): + import httpx + + from apis.app_api.web_sources import crawler + + pages = { + self.ROOT: '

fresh root words

n', + self.NEW: "

a brand new page

", + } + + def handler(request): + url = str(request.url) + if url in pages: + return httpx.Response(200, text=pages[url], headers={"content-type": "text/html"}) + return httpx.Response(404, text="") + + monkeypatch.setattr( + crawler, "_default_client", lambda: httpx.AsyncClient(transport=httpx.MockTransport(handler)) + ) + puts = {"fail": True, "ok": []} + + async def put_markdown(*, assistant_id, document_id, markdown, filename): + if puts["fail"]: + raise RuntimeError("S3 PutObject failed") + puts["ok"].append(document_id) + return f"assistants/{assistant_id}/documents/{document_id}/{filename}" + + monkeypatch.setattr(crawler, "_put_markdown", put_markdown) + return puts + + async def test_failed_stages_are_restaged_next_run(self, assistants_table, site): + from apis.app_api.web_sources.crawl_repository import create_crawl_job, finalize_crawl + from apis.app_api.web_sources.models import CrawlSettings + + assistant = await create_assistant( + owner_id=USER_ID, owner_name="U", name="A", description="d", + instructions="i", vector_index_id="assistants-index", + ) + assistant_id = assistant.assistant_id + job = await create_crawl_job( + assistant_id=assistant_id, root_url=self.ROOT, + settings=CrawlSettings(max_depth=1, max_pages=10, min_delay_seconds=0, max_delay_seconds=0), + started_by_user_id=USER_ID, + ) + await finalize_crawl(assistant_id=assistant_id, crawl_id=job.crawl_id, status="complete") + await create_document( + assistant_id=assistant_id, filename="root.md", content_type="text/markdown", + size_bytes=1, s3_key=f"assistants/{assistant_id}/documents/doc-root/root.md", + document_id="doc-root", + provenance=DocumentProvenance( + source_connector_id="web", source_adapter_key="http", + source_file_id=self.ROOT, imported_by_user_id=USER_ID, + ), + ) + assistants_table.update_item( + Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-root"}, + UpdateExpression="SET contentHash = :h", + ExpressionAttributeValues={":h": "old-root-hash"}, + ) + policy = await create_sync_policy( + assistant_id=assistant_id, source_type="web_crawl", source_ref=job.crawl_id, + interval="daily", created_by_user_id=USER_ID, + ) + payload = { + "policyId": policy.policy_id, "assistantId": assistant_id, + "sourceType": "web_crawl", "sourceRef": job.crawl_id, + } + + first = await worker.run_sync(payload) + + assert first["result"] == "unchanged" + assert site["ok"] == [] + root = assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-root"})["Item"] + assert root["contentHash"] == "old-root-hash" + assert "stagedContentHash" not in root + new_docs = [ + item for item in _document_items(assistants_table, assistant_id) + if item.get("sourceFileId") == self.NEW + ] + assert len(new_docs) == 1 + assert "contentHash" not in new_docs[0] + assert new_docs[0]["status"] == "failed" + + site["fail"] = False + second = await worker.run_sync(payload) + + assert second["result"] == "changed" + assert sorted(site["ok"]) == sorted(["doc-root", new_docs[0]["documentId"]]) + root = assistants_table.get_item(Key={"PK": f"AST#{assistant_id}", "SK": "DOC#doc-root"})["Item"] + assert root["contentHash"] != "old-root-hash" + assert root["stagedContentHash"] == root["contentHash"] + + +def _document_items(assistants_table, assistant_id): + from boto3.dynamodb.conditions import Key + + return assistants_table.query( + KeyConditionExpression=Key("PK").eq(f"AST#{assistant_id}") & Key("SK").begins_with("DOC#") + )["Items"] diff --git a/backend/tests/property/test_pbt_kb_query_clamp.py b/backend/tests/property/test_pbt_kb_query_clamp.py index a61ca0a4d..94503ae59 100644 --- a/backend/tests/property/test_pbt_kb_query_clamp.py +++ b/backend/tests/property/test_pbt_kb_query_clamp.py @@ -211,7 +211,7 @@ async def test_the_facade_actually_clamps_before_dispatch(): seen = {} class _RecordingBackend: - async def search(self, kb_ref, query, top_k=5): + async def search(self, kb_ref, query, top_k=5, *, record=None): seen["query"] = query return [] diff --git a/backend/tests/property/test_pbt_kb_score_direction.py b/backend/tests/property/test_pbt_kb_score_direction.py index 62758ab4e..67491d069 100644 --- a/backend/tests/property/test_pbt_kb_score_direction.py +++ b/backend/tests/property/test_pbt_kb_score_direction.py @@ -76,7 +76,9 @@ def __init__(self, results: List[Dict[str, Any]]): # which is the order Bedrock's Retrieve returns. self._results = results - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: + async def search( + self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K, *, record=None + ) -> List[Chunk]: return [ Chunk( text=result["text"], diff --git a/backend/tests/property/test_pbt_kb_status_fail_closed.py b/backend/tests/property/test_pbt_kb_status_fail_closed.py index eff74cfd7..a7091b999 100644 --- a/backend/tests/property/test_pbt_kb_status_fail_closed.py +++ b/backend/tests/property/test_pbt_kb_status_fail_closed.py @@ -37,6 +37,17 @@ # because CI is cold and local re-runs are warm. from apis.shared.assistants.rag_service import _filter_vectors_by_document_status + +def _no_cached_tables(): + """Isolate one example from the process-cached DynamoDB handles. + + The status lookup reads its table through ``apis.shared.aws_clients``. Hypothesis + runs every example inside one test call, so the per-test reset in the root + conftest does not separate them: without this, the first example's mocked + resource would answer every later example's lookup. + """ + return patch.dict("apis.shared.aws_clients._resources", clear=True) + ASSISTANT_ID = "ast-failclosed" # --------------------------------------------------------------------------- @@ -85,7 +96,7 @@ def test_a_table_level_failure_never_leaks_a_chunk(vectors, failure): "apis.shared.assistants.rag_service.emit_count" ), patch("boto3.resource") as resource, patch.dict( "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "t", "AWS_REGION": "us-west-2"} - ): + ), _no_cached_tables(): dynamo = MagicMock() dynamo.Table.side_effect = failure resource.return_value = dynamo @@ -99,7 +110,7 @@ def test_a_missing_table_name_never_leaks_a_chunk(vectors): """The other former fail-open path: no table configured → zero chunks.""" with patch("apis.shared.kb_backend.metrics.emit_count"), patch( "apis.shared.assistants.rag_service.emit_count" - ), patch("boto3.resource") as resource, patch.dict("os.environ", {}, clear=True): + ), patch("boto3.resource") as resource, patch.dict("os.environ", {}, clear=True), _no_cached_tables(): assert _filter_vectors_by_document_status(vectors, ASSISTANT_ID) == [] # Never contacted, so this is a guard rather than a failed call. resource.assert_not_called() @@ -117,7 +128,7 @@ def test_the_degradation_is_always_reported(vectors, failure): "boto3.resource" ) as resource, patch.dict( "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "t", "AWS_REGION": "us-west-2"} - ): + ), _no_cached_tables(): dynamo = MagicMock() dynamo.Table.side_effect = failure resource.return_value = dynamo @@ -143,7 +154,7 @@ def test_a_per_document_failure_still_only_drops_that_document(vectors): "boto3.resource" ) as resource, patch.dict( "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "t", "AWS_REGION": "us-west-2"} - ): + ), _no_cached_tables(): table = MagicMock() table.get_item.side_effect = Exception("per-item failure") dynamo = MagicMock() @@ -162,7 +173,7 @@ def test_complete_documents_are_still_returned(vectors): "boto3.resource" ) as resource, patch.dict( "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "t", "AWS_REGION": "us-west-2"} - ): + ), _no_cached_tables(): table = MagicMock() table.get_item.return_value = {"Item": {"status": "complete"}} dynamo = MagicMock() @@ -183,7 +194,7 @@ def test_a_non_complete_status_is_excluded(status): "boto3.resource" ) as resource, patch.dict( "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "t", "AWS_REGION": "us-west-2"} - ): + ), _no_cached_tables(): table = MagicMock() table.get_item.return_value = {"Item": {"status": status}} dynamo = MagicMock() diff --git a/backend/tests/property/test_pbt_search_filtering.py b/backend/tests/property/test_pbt_search_filtering.py index 06a5640b3..c932d78c7 100644 --- a/backend/tests/property/test_pbt_search_filtering.py +++ b/backend/tests/property/test_pbt_search_filtering.py @@ -92,6 +92,8 @@ def mock_get_item(**kwargs): with ( patch.dict("os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "test-table"}), patch("boto3.resource", return_value=mock_dynamodb), + # One test call runs every example, so isolate each from the cached handle. + patch.dict("apis.shared.aws_clients._resources", clear=True), ): from apis.shared.assistants.rag_service import ( _filter_vectors_by_document_status, diff --git a/backend/tests/routes/test_inference_kb_search_ahead.py b/backend/tests/routes/test_inference_kb_search_ahead.py new file mode 100644 index 000000000..83b9e283c --- /dev/null +++ b/backend/tests/routes/test_inference_kb_search_ahead.py @@ -0,0 +1,204 @@ +"""The knowledge-base search overlaps the agent build (turn-path spec §5 P3b). + +An agent turn used to await its knowledge-base search in the route, before the +stream opened, and only then build the agent. The build reads neither the +retrieved chunks nor the augmented message — only the stream does, for the +citation frames and `final_message` — so the search now starts where it always +did and is awaited after the build. What must hold, most expensive first: + +1. **The model sees the same bytes.** `final_message` is the augmented message + the inline search produced, and the citations are the same — they land in + the persisted user message, which is the cacheable prefix on every later + turn. +2. **The search is in flight while the agent is built.** The fake search below + cannot finish until the build has started; awaited inline, it never would. +3. **Frame order is unchanged**: `prepared` → `citation`* → `message_start`. +4. A failed search still fails open, and `KB_SEARCH_AHEAD_ENABLED=false` + restores the inline await. + +Driven through the real `/invocations` handler; only the services behind it +are stubbed. +""" + +import asyncio +import json +from typing import Any, List +from unittest.mock import AsyncMock, patch + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +import apis.inference_api.chat.routes as routes_module +from apis.inference_api.chat.routes import router +from apis.shared.assistants.models import Assistant +from apis.shared.assistants.rag_service import augment_prompt_with_context +from apis.shared.auth.dependencies import get_current_user_trusted + +CHUNKS = [ + {"text": "Register in myBoiseState under Student Center.", "distance": 0.1, + "metadata": {"document_id": "doc-1", "source": "register.pdf"}, "key": "k1"}, + {"text": "Add classes to your shopping cart first.", "distance": 0.2, + "metadata": {"document_id": "doc-2", "source": "cart.pdf"}, "key": "k2"}, +] +CAP = 2000 +MESSAGE = "How do I register for classes?" + + +def _assistant() -> Assistant: + return Assistant( + assistantId="ast-kb", + ownerId="user-1", + ownerName="Owner", + name="Registrar", + description="d", + instructions="Answer from the knowledge base.", + vectorIndexId="idx", + visibility="PRIVATE", + createdAt="t", + updatedAt="t", + status="COMPLETE", + ) + + +def _frames(body: str, kind: str) -> List[dict]: + prefix = f"event: {kind}\ndata: " + return [json.loads(f[len(prefix):]) for f in body.split("\n\n") if f.startswith(prefix)] + + +@pytest.fixture +def client(make_user): + app = FastAPI() + app.include_router(router) + user = make_user(raw_token="fake-jwt-token", user_id="user-1") + app.dependency_overrides[get_current_user_trusted] = lambda: user + return TestClient(app) + + +class _Turn: + """One agent turn: a build that marks when it starts, a search that can be + made to wait for it, and an agent that records what it was handed.""" + + def __init__(self, *, search_waits_for_build: bool, search_fails: bool = False) -> None: + self.build_started = asyncio.Event() + self.search_waits_for_build = search_waits_for_build + self.search_fails = search_fails + self.search_saw_build = None + self.stream_kwargs: dict = {} + self.final_message: Any = None + + def get_agent(self, *args, **kwargs): + self.build_started.set() + turn = self + + class _Agent: + # A plain object, not a MagicMock: the route reads optional + # attributes off the agent (pending MCP App context, interrupts), + # and a mock answers every one of them with something truthy. + async def stream_async(self, message, **kw): + turn.final_message = message + turn.stream_kwargs = kw + yield 'event: message_start\ndata: {"role": "assistant"}\n\n' + yield "event: done\ndata: {}\n\n" + + return _Agent() + + async def search(self, **kwargs): + if self.search_waits_for_build: + try: + await asyncio.wait_for(self.build_started.wait(), timeout=2) + self.search_saw_build = True + except asyncio.TimeoutError: + self.search_saw_build = False + raise + if self.search_fails: + raise RuntimeError("s3vectors throttled") + return list(CHUNKS), CAP + + def post(self, client: TestClient, session_id: str = "preview-kb-1"): + assistant = _assistant() + with patch.object(routes_module, "get_agent", side_effect=self.get_agent), patch.object( + routes_module, "is_quota_enforcement_enabled", return_value=False + ), patch.object(routes_module, "agents_enabled", return_value=False), patch.object( + routes_module, "_session_has_messages", AsyncMock(return_value=False) + ), patch( + "apis.shared.assistants.service.get_assistant_with_access_check", + AsyncMock(return_value=(assistant, "owner")), + ), patch( + "apis.shared.assistants.version_resolution.resolve_invocation_agent", + AsyncMock(return_value=(assistant, None)), + ), patch( + "apis.shared.assistants.service.bump_last_used_at", AsyncMock(return_value=False) + ), patch( + "apis.shared.sessions.metadata.get_session_metadata", AsyncMock(return_value=None) + ), patch( + "apis.shared.sessions.metadata.store_session_metadata", AsyncMock() + ), patch( + "apis.shared.assistants.rag_service.search_assistant_knowledgebase_with_cap", + side_effect=self.search, + ): + return client.post( + "/invocations", + json={"session_id": session_id, "message": MESSAGE, "rag_assistant_id": "ast-kb"}, + ) + + +EXPECTED = augment_prompt_with_context(user_message=MESSAGE, context_chunks=CHUNKS, max_context_length=CAP) + + +class TestSearchAheadOfTheBuild: + def test_the_search_is_in_flight_while_the_agent_is_built(self, client, monkeypatch): + monkeypatch.delenv("KB_SEARCH_AHEAD_ENABLED", raising=False) + turn = _Turn(search_waits_for_build=True) + + resp = turn.post(client) + + assert resp.status_code == 200 + assert turn.search_saw_build is True + assert turn.final_message == EXPECTED + + def test_the_model_gets_the_same_message_and_citations_either_way(self, client, monkeypatch): + monkeypatch.delenv("KB_SEARCH_AHEAD_ENABLED", raising=False) + ahead = _Turn(search_waits_for_build=False) + ahead_body = ahead.post(client).text + + monkeypatch.setenv("KB_SEARCH_AHEAD_ENABLED", "false") + inline = _Turn(search_waits_for_build=False) + inline_body = inline.post(client).text + + assert ahead.final_message == inline.final_message == EXPECTED + assert ahead.stream_kwargs["citations"] == inline.stream_kwargs["citations"] + assert len(ahead.stream_kwargs["citations"]) == len(CHUNKS) + assert ahead.stream_kwargs["original_message"] == MESSAGE + assert _frames(ahead_body, "citation") == _frames(inline_body, "citation") + + def test_citations_go_out_after_prepared_and_before_the_answer(self, client, monkeypatch): + monkeypatch.delenv("KB_SEARCH_AHEAD_ENABLED", raising=False) + body = _Turn(search_waits_for_build=False).post(client).text + + prepared = body.index('"phase": "prepared"') + first_citation = body.index("event: citation") + message_start = body.index("event: message_start") + assert prepared < first_citation < message_start + assert len(_frames(body, "citation")) == len(CHUNKS) + + def test_a_failed_search_still_fails_open(self, client, monkeypatch): + monkeypatch.delenv("KB_SEARCH_AHEAD_ENABLED", raising=False) + turn = _Turn(search_waits_for_build=False, search_fails=True) + + body = turn.post(client).text + + assert turn.final_message == MESSAGE + assert "event: citation" not in body + assert turn.stream_kwargs["citations"] is None + + def test_kill_switch_awaits_the_search_before_the_build(self, client, monkeypatch): + """Inline, a search that waits for the build can never see it — it + times out and the turn fails open, which is what proves the order.""" + monkeypatch.setenv("KB_SEARCH_AHEAD_ENABLED", "false") + turn = _Turn(search_waits_for_build=True) + + turn.post(client) + + assert turn.search_saw_build is False + assert turn.final_message == MESSAGE diff --git a/backend/tests/routes/test_models.py b/backend/tests/routes/test_models.py index b7b886a60..c8db12ff4 100644 --- a/backend/tests/routes/test_models.py +++ b/backend/tests/routes/test_models.py @@ -138,3 +138,39 @@ def test_returns_401_unauthenticated(self, app, unauthenticated_client): resp = client.get("/models") assert resp.status_code == 401 + + +class TestSpeechRowsAreNeverOffered: + """A speech-to-speech row prices voice sessions; it answers no chat API.""" + + def test_speech_model_is_dropped_before_access_filtering(self, app, make_user): + user = make_user() + mock_auth_user(app, user) + + sonic = SAMPLE_MODEL.model_copy( + update={ + "id": "model-sonic", + "model_id": "amazon.nova-2-sonic-v1:0", + "model_name": "Nova 2 Sonic", + "input_modalities": ["SPEECH", "TEXT"], + "output_modalities": ["SPEECH", "TEXT"], + } + ) + mock_service = MagicMock(spec=ModelAccessService) + mock_service.filter_accessible_models = AsyncMock(side_effect=lambda _u, models: models) + + # `Depends(get_model_access_service)` captured the function at import, + # so the dependency is overridden on the app rather than patched. + app.dependency_overrides[get_model_access_service] = lambda: mock_service + with patch( + f"{ROUTES_MODULE}.list_all_managed_models", + new_callable=AsyncMock, + return_value=[SAMPLE_MODEL, sonic], + ): + resp = TestClient(app).get("/models") + + assert resp.status_code == 200 + assert [m["modelId"] for m in resp.json()["models"]] == ["anthropic.claude-3-haiku"] + # The access service never even sees the speech row. + seen = mock_service.filter_accessible_models.await_args.args[1] + assert [m.model_id for m in seen] == ["anthropic.claude-3-haiku"] diff --git a/backend/tests/shared/test_agent_build_shared_session_flag.py b/backend/tests/shared/test_agent_build_shared_session_flag.py new file mode 100644 index 000000000..3dff4dfa2 --- /dev/null +++ b/backend/tests/shared/test_agent_build_shared_session_flag.py @@ -0,0 +1,29 @@ +"""`agent_build_shared_session_enabled`: default ON, `false` is the kill switch.""" + +import pytest + +from apis.shared.feature_flags import agent_build_shared_session_enabled + + +@pytest.mark.parametrize("value", [None, "", "true", "on", "ab", "shared_clients", "nonsense"]) +def test_everything_but_false_is_on(monkeypatch, value): + if value is None: + monkeypatch.delenv("AGENT_BUILD_SHARED_SESSION_ENABLED", raising=False) + else: + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", value) + + assert agent_build_shared_session_enabled() + + +@pytest.mark.parametrize("value", ["false", "FALSE", " False "]) +def test_false_is_off(monkeypatch, value): + monkeypatch.setenv("AGENT_BUILD_SHARED_SESSION_ENABLED", value) + assert not agent_build_shared_session_enabled() + + +def test_the_retired_experiment_variable_is_ignored(monkeypatch): + """`AGENT_BUILD_EXPERIMENT` drove the A/B that decided this; a Runtime + still carrying it must not read as anything.""" + monkeypatch.setenv("AGENT_BUILD_EXPERIMENT", "ab") + monkeypatch.delenv("AGENT_BUILD_SHARED_SESSION_ENABLED", raising=False) + assert agent_build_shared_session_enabled() diff --git a/backend/tests/shared/test_aws_clients.py b/backend/tests/shared/test_aws_clients.py index eb5fb3a64..91dd321e9 100644 --- a/backend/tests/shared/test_aws_clients.py +++ b/backend/tests/shared/test_aws_clients.py @@ -69,6 +69,54 @@ def test_the_aws_fixture_leaves_no_client_behind(self): assert aws_clients._clients == {} +class TestSharedSession: + """One process-wide `boto3.Session` for the SDKs that build their own + clients (docs/specs/turn-path-ttft.md §5 P2).""" + + def test_the_same_session_every_time(self): + assert aws_clients.shared_boto_session() is aws_clients.shared_boto_session() + + def test_reset_drops_the_shared_session_too(self): + """A session built under one moto backend is as stale as a client + built under it.""" + before = aws_clients.shared_boto_session() + + aws_clients.reset_cached_clients() + + assert aws_clients._shared_session is None + assert aws_clients.shared_boto_session() is not before + + def test_one_client_per_service_region_and_config(self): + session = aws_clients.ClientReusingSession() + + first = session.client("sts", region_name="us-west-2") + again = session.client(service_name="sts", region_name="us-west-2") + other_region = session.client("sts", region_name="us-east-1") + + assert first is again + assert first is not other_region + assert first.meta.config.max_pool_connections == aws_clients.SHARED_SESSION_MAX_POOL_CONNECTIONS + + def test_a_none_keyword_is_not_an_override(self): + """Strands' BedrockModel passes `endpoint_url=None`; that must still + land on the shared client, or the arm silently builds a fresh one.""" + session = aws_clients.ClientReusingSession() + + plain = session.client("sts", region_name="us-west-2") + with_none = session.client("sts", region_name="us-west-2", endpoint_url=None) + + assert with_none is plain + + def test_a_real_override_is_never_shared(self): + session = aws_clients.ClientReusingSession() + + plain = session.client("sts", region_name="us-west-2") + custom = session.client("sts", region_name="us-west-2", endpoint_url="https://sts.example") + + assert custom is not plain + assert session.client("sts", region_name="us-west-2") is plain + + class TestMetadataUsesIt: @pytest.mark.asyncio async def test_session_reads_go_through_one_cached_resource( diff --git a/backend/tests/shared/test_bedrock_embeddings_client.py b/backend/tests/shared/test_bedrock_embeddings_client.py new file mode 100644 index 000000000..03afbc5fb --- /dev/null +++ b/backend/tests/shared/test_bedrock_embeddings_client.py @@ -0,0 +1,36 @@ +"""generate_embeddings shares one bedrock-runtime client across calls.""" + +import io +import json +from unittest.mock import MagicMock, patch + +import pytest + +from apis.shared.embeddings import bedrock_embeddings as be + + +@pytest.fixture(autouse=True) +def _fresh_bedrock_client(monkeypatch): + """The Bedrock client is cached per process; each test builds its own.""" + monkeypatch.setattr(be, "_bedrock_runtime_client", None) + + +def _client() -> MagicMock: + client = MagicMock() + client.invoke_model.side_effect = lambda **kwargs: { + "body": io.BytesIO(json.dumps({"embedding": [0.1, 0.2]}).encode()) + } + return client + + +@pytest.mark.asyncio +async def test_calls_reuse_one_bedrock_client(): + """A client per call paid botocore's model load and a fresh TLS handshake on + every knowledge base search, ahead of the model's first token.""" + client = _client() + with patch.object(be.boto3, "client", return_value=client) as factory: + for _ in range(3): + assert await be.generate_embeddings(["a", "b"]) == [[0.1, 0.2], [0.1, 0.2]] + + factory.assert_called_once_with("bedrock-runtime", region_name=be.AWS_REGION) + assert client.invoke_model.call_count == 6 diff --git a/backend/tests/shared/test_build_stages.py b/backend/tests/shared/test_build_stages.py index f9fe72d6b..be74a481d 100644 --- a/backend/tests/shared/test_build_stages.py +++ b/backend/tests/shared/test_build_stages.py @@ -84,6 +84,42 @@ class _Hostile: build_stages.reset_stage_recorder(_Hostile()) # must not raise +class TestRecordDetail: + def test_without_a_recorder_it_is_a_no_op(self): + build_stages.record_detail("mcpServers", []) # must not raise + + def test_routes_to_the_detail_recorder(self): + details = {} + token = build_stages.set_stage_recorder(None, details.__setitem__) + try: + build_stages.record_detail("mcpServers", [{"id": "canvas"}]) + finally: + build_stages.reset_stage_recorder(token) + + assert details == {"mcpServers": [{"id": "canvas"}]} + + def test_a_stage_only_recorder_drops_details(self): + seen = [] + token = build_stages.set_stage_recorder(seen.append) + try: + build_stages.record_detail("mcpServers", []) + build_stages.mark_stage("tools.mcp") + finally: + build_stages.reset_stage_recorder(token) + + assert seen == ["tools.mcp"] + + def test_a_failing_detail_recorder_never_raises(self): + def _boom(_key, _value): + raise RuntimeError("boom") + + token = build_stages.set_stage_recorder(None, _boom) + try: + build_stages.record_detail("mcpServers", []) # must not raise + finally: + build_stages.reset_stage_recorder(token) + + class TestPreludeIntegration: def test_sub_stages_group_under_agent_build(self): """`groups.agent_build` has to reproduce the pre-split number, or the diff --git a/backend/tests/shared/test_classic_kb_search_off_loop.py b/backend/tests/shared/test_classic_kb_search_off_loop.py new file mode 100644 index 000000000..a30e047b4 --- /dev/null +++ b/backend/tests/shared/test_classic_kb_search_off_loop.py @@ -0,0 +1,65 @@ +"""The classic (S3 Vectors) knowledge-base search runs off the event loop. + +An agent turn now starts its knowledge-base search before the agent build and +awaits it after (turn-path spec §5 P3b), and a cold build holds the event loop +while it runs. If either round trip of the classic search — the query +embedding, then `QueryVectors` — were sent from the loop, it would wait for the +build instead of overlapping it. So both go out back to back on one worker +thread, on the process-cached `s3vectors` client. +""" + +import io +import json +import threading + +import pytest + +from apis.shared.embeddings import bedrock_embeddings + + +class _Runtime: + def __init__(self, threads): + self.threads = threads + + def invoke_model(self, **kwargs): + self.threads.append(("embed", threading.current_thread())) + self.body = json.loads(kwargs["body"]) + return {"body": io.BytesIO(json.dumps({"embedding": [0.1, 0.2]}).encode())} + + +class _Vectors: + def __init__(self, threads): + self.threads = threads + + def query_vectors(self, **kwargs): + self.threads.append(("query", threading.current_thread())) + self.kwargs = kwargs + return {"vectors": []} + + +@pytest.mark.asyncio +async def test_both_round_trips_run_on_one_worker_thread(monkeypatch): + threads: list = [] + runtime, vectors = _Runtime(threads), _Vectors(threads) + clients_asked: list = [] + monkeypatch.setattr(bedrock_embeddings, "_get_bedrock_runtime_client", lambda: runtime) + monkeypatch.setattr(bedrock_embeddings, "_VECTOR_STORE_BUCKET_NAME", "bucket") + monkeypatch.setattr(bedrock_embeddings, "_VECTOR_STORE_INDEX_NAME", "index") + + def _get_client(): + clients_asked.append("s3vectors") + return vectors + + monkeypatch.setattr(bedrock_embeddings, "_get_s3vectors_client", _get_client) + + response = await bedrock_embeddings.search_assistant_knowledgebase("ast-1", "how do I register?") + + assert response == {"vectors": []} + assert [step for step, _ in threads] == ["embed", "query"] + assert threads[0][1] is threads[1][1] + assert threads[0][1] is not threading.current_thread() + assert clients_asked == ["s3vectors"] + assert runtime.body == {"inputText": "how do I register?"} + assert vectors.kwargs["queryVector"] == {"float32": [0.1, 0.2]} + assert vectors.kwargs["filter"] == {"assistant_id": "ast-1"} + assert vectors.kwargs["topK"] == 5 diff --git a/backend/tests/shared/test_context_window_resolution.py b/backend/tests/shared/test_context_window_resolution.py index da478315b..c2d5731ab 100644 --- a/backend/tests/shared/test_context_window_resolution.py +++ b/backend/tests/shared/test_context_window_resolution.py @@ -34,18 +34,29 @@ def test_haiku_45_really_is_200k(self): @pytest.mark.parametrize( "model_id", [ - "us.openai.gpt-6-astra", - "us.openai.gpt-5.6-sol", "us.moonshotai.kimi-k3", "us.deepseek.v3-2", ], ) def test_non_anthropic_ids_are_absent(self, model_id): - # The fallback is a partial safety net. If this starts returning values, - # the 272K pricing cap on the OpenAI rows needs re-checking (it would - # begin tripping the disagreement warning). + # The fallback is a partial safety net. If one of these starts + # returning a value, check it against the catalog row the way the + # hosted OpenAI rows below were checked. assert cw.sdk_context_window(model_id) is None + @pytest.mark.parametrize("model_id", ["us.openai.gpt-6-astra", "us.openai.gpt-5.6-sol"]) + def test_hosted_openai_ids_now_resolve_to_their_full_window(self, model_id): + # strands-agents 1.56+ strips nested prefixes (`us.` then `openai.`) + # and its table carries these at 1,050,000. The curated rows pin + # maxInputTokens at 272,000 on purpose (the pricing tier), so the + # catalog value still wins; the pair now logs one disagreement warning + # per process, which is expected rather than a stale row. + assert cw.sdk_context_window(model_id) == 1_050_000 + + def test_the_curated_272k_cap_still_wins_over_the_sdk_window(self): + window, source = cw.resolve_context_window("us.openai.gpt-6-astra", 272_000) + assert (window, source) == (272_000, "disagreement") + def test_missing_model_id_is_not_an_error(self): assert cw.sdk_context_window(None) is None assert cw.sdk_context_window("") is None @@ -77,7 +88,7 @@ def test_sdk_fills_an_absent_catalog_value(self): assert (window, source) == (1_000_000, "sdk") def test_both_absent_reports_nothing_rather_than_guessing(self): - assert cw.resolve_context_window("us.openai.gpt-6-astra", None) == (None, None) + assert cw.resolve_context_window("us.moonshotai.kimi-k3", None) == (None, None) @pytest.mark.parametrize("bad", [0, -1, "", "not-a-number", None]) def test_unusable_catalog_values_fall_through(self, bad): diff --git a/backend/tests/shared/test_document_digest.py b/backend/tests/shared/test_document_digest.py index b0f9a0b87..251637071 100644 --- a/backend/tests/shared/test_document_digest.py +++ b/backend/tests/shared/test_document_digest.py @@ -13,6 +13,7 @@ import pytest +from apis.shared import aws_clients from apis.shared.files import document_digest as dd from tests.shared.test_document_read import build_docx, build_pdf @@ -123,9 +124,19 @@ def _bedrock(monkeypatch, text="A crisp abstract.", stop="end_turn", fail=False) boto3 = MagicMock() boto3.client.return_value = client monkeypatch.setitem(__import__("sys").modules, "boto3", boto3) + # The client is cached per process; drop the previous fake so this one is built. + aws_clients.reset_cached_clients() return client +@pytest.fixture(autouse=True) +def _fresh_bedrock_client(): + """The Bedrock client is cached per process; each test builds its own.""" + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + class TestAbstract: @pytest.mark.asyncio async def test_calls_the_cheap_model_with_outline_and_sample(self, monkeypatch): @@ -146,6 +157,25 @@ async def test_truncated_or_failed_generations_are_none(self, monkeypatch): assert await dd.generate_abstract(dd.DocumentDigest(), "text") is None assert await dd.generate_abstract(dd.DocumentDigest(), " ") is None + @pytest.mark.asyncio + @pytest.mark.parametrize("stop", ["guardrail_intervened", "content_filtered", "something_new", None]) + async def test_a_refused_generation_is_none(self, monkeypatch, stop): + """Only a finished generation is an abstract; a refusal's text is not.""" + _bedrock(monkeypatch, text="Sorry, the model cannot answer this.", stop=stop) + assert await dd.generate_abstract(dd.DocumentDigest(), "text") is None + + @pytest.mark.asyncio + async def test_abstracts_reuse_one_bedrock_client(self, monkeypatch): + """A client per upload paid botocore's model load and a fresh TLS handshake.""" + client = _bedrock(monkeypatch) + factory = __import__("sys").modules["boto3"].client + + for _ in range(3): + assert await dd.generate_abstract(dd.DocumentDigest(), "text") == "A crisp abstract." + + assert factory.call_count == 1 + assert client.converse.call_count == 3 + class TestBuild: @pytest.mark.asyncio diff --git a/backend/tests/shared/test_history_count_prefetch_flag.py b/backend/tests/shared/test_history_count_prefetch_flag.py new file mode 100644 index 000000000..a0e9930a6 --- /dev/null +++ b/backend/tests/shared/test_history_count_prefetch_flag.py @@ -0,0 +1,21 @@ +"""`history_count_prefetch_enabled`: default ON, `false` is the kill switch.""" + +import pytest + +from apis.shared.feature_flags import history_count_prefetch_enabled + + +@pytest.mark.parametrize("value", [None, "", "true", "on", "nonsense"]) +def test_everything_but_false_is_on(monkeypatch, value): + if value is None: + monkeypatch.delenv("HISTORY_COUNT_PREFETCH_ENABLED", raising=False) + else: + monkeypatch.setenv("HISTORY_COUNT_PREFETCH_ENABLED", value) + + assert history_count_prefetch_enabled() + + +@pytest.mark.parametrize("value", ["false", "FALSE", " False "]) +def test_false_is_off(monkeypatch, value): + monkeypatch.setenv("HISTORY_COUNT_PREFETCH_ENABLED", value) + assert not history_count_prefetch_enabled() diff --git a/backend/tests/shared/test_inline_persist.py b/backend/tests/shared/test_inline_persist.py new file mode 100644 index 000000000..f83395119 --- /dev/null +++ b/backend/tests/shared/test_inline_persist.py @@ -0,0 +1,208 @@ +"""``apis.shared.files.inline_persist`` — inline attachment bytes become session files. + +What these pin: the persisted row is byte-compatible with an upload (same key +layout, READY, quota counted, ``source="inline"``); every validation the SPA +upload path enforces is enforced here too; and no single bad file, S3 error or +metadata error can fail the batch — each failure is reported per file with a +user-facing reason, and an S3 object whose row could not be written is removed +rather than orphaned. +""" + +from __future__ import annotations + +import base64 +from dataclasses import dataclass +from typing import Any, Dict, List, Optional + +import pytest + +from apis.shared.files.inline_persist import ( + INLINE_SOURCE, + InlineAttachmentPersister, + PersistFailure, + s3_key_for, +) +from apis.shared.files.models import FileMetadata, FileStatus, UserFileQuota + +CSV = "text/csv" +PPTX = "application/vnd.openxmlformats-officedocument.presentationml.presentation" + + +@dataclass +class _File: + filename: str + content_type: str + bytes: str + + +def _b64(raw: bytes) -> str: + return base64.b64encode(raw).decode() + + +class _FakeS3: + def __init__(self, fail_keys: Optional[set] = None) -> None: + self.objects: Dict[str, Dict[str, Any]] = {} + self.deleted: List[str] = [] + self.fail_keys = fail_keys or set() + + def put_object(self, *, Bucket: str, Key: str, Body: bytes, ContentType: str) -> None: + if any(Key.endswith(k) for k in self.fail_keys): + raise RuntimeError("boom") + self.objects[Key] = {"bucket": Bucket, "body": Body, "content_type": ContentType} + + def delete_object(self, *, Bucket: str, Key: str) -> None: + self.deleted.append(Key) + self.objects.pop(Key, None) + + +class _FakeRepo: + def __init__(self, used: int = 0, fail_create_for: Optional[set] = None) -> None: + self.rows: List[FileMetadata] = [] + self.quota_increments: List[int] = [] + self.used = used + self.fail_create_for = fail_create_for or set() + + async def get_user_quota(self, user_id: str) -> UserFileQuota: + return UserFileQuota(user_id=user_id, total_bytes=self.used, file_count=len(self.rows)) + + async def create_file(self, meta: FileMetadata) -> FileMetadata: + if meta.filename in self.fail_create_for: + raise RuntimeError("ddb down") + self.rows.append(meta) + return meta + + async def increment_quota(self, user_id: str, size_bytes: int) -> UserFileQuota: + self.quota_increments.append(size_bytes) + self.used += size_bytes + return UserFileQuota(user_id=user_id, total_bytes=self.used, file_count=len(self.rows)) + + +def _persister(repo: _FakeRepo, s3: _FakeS3) -> InlineAttachmentPersister: + return InlineAttachmentPersister(repository=repo, s3_client=s3, bucket_name="bucket-under-test") + + +@pytest.mark.asyncio +async def test_persists_like_an_upload(): + repo, s3 = _FakeRepo(), _FakeS3() + raw = b"a,b\n1,2\n" + persisted, failures = await _persister(repo, s3).persist(user_id="u1", session_id="s1", files=[_File("data.csv", CSV, _b64(raw))]) + + assert failures == [] + assert len(persisted) == 1 + meta = persisted[0] + assert meta.s3_key == s3_key_for("u1", "s1", meta.upload_id, "data.csv") + assert meta.s3_key.startswith("user-files/u1/s1/") + assert meta.s3_bucket == "bucket-under-test" + assert meta.status == FileStatus.READY.value + assert meta.source == INLINE_SOURCE + assert meta.size_bytes == len(raw) + assert meta.mime_type == CSV + assert s3.objects[meta.s3_key]["body"] == raw + assert s3.objects[meta.s3_key]["content_type"] == CSV + assert repo.rows == [meta] + assert repo.quota_increments == [len(raw)] + + +@pytest.mark.asyncio +async def test_empty_input_touches_nothing(): + repo, s3 = _FakeRepo(), _FakeS3() + assert await _persister(repo, s3).persist(user_id="u", session_id="s", files=[]) == ([], []) + assert s3.objects == {} and repo.rows == [] + + +@pytest.mark.asyncio +async def test_rejects_disallowed_mime_type(): + repo, s3 = _FakeRepo(), _FakeS3() + persisted, failures = await _persister(repo, s3).persist( + user_id="u", session_id="s", files=[_File("x.exe", "application/x-msdownload", _b64(b"MZ"))] + ) + assert persisted == [] + assert [f.file.filename for f in failures] == ["x.exe"] + assert "not accepted" in failures[0].reason + assert s3.objects == {} + + +@pytest.mark.asyncio +async def test_rejects_invalid_base64_without_touching_s3(): + repo, s3 = _FakeRepo(), _FakeS3() + _, failures = await _persister(repo, s3).persist(user_id="u", session_id="s", files=[_File("data.csv", CSV, "not base64!!")]) + assert [f.reason for f in failures] == ["its contents were not valid base64"] + assert s3.objects == {} + + +@pytest.mark.asyncio +async def test_size_cap_is_per_class(monkeypatch): + monkeypatch.setenv("FILE_UPLOAD_MAX_SIZE_BYTES", str(1024 * 1024)) + monkeypatch.setenv("FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION", str(3 * 1024 * 1024)) + repo, s3 = _FakeRepo(), _FakeS3() + two_mb = _b64(b"x" * (2 * 1024 * 1024)) + persisted, failures = await _persister(repo, s3).persist( + user_id="u", + session_id="s", + files=[_File("big.csv", CSV, two_mb), _File("deck.pptx", PPTX, two_mb)], + ) + # 2 MB is over the 1 MB general cap but under the 3 MB presentation cap. + assert [m.filename for m in persisted] == ["deck.pptx"] + assert [f.file.filename for f in failures] == ["big.csv"] + assert "1 MB limit" in failures[0].reason + + +@pytest.mark.asyncio +async def test_user_quota_is_charged_cumulatively_across_the_batch(monkeypatch): + monkeypatch.setenv("FILE_UPLOAD_USER_QUOTA_BYTES", "100") + repo, s3 = _FakeRepo(used=40), _FakeS3() + files = [_File("a.csv", CSV, _b64(b"x" * 30)), _File("b.csv", CSV, _b64(b"y" * 30)), _File("c.csv", CSV, _b64(b"z" * 30))] + persisted, failures = await _persister(repo, s3).persist(user_id="u", session_id="s", files=files) + # 40 + 30 + 30 = 100 fits; the third would be 130. + assert [m.filename for m in persisted] == ["a.csv", "b.csv"] + assert [(f.file.filename, f.reason) for f in failures] == [("c.csv", "your file storage quota is full")] + + +@pytest.mark.asyncio +async def test_s3_failure_is_per_file_and_writes_no_row(): + repo, s3 = _FakeRepo(), _FakeS3(fail_keys={"bad.csv"}) + persisted, failures = await _persister(repo, s3).persist( + user_id="u", + session_id="s", + files=[_File("ok.csv", CSV, _b64(b"1")), _File("bad.csv", CSV, _b64(b"2")), _File("also.csv", CSV, _b64(b"3"))], + ) + assert [m.filename for m in persisted] == ["ok.csv", "also.csv"] + assert [f.file.filename for f in failures] == ["bad.csv"] + assert failures[0].reason == "it could not be stored" + assert [r.filename for r in repo.rows] == ["ok.csv", "also.csv"] + + +@pytest.mark.asyncio +async def test_metadata_failure_removes_the_orphan_object(): + repo, s3 = _FakeRepo(fail_create_for={"orphan.csv"}), _FakeS3() + persisted, failures = await _persister(repo, s3).persist(user_id="u", session_id="s", files=[_File("orphan.csv", CSV, _b64(b"1"))]) + assert persisted == [] + assert [f.file.filename for f in failures] == ["orphan.csv"] + assert s3.objects == {} + assert len(s3.deleted) == 1 and s3.deleted[0].endswith("/orphan.csv") + assert repo.quota_increments == [] + + +@pytest.mark.asyncio +async def test_results_keep_attachment_order_across_mixed_outcomes(): + repo, s3 = _FakeRepo(), _FakeS3() + files = [ + _File("1.csv", CSV, _b64(b"a")), + _File("2.csv", CSV, "###"), # invalid base64 + _File("3.csv", CSV, _b64(b"c")), + ] + persisted, failures = await _persister(repo, s3).persist(user_id="u", session_id="s", files=files) + assert [m.filename for m in persisted] == ["1.csv", "3.csv"] + assert [f.file.filename for f in failures] == ["2.csv"] + assert all(isinstance(f, PersistFailure) for f in failures) + + +@pytest.mark.asyncio +async def test_quota_lookup_failure_does_not_block_the_turn(): + class _BrokenQuotaRepo(_FakeRepo): + async def get_user_quota(self, user_id: str) -> UserFileQuota: # type: ignore[override] + raise RuntimeError("ddb") + + repo, s3 = _BrokenQuotaRepo(), _FakeS3() + persisted, failures = await _persister(repo, s3).persist(user_id="u", session_id="s", files=[_File("a.csv", CSV, _b64(b"1"))]) + assert [m.filename for m in persisted] == ["a.csv"] and failures == [] diff --git a/backend/tests/shared/test_kb_authorization.py b/backend/tests/shared/test_kb_authorization.py index 4a5dd4629..a31647ba5 100644 --- a/backend/tests/shared/test_kb_authorization.py +++ b/backend/tests/shared/test_kb_authorization.py @@ -76,7 +76,9 @@ def __init__(self, chunks: List[Chunk] = None): self._chunks = chunks or [] self.calls: List[Dict[str, Any]] = [] - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: + async def search( + self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K, *, record=None + ) -> List[Chunk]: self.calls.append({"kb_ref": kb_ref, "query": query, "top_k": top_k}) return list(self._chunks) @@ -114,7 +116,7 @@ async def _search(access, backend: RecordingBackend, top_k: int = 5): with patch.dict("os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": TABLE_NAME}, clear=False), patch( "apis.shared.assistants.rag_service.resolve_backend", return_value=backend ), patch( - "apis.shared.assistants.rag_service.boto3.resource", + "boto3.resource", return_value=_complete_document_table(), ), patch( "apis.shared.assistants.rag_service.emit_count" diff --git a/backend/tests/shared/test_kb_backend_parity.py b/backend/tests/shared/test_kb_backend_parity.py index 6d7f5c59a..ad387c416 100644 --- a/backend/tests/shared/test_kb_backend_parity.py +++ b/backend/tests/shared/test_kb_backend_parity.py @@ -31,6 +31,7 @@ MAX_CONTEXT_CHARS, augment_prompt_with_context, resolve_context_cap, + search_assistant_knowledgebase_with_cap, search_assistant_knowledgebase_with_formatting, ) from apis.shared.kb_backend.protocol import ( @@ -86,8 +87,10 @@ def __init__(self, chunks: List[Chunk]): self._chunks = chunks self.calls: List[Dict[str, Any]] = [] - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: - self.calls.append({"kb_ref": kb_ref, "query": query, "top_k": top_k}) + async def search( + self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K, *, record=None + ) -> List[Chunk]: + self.calls.append({"kb_ref": kb_ref, "query": query, "top_k": top_k, "record": record}) return list(self._chunks) async def ingest(self, kb_ref: str, source) -> None: # pragma: no cover - unused @@ -241,6 +244,64 @@ def test_context_cap_keys_on_the_same_kb_record_read_as_the_backend(): assert DEFAULT_TOP_K == 5, "the parity contract pins top_k at 5" +def _kb_record_reads(table: MagicMock) -> int: + return sum( + 1 for call in table.get_item.call_args_list if call.kwargs["Key"]["SK"].startswith("KB#") + ) + + +def test_a_managed_search_reads_the_kb_record_once(managed_kb): + """One KB_Record read answers the engine, the backend's ``awsKbId`` and the cap. + + Each of those used to read the same row again on the turn's critical path + (~50ms apiece on dev, 2026-10-02). The record the backend receives is the one + the facade read, and the cap returned with the results is the managed one. + """ + backend = managed_kb([_managed_chunk("doc-a", 0)]) + boto_patch, table = _patch_record_and_statuses({"doc-a": "complete"}) + + with patch.dict("os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": TABLE_NAME}), boto_patch: + results, cap = asyncio.run( + search_assistant_knowledgebase_with_cap(ASSISTANT_ID, "q", access=OWNER_ACCESS) + ) + + assert len(results) == 1 + assert cap == 8000 + assert backend.calls[0]["record"] == {"retrievalEngine": ENGINE_MANAGED} + assert _kb_record_reads(table) == 1 + + +def test_a_legacy_search_reads_the_kb_record_once_and_returns_the_legacy_cap(): + boto_patch, table = _patch_legacy_record_and_statuses({"doc-a": "complete"}) + vectors = { + "vectors": [ + {"key": "doc-a#0", "distance": 0.1, "metadata": {"document_id": "doc-a", "text": "t"}} + ] + } + with patch.dict("os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": TABLE_NAME}), boto_patch, patch( + "apis.shared.embeddings.bedrock_embeddings.search_assistant_knowledgebase", + return_value=vectors, + ): + results, cap = asyncio.run( + search_assistant_knowledgebase_with_cap(ASSISTANT_ID, "q", access=OWNER_ACCESS) + ) + + assert len(results) == 1 + assert cap == 2000 + assert _kb_record_reads(table) == 1 + + +def test_a_refused_search_reads_nothing_and_returns_the_legacy_cap(): + boto_patch, table = _patch_record_and_statuses({}) + with patch.dict("os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": TABLE_NAME}), boto_patch: + results, cap = asyncio.run( + search_assistant_knowledgebase_with_cap(ASSISTANT_ID, "q", access=None) + ) + + assert (results, cap) == ([], 2000) + table.get_item.assert_not_called() + + def test_managed_path_narrows_results_to_top_k(managed_kb): """An over-generous backend is narrowed by the facade, not trusted. diff --git a/backend/tests/shared/test_kb_dual_read.py b/backend/tests/shared/test_kb_dual_read.py index 9e373078c..308248b79 100644 --- a/backend/tests/shared/test_kb_dual_read.py +++ b/backend/tests/shared/test_kb_dual_read.py @@ -58,7 +58,9 @@ def __init__( self._error = error self.calls: List[Dict[str, Any]] = [] - async def search(self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K) -> List[Chunk]: + async def search( + self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K, *, record=None + ) -> List[Chunk]: self.calls.append({"kb_ref": kb_ref, "query": query, "top_k": top_k}) if self._delay: await asyncio.sleep(self._delay) @@ -128,7 +130,7 @@ async def _search(legacy: StubBackend, record: Dict[str, Any], top_k: int = 5): ), patch( "apis.shared.assistants.rag_service.resolve_backend", return_value=legacy ), patch( - "apis.shared.assistants.rag_service.boto3.resource", + "boto3.resource", return_value=_all_documents_complete(), ), patch( "apis.shared.assistants.rag_service.emit_count" @@ -290,19 +292,32 @@ async def test_a_legacy_failure_still_fails_closed(self, managed_registered): """And does not leave the managed task orphaned with an unretrieved exception. - ``managed.calls`` is empty because the facade cancels the task it started - when legacy raised before the comparison was detached. Without that - cancel the task runs to completion — paying for a Retrieve nobody will - ever read — and Python reports a task whose exception was never - retrieved. + The facade cancels the task it started when legacy raised before the + comparison was detached. Without that cancel the task runs to + completion — paying for a Retrieve nobody will ever read — and Python + reports a task whose exception was never retrieved. + + The managed read does *start*: the facade awaits the legacy search on a + worker thread (turn-path spec §5 P3b), which yields to the loop, as a + real legacy search always has — its S3 Vectors call was already off the + loop. What must hold is that it never finishes. """ - managed = managed_registered(StubBackend([_chunk("doc-managed-1")])) + finished: List[bool] = [] + + class _Tracked(StubBackend): + async def search(self, kb_ref, query, top_k=DEFAULT_TOP_K, *, record=None): + result = await super().search(kb_ref, query, top_k) + finished.append(True) + return result + + managed = managed_registered(_Tracked([_chunk("doc-managed-1")], delay=0.2)) legacy = StubBackend(error=RuntimeError("s3 vectors threw")) results, _, _ = await _search(legacy, {DUAL_READ_ATTR: True}) + await asyncio.sleep(0.4) assert results == [] - assert managed.calls == [], ( + assert finished == [], ( "the orphaned managed read was left running after legacy failed" ) @@ -328,7 +343,7 @@ async def test_the_caller_does_not_wait_for_the_managed_read(self, managed_regis ), patch( "apis.shared.assistants.rag_service.resolve_backend", return_value=legacy ), patch( - "apis.shared.assistants.rag_service.boto3.resource", + "boto3.resource", return_value=_all_documents_complete(), ), patch( "apis.shared.assistants.rag_service.emit_count" @@ -355,14 +370,14 @@ async def test_the_managed_read_starts_before_legacy_resolves(self, managed_regi order: List[str] = [] class OrderedLegacy(StubBackend): - async def search(self, kb_ref, query, top_k=DEFAULT_TOP_K): + async def search(self, kb_ref, query, top_k=DEFAULT_TOP_K, *, record=None): order.append("legacy-start") await asyncio.sleep(0.05) order.append("legacy-end") return [_chunk("doc-legacy-1")] class OrderedManaged(StubBackend): - async def search(self, kb_ref, query, top_k=DEFAULT_TOP_K): + async def search(self, kb_ref, query, top_k=DEFAULT_TOP_K, *, record=None): order.append("managed-start") return [_chunk("doc-managed-1")] diff --git a/backend/tests/shared/test_kb_post_search_off_loop.py b/backend/tests/shared/test_kb_post_search_off_loop.py new file mode 100644 index 000000000..2a086e95f --- /dev/null +++ b/backend/tests/shared/test_kb_post_search_off_loop.py @@ -0,0 +1,120 @@ +"""The knowledge-base search's follow-up reads stay off the event loop. + +An agent turn starts its knowledge-base search before the agent build and +awaits it after (turn-path spec §5 P3b), and a cold build holds the event loop. +After #1411 two DynamoDB reads still ran on the loop once the search returned, +so they waited for the build to let go — ~120ms of residual `rag_wait` on dev: + +- the document-status filter, which drops chunks from documents that are not + `complete` (fail-closed), and +- `resolve_context_cap`'s knowledge-base record read. + +The filter now runs on the same worker thread as the backend search, right +after it. The cap needs no read of its own: it is decided from the record the +search already read, and comes back with the results. Neither changes what +reaches the model. +""" + +import asyncio +import threading +from typing import Any, List +from unittest.mock import patch + +import pytest + +import apis.inference_api.chat.routes as routes +from apis.shared.assistants import rag_service +from apis.shared.assistants.kb_access import granted +from apis.shared.kb_backend.protocol import DEFAULT_TOP_K, Chunk + +ASSISTANT_ID = "ast-offloop-001" +ACCESS = granted(ASSISTANT_ID, "user-1", "owner") + + +def _chunk(doc: str) -> Chunk: + return Chunk(text=f"text from {doc}", relevance=0.9, document_id=doc, key=f"k-{doc}", metadata={"document_id": doc}) + + +class _Backend: + def __init__(self, chunks: List[Chunk]) -> None: + self.chunks = chunks + self.thread = None + + async def search( + self, kb_ref: str, query: str, top_k: int = DEFAULT_TOP_K, *, record=None + ) -> List[Chunk]: + self.thread = threading.current_thread() + return list(self.chunks) + + +@pytest.mark.asyncio +async def test_the_status_filter_runs_on_the_search_thread_not_the_loop(): + backend = _Backend([_chunk("doc-1"), _chunk("doc-gone")]) + filter_threads: List[Any] = [] + + def _filter(chunks, assistant_id): + filter_threads.append(threading.current_thread()) + return [c for c in chunks if c.metadata["document_id"] != "doc-gone"] + + with patch.object(rag_service, "load_record", return_value={}), patch.object( + rag_service, "resolve_backend", return_value=backend + ), patch.object(rag_service, "_filter_chunks_by_document_status", side_effect=_filter), patch.object( + rag_service, "emit_count" + ): + results = await rag_service.search_assistant_knowledgebase_with_formatting( + ASSISTANT_ID, "q", 5, access=ACCESS + ) + + assert [r["metadata"]["document_id"] for r in results] == ["doc-1"] + assert filter_threads == [backend.thread] + assert backend.thread is not threading.current_thread() + + +@pytest.mark.asyncio +async def test_no_chunks_means_no_filter_read(): + backend = _Backend([]) + with patch.object(rag_service, "load_record", return_value={}), patch.object( + rag_service, "resolve_backend", return_value=backend + ), patch.object( + rag_service, "_filter_chunks_by_document_status", side_effect=AssertionError("filtered nothing") + ), patch.object(rag_service, "emit_count"): + assert await rag_service.search_assistant_knowledgebase_with_formatting( + ASSISTANT_ID, "q", 5, access=ACCESS + ) == [] + + +CHUNKS = [{"text": "Use a four-level scale.", "distance": 0.1, "metadata": {"document_id": "d1"}, "key": "k1"}] + + +@pytest.mark.asyncio +async def test_the_cap_comes_back_with_the_search_and_is_not_read_again(): + """The route takes the cap the search decided from its one record read. + + A second read here — on the loop or on a worker — is the record read three + times again (~50ms apiece on dev before this). + """ + async def _search(**kwargs): + return list(CHUNKS), 8000 + + with patch.object( + rag_service, "resolve_context_cap", side_effect=AssertionError("the cap was read again") + ), patch.object(rag_service, "search_assistant_knowledgebase_with_cap", side_effect=_search): + chunks, augmented = await routes._search_and_augment( + assistant_id=ASSISTANT_ID, message="How many levels?", access=ACCESS + ) + + assert chunks == CHUNKS + assert augmented == rag_service.augment_prompt_with_context( + user_message="How many levels?", context_chunks=CHUNKS, max_context_length=8000 + ) + + +@pytest.mark.asyncio +async def test_a_failed_search_fails_open_like_before(): + async def _search(**kwargs): + raise RuntimeError("ddb down") + + with patch.object(rag_service, "search_assistant_knowledgebase_with_cap", side_effect=_search): + assert await routes._search_and_augment( + assistant_id=ASSISTANT_ID, message="m", access=ACCESS + ) == (None, "m") diff --git a/backend/tests/shared/test_kb_search_ahead_flag.py b/backend/tests/shared/test_kb_search_ahead_flag.py new file mode 100644 index 000000000..5e8d8ec92 --- /dev/null +++ b/backend/tests/shared/test_kb_search_ahead_flag.py @@ -0,0 +1,21 @@ +"""`kb_search_ahead_enabled`: default ON, `false` is the kill switch.""" + +import pytest + +from apis.shared.feature_flags import kb_search_ahead_enabled + + +@pytest.mark.parametrize("value", [None, "", "true", "on", "nonsense"]) +def test_everything_but_false_is_on(monkeypatch, value): + if value is None: + monkeypatch.delenv("KB_SEARCH_AHEAD_ENABLED", raising=False) + else: + monkeypatch.setenv("KB_SEARCH_AHEAD_ENABLED", value) + + assert kb_search_ahead_enabled() + + +@pytest.mark.parametrize("value", ["false", "FALSE", " False "]) +def test_false_is_off(monkeypatch, value): + monkeypatch.setenv("KB_SEARCH_AHEAD_ENABLED", value) + assert not kb_search_ahead_enabled() diff --git a/backend/tests/shared/test_kb_table_handles.py b/backend/tests/shared/test_kb_table_handles.py new file mode 100644 index 000000000..f68d594b8 --- /dev/null +++ b/backend/tests/shared/test_kb_table_handles.py @@ -0,0 +1,58 @@ +"""The knowledge base search's DynamoDB handles are process-cached. + +A fresh ``boto3.resource`` per read paid ~45–55ms for its first request in the +Runtime (a new connection pool) against ~4ms on a reused one, measured from the +botocore spans of a managed-KB follow-up turn on dev (2026-10-02). The search read +the KB_Record and the document statuses on four fresh resources, all on the turn's +critical path. +""" + +from unittest.mock import MagicMock, patch + +import pytest + +from apis.shared.assistants import rag_service +from apis.shared.kb_backend import records + + +@pytest.fixture +def resource(): + dynamo = MagicMock() + with patch("boto3.resource", return_value=dynamo) as factory, patch.dict( + "os.environ", {"DYNAMODB_ASSISTANTS_TABLE_NAME": "assistants"} + ): + yield factory, dynamo + + +class TestTheKbRecordTable: + def test_reads_share_one_resource(self, resource): + factory, dynamo = resource + + records._table() + records._table() + + assert factory.call_count == 1 + dynamo.Table.assert_called_with("assistants") + + +class TestTheDocumentStatusTable: + def test_lookups_share_one_resource(self, resource): + factory, dynamo = resource + + rag_service._documents_table("assistants", "us-west-2") + rag_service._documents_table("assistants", "us-west-2") + + assert factory.call_count == 1 + dynamo.Table.assert_called_with("assistants") + + def test_shares_the_kb_record_reads_resource(self, resource): + """Same table, same region: the record read and the status lookups ride one + connection pool, so the status lookup finds the record read's connection + already open.""" + factory, _ = resource + + with patch.dict("os.environ", {"AWS_REGION": "us-west-2"}): + records._table() + rag_service._documents_table("assistants", "us-west-2") + + assert factory.call_count == 1 diff --git a/backend/tests/shared/test_managed_kb_backend.py b/backend/tests/shared/test_managed_kb_backend.py index f82ec3be9..a0ca4bee5 100644 --- a/backend/tests/shared/test_managed_kb_backend.py +++ b/backend/tests/shared/test_managed_kb_backend.py @@ -1198,6 +1198,36 @@ async def test_the_kb_record_supplies_the_ids_when_no_locator_is_given(self, tab assert runtime.calls[0]["knowledgeBaseId"] == AWS_KB_ID + @pytest.mark.asyncio + async def test_a_record_the_caller_already_read_is_not_read_again(self): + """The facade read this row to choose the backend; a second read of the + same row cost ~50ms on a turn's critical path.""" + looked_up: List[str] = [] + + def _locate(kb_ref: str) -> Tuple[str, str]: + looked_up.append(kb_ref) + return "KB-FROM-LOCATOR", AWS_DS_ID + + runtime = FakeBedrockAgentRuntime([]) + backend = mb.ManagedKbBackend(runtime_client=runtime, locator=_locate) + + await backend.search(APP_KB_ID, "query", record={"awsKbId": "KB-FROM-RECORD"}) + + assert runtime.calls[0]["knowledgeBaseId"] == "KB-FROM-RECORD" + assert looked_up == [] + + @pytest.mark.asyncio + @pytest.mark.parametrize("record", [{}, {"awsKbId": None}, {"retrievalEngine": "managed"}]) + async def test_a_record_without_an_aws_kb_id_falls_back_to_the_lookup(self, record): + """A record that cannot answer is not trusted to: the backend resolves the + id itself, and an unprovisioned knowledge base still raises from there.""" + runtime = FakeBedrockAgentRuntime([]) + backend = mb.ManagedKbBackend(runtime_client=runtime, locator=_locator("KB-RESOLVED")) + + await backend.search(APP_KB_ID, "query", record=record) + + assert runtime.calls[0]["knowledgeBaseId"] == "KB-RESOLVED" + # =========================================================================== # 8.4 — direct ingestion and deletion diff --git a/backend/tests/shared/test_managed_models.py b/backend/tests/shared/test_managed_models.py index 28fbb4491..b0b7fd7aa 100644 --- a/backend/tests/shared/test_managed_models.py +++ b/backend/tests/shared/test_managed_models.py @@ -363,3 +363,60 @@ async def catalog(): monkeypatch.setattr(mm, "list_all_managed_models", catalog) assert await mm.get_default_managed_model() is None + + +class TestSpeechPricing: + """Speech-to-speech rows carry a second rate card; text rows carry none.""" + + @pytest.fixture(autouse=True) + def _patch_dynamodb(self, managed_models_table, monkeypatch): + import apis.shared.models.managed_models as mm + monkeypatch.setattr(mm, "dynamodb", boto3.resource("dynamodb", region_name="us-east-1")) + + @pytest.mark.asyncio + async def test_speech_rates_round_trip(self): + from apis.shared.models.managed_models import create_managed_model, get_managed_model + data = _make_model_data( + "amazon.nova-2-sonic-v1:0", + inputModalities=["speech", "text"], outputModalities=["speech", "text"], + inputPricePerMillionTokens=0.319, outputPricePerMillionTokens=2.651, + speechInputPricePerMillionTokens=3.0, speechOutputPricePerMillionTokens=12.0, + ) + created = await create_managed_model(data) + stored = await get_managed_model(created.id) + assert stored.speech_input_price_per_million_tokens == pytest.approx(3.0) + assert stored.speech_output_price_per_million_tokens == pytest.approx(12.0) + + @pytest.mark.asyncio + async def test_text_rows_store_no_speech_rates(self): + from apis.shared.models.managed_models import create_managed_model, get_managed_model + created = await create_managed_model(_make_model_data("text-only")) + stored = await get_managed_model(created.id) + assert stored.speech_input_price_per_million_tokens is None + assert stored.speech_output_price_per_million_tokens is None + + @pytest.mark.asyncio + async def test_speech_rates_reach_the_pricing_dict(self): + from apis.shared.models.managed_models import create_managed_model + from apis.shared.costs.pricing_config import get_model_pricing + await create_managed_model(_make_model_data( + "amazon.nova-2-sonic-v1:0", + inputPricePerMillionTokens=0.319, outputPricePerMillionTokens=2.651, + speechInputPricePerMillionTokens=3.0, speechOutputPricePerMillionTokens=12.0, + )) + pricing = await get_model_pricing("amazon.nova-2-sonic-v1:0") + assert pricing["speechInputPricePerMtok"] == pytest.approx(3.0) + assert pricing["speechOutputPricePerMtok"] == pytest.approx(12.0) + assert pricing["inputPricePerMtok"] == pytest.approx(0.319) + + @pytest.mark.asyncio + async def test_update_can_add_speech_rates_later(self): + from apis.shared.models.managed_models import create_managed_model, update_managed_model + from apis.shared.models.models import ManagedModelUpdate + created = await create_managed_model(_make_model_data("sonic")) + updated = await update_managed_model( + created.id, + ManagedModelUpdate(speechInputPricePerMillionTokens=3.0, speechOutputPricePerMillionTokens=12.0), + ) + assert updated.speech_input_price_per_million_tokens == pytest.approx(3.0) + assert updated.speech_output_price_per_million_tokens == pytest.approx(12.0) diff --git a/backend/tests/shared/test_mcp_parallel_preflight_flag.py b/backend/tests/shared/test_mcp_parallel_preflight_flag.py new file mode 100644 index 000000000..ebef38623 --- /dev/null +++ b/backend/tests/shared/test_mcp_parallel_preflight_flag.py @@ -0,0 +1,21 @@ +"""`mcp_parallel_preflight_enabled`: default ON, `false` is the kill switch.""" + +import pytest + +from apis.shared.feature_flags import mcp_parallel_preflight_enabled + + +@pytest.mark.parametrize("value", [None, "", "true", "on", "nonsense"]) +def test_everything_but_false_is_on(monkeypatch, value): + if value is None: + monkeypatch.delenv("MCP_PARALLEL_PREFLIGHT_ENABLED", raising=False) + else: + monkeypatch.setenv("MCP_PARALLEL_PREFLIGHT_ENABLED", value) + + assert mcp_parallel_preflight_enabled() + + +@pytest.mark.parametrize("value", ["false", "FALSE", " False "]) +def test_false_is_off(monkeypatch, value): + monkeypatch.setenv("MCP_PARALLEL_PREFLIGHT_ENABLED", value) + assert not mcp_parallel_preflight_enabled() diff --git a/backend/tests/shared/test_memory_retrieval_prefetch_flag.py b/backend/tests/shared/test_memory_retrieval_prefetch_flag.py new file mode 100644 index 000000000..5a5112929 --- /dev/null +++ b/backend/tests/shared/test_memory_retrieval_prefetch_flag.py @@ -0,0 +1,21 @@ +"""`memory_retrieval_prefetch_enabled`: default ON, `false` is the kill switch.""" + +import pytest + +from apis.shared.feature_flags import memory_retrieval_prefetch_enabled + + +@pytest.mark.parametrize("value", [None, "", "true", "on", "nonsense"]) +def test_everything_but_false_is_on(monkeypatch, value): + if value is None: + monkeypatch.delenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", raising=False) + else: + monkeypatch.setenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", value) + + assert memory_retrieval_prefetch_enabled() + + +@pytest.mark.parametrize("value", ["false", "FALSE", " False "]) +def test_false_is_off(monkeypatch, value): + monkeypatch.setenv("MEMORY_RETRIEVAL_PREFETCH_ENABLED", value) + assert not memory_retrieval_prefetch_enabled() diff --git a/backend/tests/shared/test_modalities.py b/backend/tests/shared/test_modalities.py new file mode 100644 index 000000000..3b2b5a5a3 --- /dev/null +++ b/backend/tests/shared/test_modalities.py @@ -0,0 +1,26 @@ +"""Speech-to-speech rows are priced, never picked.""" + +from types import SimpleNamespace + +from apis.shared.models.modalities import is_speech_model + + +def test_speech_output_is_a_speech_model_regardless_of_case(): + assert is_speech_model(SimpleNamespace(output_modalities=["SPEECH", "TEXT"])) + assert is_speech_model(SimpleNamespace(output_modalities=["speech"])) + + +def test_text_and_image_rows_are_not(): + assert not is_speech_model(SimpleNamespace(output_modalities=["TEXT"])) + assert not is_speech_model(SimpleNamespace(output_modalities=["TEXT", "IMAGE"])) + + +def test_missing_or_empty_modalities_are_not(): + assert not is_speech_model(SimpleNamespace(output_modalities=None)) + assert not is_speech_model(SimpleNamespace(output_modalities=[])) + assert not is_speech_model(SimpleNamespace()) + + +def test_speech_input_alone_does_not_make_a_speech_model(): + # Voxtral-style speech-to-text answers Converse; it belongs in the picker. + assert not is_speech_model(SimpleNamespace(input_modalities=["SPEECH"], output_modalities=["TEXT"])) diff --git a/backend/tests/shared/test_session_has_messages.py b/backend/tests/shared/test_session_has_messages.py new file mode 100644 index 000000000..f96bb5d48 --- /dev/null +++ b/backend/tests/shared/test_session_has_messages.py @@ -0,0 +1,134 @@ +"""`session_has_messages`: an existence check priced like one. + +The agent-binding rules ask "does this thread already have messages?" before +the stream opens, on an agent conversation's first turn and on every +`@`-mention. It used to be answered with `get_messages(limit=1)`, which read the +whole history plus its metadata, interrupts, UI resources and tool summaries +to return one message (~540ms on dev, growing with history), and wrote a +`SESSION` event into a session that had none. Now it is one `ListEvents`: +`maxResults=1`, no payloads, filtered server-side to events that are not the +Memory SDK's own state records. + +What must hold: the same answer (any message from any agent → True; only the +SDK's state records, or nothing → False), no write, fail-open on a failed read. +""" + +from typing import Any, Dict, List + +import pytest + +from apis.shared.sessions import messages as messages_module +from apis.shared.sessions.messages import session_has_messages + + +class _Client: + def __init__(self, pages: List[Dict[str, Any]] = None, raises: Exception = None) -> None: + self.pages = list(pages or []) + self.raises = raises + self.calls: List[Dict[str, Any]] = [] + + def list_events(self, **kwargs): + self.calls.append(kwargs) + if self.raises: + raise self.raises + return self.pages.pop(0) + + +@pytest.fixture +def client(monkeypatch): + holder = {} + + def _install(**kwargs): + holder["client"] = _Client(**kwargs) + monkeypatch.setattr( + "apis.shared.aws_clients.get_client", lambda service, region_name=None: holder["client"] + ) + return holder["client"] + + monkeypatch.setenv("AGENTCORE_MEMORY_ID", "mem-1") + return _install + + +@pytest.mark.asyncio +async def test_asks_for_one_non_state_event_without_payloads(client): + fake = client(pages=[{"events": [{"eventId": "e1"}]}]) + + assert await session_has_messages("sess-1", "user-1") is True + + [call] = fake.calls + assert call == { + "memoryId": "mem-1", + "actorId": "user-1", + "sessionId": "sess-1", + "maxResults": 1, + "includePayloads": False, + "filter": { + "eventMetadata": [{"left": {"metadataKey": "stateType"}, "operator": "NOT_EXISTS"}] + }, + } + + +@pytest.mark.asyncio +async def test_no_message_events_is_no_messages(client): + client(pages=[{"events": []}]) + assert await session_has_messages("sess-1", "user-1") is False + + +@pytest.mark.asyncio +async def test_an_empty_page_with_a_token_is_not_the_answer(client): + fake = client(pages=[{"events": [], "nextToken": "t1"}, {"events": [{"eventId": "e9"}]}]) + + assert await session_has_messages("sess-1", "user-1") is True + assert fake.calls[1]["nextToken"] == "t1" + + +@pytest.mark.asyncio +async def test_a_failed_read_fails_open(client): + client(raises=RuntimeError("throttled")) + assert await session_has_messages("sess-1", "user-1") is False + + +@pytest.mark.asyncio +async def test_no_memory_configured_still_raises(monkeypatch): + monkeypatch.delenv("AGENTCORE_MEMORY_ID", raising=False) + with pytest.raises(ValueError): + await session_has_messages("sess-1", "user-1") + + +@pytest.mark.asyncio +async def test_never_builds_a_session_manager(client, monkeypatch): + """Constructing one writes a SESSION event into a session that has none.""" + + def _no(*args, **kwargs): + raise AssertionError("session_has_messages must not construct a session manager") + + monkeypatch.setattr(messages_module, "AgentCoreMemorySessionManager", _no, raising=False) + client(pages=[{"events": []}]) + + assert await session_has_messages("sess-1", "user-1") is False + + +def test_the_sdk_still_tags_its_state_records_with_the_key_we_filter_on(): + """The whole check rests on this: message events carry no `stateType`, the + SDK's SESSION/AGENT records do, and user metadata cannot set it.""" + from bedrock_agentcore.memory.integrations.strands import session_manager as sdk + + assert sdk.STATE_TYPE_KEY == messages_module._SDK_STATE_METADATA_KEY + assert sdk.STATE_TYPE_KEY in sdk.RESERVED_METADATA_KEYS + assert {s.value for s in sdk.StateType} >= {"SESSION", "AGENT"} + + +@pytest.mark.asyncio +async def test_the_route_check_delegates_to_it(monkeypatch): + import apis.inference_api.chat.routes as routes + + seen = [] + + async def _fake(session_id, user_id): + seen.append((session_id, user_id)) + return True + + monkeypatch.setattr(messages_module, "session_has_messages", _fake) + + assert await routes._session_has_messages(session_id="sess-1", user_id="user-1") is True + assert seen == [("sess-1", "user-1")] diff --git a/backend/tests/shared/test_sessions_metadata.py b/backend/tests/shared/test_sessions_metadata.py index cf659cf2e..60b2d23f8 100644 --- a/backend/tests/shared/test_sessions_metadata.py +++ b/backend/tests/shared/test_sessions_metadata.py @@ -1590,3 +1590,42 @@ async def test_marker_survives_the_metadata_read_model(self, sessions_metadata_t meta = await get_session_metadata("s1", "u1") assert meta.pending_attachment_upload_ids == ["up-1", "up-2"] assert meta.pending_attachments_at is not None + + +class TestUpdateSessionTitle: + """One round trip on the born-static row; the GSI only for legacy SKs.""" + + @pytest.mark.asyncio + async def test_writes_a_born_static_row(self, sessions_metadata_table): + from apis.shared.sessions.metadata import ( + ensure_session_metadata_exists, + get_session_metadata, + update_session_title, + ) + await ensure_session_metadata_exists("s1", "u1") + + await update_session_title("s1", "u1", "Biology Syllabus") + + assert (await get_session_metadata("s1", "u1")).title == "Biology Syllabus" + + @pytest.mark.asyncio + async def test_falls_back_to_the_gsi_for_a_legacy_row(self, sessions_metadata_table): + from apis.shared.sessions.metadata import update_session_title + + _put_legacy_row(sessions_metadata_table, "s1", "2026-01-01T00:00:00Z") + + await update_session_title("s1", "u1", "Biology Syllabus") + + items = sessions_metadata_table.scan()["Items"] + assert [(i["SK"], i["title"]) for i in items] == [ + ("S#ACTIVE#2026-01-01T00:00:00Z#s1", "Biology Syllabus") + ] + + @pytest.mark.asyncio + async def test_a_missing_row_is_not_created(self, sessions_metadata_table): + """The guard is what keeps the keyed write from upserting a ghost row.""" + from apis.shared.sessions.metadata import update_session_title + + await update_session_title("s1", "u1", "Biology Syllabus") + + assert sessions_metadata_table.scan()["Items"] == [] diff --git a/backend/tests/shared/test_side_channel_inference_config.py b/backend/tests/shared/test_side_channel_inference_config.py index c01f86076..c4a65c457 100644 --- a/backend/tests/shared/test_side_channel_inference_config.py +++ b/backend/tests/shared/test_side_channel_inference_config.py @@ -8,31 +8,53 @@ other side channels to the same shape. """ +import asyncio from unittest.mock import AsyncMock, MagicMock import pytest import apis.inference_api.chat.service as chat_service +from apis.shared import aws_clients from apis.shared.files import document_digest as dd from apis.shared.tool_summaries.summarizer import summarize_tool_batch -def _client(text: str) -> MagicMock: +def _client(text: str, stop_reason: str = "end_turn") -> MagicMock: def converse(**kwargs): config = kwargs.get("inferenceConfig", {}) if "temperature" in config and "topP" in config: raise RuntimeError("ValidationException: `temperature` and `top_p` cannot both be specified") - return {"stopReason": "end_turn", "output": {"message": {"content": [{"text": text}]}}} + return {"stopReason": stop_reason, "output": {"message": {"content": [{"text": text}]}}} client = MagicMock() client.converse.side_effect = converse return client +@pytest.fixture(autouse=True) +def _fresh_title_client(monkeypatch): + """The title client is cached per process; each test builds its own.""" + monkeypatch.setattr(chat_service, "_title_bedrock_client", None) + + +async def _drain_title_writes() -> None: + await asyncio.gather(*list(chat_service._pending_title_writes)) + + def _patch_boto3_module(monkeypatch, client: MagicMock) -> None: module = MagicMock() module.client.return_value = client monkeypatch.setitem(__import__("sys").modules, "boto3", module) + # The side channels share a process-cached client; drop the previous fake. + aws_clients.reset_cached_clients() + + +@pytest.fixture(autouse=True) +def _fresh_bedrock_client(): + """The Bedrock client is cached per process; each test builds its own.""" + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() @pytest.mark.asyncio @@ -64,3 +86,67 @@ async def test_conversation_title(monkeypatch): title = await chat_service.generate_conversation_title(session_id="s", user_id="u", user_input="hi") assert title == "Planning a biology syllabus" assert "topP" not in client.converse.call_args.kwargs["inferenceConfig"] + await _drain_title_writes() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("stop_reason", ["guardrail_intervened", "content_filtered", None]) +async def test_conversation_title_refusal_keeps_the_placeholder(monkeypatch, stop_reason): + """A guardrail stop's text is the refusal; it must never become the title.""" + client = _client("Sorry, the model cannot answer this.", stop_reason=stop_reason) + monkeypatch.setattr(chat_service.boto3, "client", MagicMock(return_value=client)) + update = AsyncMock() + monkeypatch.setattr(chat_service, "update_session_title", update) + title = await chat_service.generate_conversation_title(session_id="s", user_id="u", user_input="hi") + assert title == "New Conversation" + update.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_conversation_title_at_the_token_ceiling_is_still_clipped(monkeypatch): + client = _client("A" * 80, stop_reason="max_tokens") + monkeypatch.setattr(chat_service.boto3, "client", MagicMock(return_value=client)) + monkeypatch.setattr(chat_service, "update_session_title", AsyncMock()) + title = await chat_service.generate_conversation_title(session_id="s", user_id="u", user_input="hi") + assert title == "A" * 47 + "..." + await _drain_title_writes() + + +@pytest.mark.asyncio +async def test_conversation_title_returns_before_its_write_lands(monkeypatch): + """The stream pushes `session_title` when this task finishes, so the + DynamoDB write must not sit between Nova answering and the user seeing it.""" + client = _client("Planning a biology syllabus") + monkeypatch.setattr(chat_service.boto3, "client", MagicMock(return_value=client)) + write_may_finish = asyncio.Event() + written: list = [] + + async def slow_write(session_id, user_id, title): + await write_may_finish.wait() + written.append(title) + + monkeypatch.setattr(chat_service, "update_session_title", slow_write) + + title = await chat_service.generate_conversation_title(session_id="s", user_id="u", user_input="hi") + + assert title == "Planning a biology syllabus" + assert written == [] + write_may_finish.set() + await _drain_title_writes() + assert written == ["Planning a biology syllabus"] + + +@pytest.mark.asyncio +async def test_conversation_title_reuses_one_bedrock_client(monkeypatch): + """A client per title paid botocore's model load and a fresh TLS handshake.""" + client = _client("Planning a biology syllabus") + factory = MagicMock(return_value=client) + monkeypatch.setattr(chat_service.boto3, "client", factory) + monkeypatch.setattr(chat_service, "update_session_title", AsyncMock()) + + for _ in range(3): + await chat_service.generate_conversation_title(session_id="s", user_id="u", user_input="hi") + await _drain_title_writes() + + assert factory.call_count == 1 + assert client.converse.call_count == 3 diff --git a/backend/tests/shared/test_usage_normalization.py b/backend/tests/shared/test_usage_normalization.py index 12cc96bfb..64180fc02 100644 --- a/backend/tests/shared/test_usage_normalization.py +++ b/backend/tests/shared/test_usage_normalization.py @@ -13,8 +13,9 @@ The SDK-shape tests below deliberately drive the *real* ``strands`` model classes with *real* ``openai`` usage objects rather than hand-rolled stubs. A stub that merely matches the broken behavior would hide the bug, and the whole -mapping hangs off two SDK details (the chunk-formatter method name, and the -fact that Strands drops ``cache_write_tokens``) that a version bump can move. +mapping hangs off two SDK details (the chunk-formatter method name, and +whether Strands' ``inputTokens`` still includes the cache buckets) that a +version bump can move. """ import pytest @@ -206,8 +207,14 @@ def test_chunk_formatter_seams_still_exist(self): assert hasattr(OpenAIResponsesModel, "_format_chunk") assert hasattr(OpenAIModel, "format_chunk") - def test_sdk_still_reports_inclusive_input_and_drops_cache_writes(self): - """The bug this module exists to fix, asserted against the real SDK.""" + def test_responses_sdk_surfaces_cache_writes_but_input_stays_inclusive(self): + """The bug this module exists to fix, asserted against the real SDK. + + strands-agents 1.57.2 (#4193) maps Responses ``cache_write_tokens``, so + our recovery in ``_normalize_metadata_chunk`` now writes the value the + SDK already set. ``inputTokens`` is still the inclusive total, so + disjointness remains ours to restore — exactly once. + """ from openai.types.responses.response_usage import ResponseUsage from strands.models import OpenAIResponsesModel @@ -223,12 +230,61 @@ def test_sdk_still_reports_inclusive_input_and_drops_cache_writes(self): # inputTokens is the inclusive total, not the uncached remainder... assert usage["inputTokens"] == 30_500 assert usage["cacheReadInputTokens"] == 30_000 - # ...and the write bucket never makes it out of the SDK. - assert "cacheWriteInputTokens" not in usage + # ...and the write bucket is reported, but still inside inputTokens. + assert usage["cacheWriteInputTokens"] == 400 with pytest.raises(AssertionError): _assert_disjoint(usage) + def test_chat_completions_sdk_now_surfaces_cache_writes(self): + """strands-agents 1.57 (#4361) maps Chat Completions cache writes. + + They arrive as ``cacheWriteInputTokens`` beside an ``inputTokens`` that + is still inclusive, so the shim must subtract them exactly once — which + is what ``normalize_usage`` already does for any write bucket present. + """ + from openai.types.completion_usage import CompletionUsage + from strands.models.openai import OpenAIModel + + wire = { + "prompt_tokens": 10_000, + "completion_tokens": 50, + "total_tokens": 10_050, + "prompt_tokens_details": {"cached_tokens": 6_000, "cache_write_tokens": 3_000}, + } + usage_obj = CompletionUsage.model_validate(wire) + + raw = OpenAIModel(client_args={"api_key": "test-key"}, model_id="openai.gpt-5.6").format_chunk( + {"chunk_type": "metadata", "data": usage_obj} + )["metadata"]["usage"] + assert raw["inputTokens"] == 10_000 + assert raw["cacheWriteInputTokens"] == 3_000 + + normalized = usage_normalized(OpenAIModel)( + client_args={"api_key": "test-key"}, model_id="openai.gpt-5.6" + ).format_chunk({"chunk_type": "metadata", "data": usage_obj})["metadata"]["usage"] + assert normalized["inputTokens"] == 1_000 + assert normalized["cacheReadInputTokens"] == 6_000 + assert normalized["cacheWriteInputTokens"] == 3_000 + _assert_disjoint(normalized) + + def test_bedrock_usage_is_still_disjoint_upstream(self): + """Tripwire for strands-agents/harness-sdk#4618. + + That proposal would make the Bedrock adapter fold cache tokens into + ``inputTokens``, and delete the read-time ``_total_prompt_tokens`` sum + that exists only because Bedrock reports disjoint buckets. When this + helper disappears, ``normalize_usage``'s "leave Bedrock untouched" + contract inverts and every cached Bedrock token is priced twice — the + bump that removes it must carry the matching normalization change. + """ + from strands.telemetry import metrics + + assert hasattr(metrics, "_total_prompt_tokens") + usage = {"inputTokens": 100, "outputTokens": 1, "totalTokens": 30_501, "cacheReadInputTokens": 30_000, + "cacheWriteInputTokens": 400} + assert metrics._total_prompt_tokens(usage) == 30_500 + def test_usage_normalized_raises_when_the_seam_disappears(self): class _Seamless: pass diff --git a/backend/tests/shared/test_voice_catalog.py b/backend/tests/shared/test_voice_catalog.py new file mode 100644 index 000000000..5f5bb740b --- /dev/null +++ b/backend/tests/shared/test_voice_catalog.py @@ -0,0 +1,69 @@ +"""The Nova 2 Sonic voice catalog — the one list every voice consumer agrees on.""" + +from apis.shared.voice_catalog import ( + DEFAULT_VOICE_ID, + NOVA_SONIC_VOICES, + VOICE_IDS, + get_voice, + is_valid_voice_id, + resolve_voice_id, +) + +# Every voiceId the Nova 2 user guide lists for `audioOutputConfiguration` +# (sonic-input-events, verified 2026-09-30). Sending anything else fails the +# session at promptStart. +PUBLISHED_VOICE_IDS = { + "matthew", "tiffany", "amy", "olivia", "lupe", "carlos", "ambre", "florian", + "lennart", "beatrice", "lorenzo", "tina", "carolina", "leo", "kiara", "arjun", +} + + +def test_catalog_matches_the_published_voice_ids_exactly(): + assert VOICE_IDS == PUBLISHED_VOICE_IDS + + +def test_ids_are_unique(): + ids = [v.voice_id for v in NOVA_SONIC_VOICES] + assert len(ids) == len(set(ids)) + + +def test_default_is_a_polyglot_voice(): + # The default has to survive a user switching language mid-conversation, + # which only the two polyglot voices do. + assert get_voice(DEFAULT_VOICE_ID).polyglot is True + + +def test_exactly_two_polyglot_voices(): + assert {v.voice_id for v in NOVA_SONIC_VOICES if v.polyglot} == {"tiffany", "matthew"} + + +def test_every_voice_declares_a_presentation_gender(): + assert {v.gender for v in NOVA_SONIC_VOICES} == {"feminine", "masculine"} + + +def test_lookup_is_case_and_whitespace_insensitive(): + assert get_voice(" Matthew ").voice_id == "matthew" + assert is_valid_voice_id("AMY") + + +def test_unknown_blank_and_non_string_are_invalid(): + assert get_voice("siri") is None + assert get_voice("") is None + assert get_voice(None) is None + assert get_voice(42) is None # type: ignore[arg-type] + assert not is_valid_voice_id("siri") + + +def test_resolve_returns_the_catalog_id_for_a_valid_request(): + assert resolve_voice_id("Carlos") == "carlos" + + +def test_resolve_falls_back_for_an_unknown_request(): + assert resolve_voice_id("siri") == DEFAULT_VOICE_ID + assert resolve_voice_id(None) == DEFAULT_VOICE_ID + assert resolve_voice_id("siri", fallback="matthew") == "matthew" + + +def test_resolve_never_trusts_a_bad_fallback(): + # A misconfigured NOVA_SONIC_VOICE env var must not reach promptStart. + assert resolve_voice_id("siri", fallback="alexa") == DEFAULT_VOICE_ID diff --git a/backend/tests/supply_chain/test_ci_path_filter.py b/backend/tests/supply_chain/test_ci_path_filter.py new file mode 100644 index 000000000..4bf3289b7 --- /dev/null +++ b/backend/tests/supply_chain/test_ci_path_filter.py @@ -0,0 +1,191 @@ +"""The PR gate runs only the suites a change can reach. + +ci.yml's ``changes`` job feeds the pull request's file list through +``scripts/ci/classify-changes.sh`` and wires each output to a ``tests.yml`` +input. These tests pin three things: + +1. The classifier's rules — which paths reach which suite — and above all its + fail-open cases: an unknown directory, an empty list, a workflow or script + change all run everything. Only an explicit allowlist runs nothing. +2. That every ``run_*`` input ``tests.yml`` offers is wired on the PR gate. A + new suite whose input ci.yml never passes is a suite that runs nowhere on a + pull request, and nothing else would notice. +3. That the backend job fans out over logical CPUs. ``-n auto`` counts + physical cores when psutil is installed, which on the 4-vCPU runner is 2. +""" + +import subprocess +from pathlib import Path + +import pytest +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[3] +SCRIPT = REPO_ROOT / "scripts" / "ci" / "classify-changes.sh" +WORKFLOWS = REPO_ROOT / ".github" / "workflows" + +SUITES = ("backend", "backend_contracts", "frontend", "infra", "load", "scripts") + + +def classify(paths: list[str]) -> dict[str, bool]: + stdin = "".join(f"{p}\n" for p in paths) + completed = subprocess.run( + ["bash", str(SCRIPT)], + input=stdin, + capture_output=True, + text=True, + check=True, + ) + result: dict[str, bool] = {} + for line in completed.stdout.splitlines(): + key, _, value = line.partition("=") + assert value in ("true", "false"), line + result[key] = value == "true" + assert set(result) == set(SUITES), completed.stdout + return result + + +def on(*names: str) -> dict[str, bool]: + unknown = set(names) - set(SUITES) + assert not unknown, unknown + return {suite: suite in names for suite in SUITES} + + +ALL = on(*SUITES) +NONE = on() + + +def _load_workflow(name: str) -> dict: + with open(WORKFLOWS / name) as f: + return yaml.safe_load(f) + + +def _triggers(workflow: dict) -> dict: + # PyYAML reads a bare `on:` key as the boolean True. + return workflow.get("on", workflow.get(True)) + + +class TestRules: + @pytest.mark.parametrize( + "paths, expected", + [ + # --- nothing reads these ----------------------------------- + (["docs-site/src/content/docs/configuration/feature-flags.md"], NONE), + (["docs/specs/turn-path-ttft.md", "docs/kaizen/review-queue.md"], NONE), + (["CHANGELOG.md", "README.md", "CONTRIBUTING.md", "CLAUDE.MD"], NONE), + (["LICENSE", ".gitignore"], NONE), + (["tui/src/app.py", "tui/pyproject.toml"], NONE), + # --- one package --------------------------------------------- + (["backend/src/apis/shared/feature_flags.py"], on("backend", "load")), + (["backend/tests/shared/test_anything.py"], on("backend", "load")), + (["backend/pyproject.toml", "backend/uv.lock"], on("backend", "load")), + (["backend/README.md"], on("backend", "load")), + (["frontend/ai.client/src/app/app.ts"], on("frontend")), + (["frontend/ai.client/src/branding/README.md"], on("frontend")), + (["infrastructure/lib/config.ts"], on("infra", "backend_contracts")), + (["infrastructure/test/platform-stack.test.ts"], on("infra", "backend_contracts")), + (["tests/supply_chain/test_backup_coverage.py"], on("backend_contracts")), + (["tests/load/locustfile.py"], on("load")), + # --- cross-package reads pinned by name ---------------------- + # pending-backfills.test.ts asserts the backfill scripts exist. + (["backend/scripts/backfill_something.py"], on("backend", "load", "infra")), + # pending-backfills.test.ts reads RELEASE_NOTES.md. + (["RELEASE_NOTES.md"], on("infra", "backend_contracts")), + # gsi-update-limit.test.ts asserts against the release skill. + ([".claude/skills/cutting-a-release/SKILL.md"], on("infra")), + # --- unions ---------------------------------------------------- + ( + ["frontend/ai.client/src/app/app.ts", "backend/src/x.py", "docs/x.md"], + on("frontend", "backend", "load"), + ), + ( + ["infrastructure/lib/x.ts", "backend/src/x.py"], + on("infra", "backend_contracts", "backend", "load"), + ), + ], + ) + def test_paths_reach_exactly_the_suites_that_read_them(self, paths, expected): + assert classify(paths) == expected + + @pytest.mark.parametrize( + "paths", + [ + # No PR, or the file-list call failed: nothing to reason from. + [], + [""], + # Workflows and composite actions are read by the supply-chain + # suites and shape every job. + [".github/workflows/tests.yml"], + [".github/workflows/ci.yml"], + [".github/actions/something/action.yml"], + [".github/dependabot.yml"], + [".github/ARTIFACT_RETENTION.md"], + # scripts/ feeds four different suites. + ["scripts/frontend/test.sh"], + ["scripts/release/check-pending-backfills.mjs"], + ["scripts/restore-data/restore.py"], + ["scripts/ci/classify-changes.sh"], + # A directory this script has never heard of. + ["brand-new-package/src/main.py"], + # A root file this script has never heard of. + ["Makefile"], + ["VERSION"], + # One unknown path in a docs PR still runs everything. + ["docs/x.md", "brand-new-package/main.py"], + ], + ) + def test_anything_unrecognised_runs_everything(self, paths): + assert classify(paths) == ALL + + def test_a_rename_is_classified_under_both_names(self): + # ci.yml feeds `.filename` and `.previous_filename` as separate lines. + assert classify(["frontend/new.ts", "backend/old.py"]) == on( + "frontend", "backend", "load" + ) + + +class TestWiring: + def test_ci_wires_every_suite_tests_yml_offers(self): + ci = _load_workflow("ci.yml") + tests = _load_workflow("tests.yml") + + offered = { + name + for name in _triggers(tests)["workflow_call"]["inputs"] + if name.startswith("run_") + } + wired = ci["jobs"]["tests"]["with"] + assert set(wired) == offered, ( + "ci.yml must pass every run_* input tests.yml declares; " + f"missing={offered - set(wired)} extra={set(wired) - offered}" + ) + for name, value in wired.items(): + assert isinstance(value, str) and "needs.changes.outputs." in value, ( + f"{name} must be decided by the changes job, not hard-coded: {value!r}" + ) + + def test_changes_job_exposes_every_classifier_output(self): + ci = _load_workflow("ci.yml") + changes = ci["jobs"]["changes"] + assert set(changes["outputs"]) == set(SUITES) + assert ci["jobs"]["tests"]["needs"] == "changes" + run_blocks = "\n".join(step.get("run", "") for step in changes["steps"]) + assert "scripts/ci/classify-changes.sh" in run_blocks + + def test_changes_job_only_reads(self): + ci = _load_workflow("ci.yml") + perms = ci["jobs"]["changes"]["permissions"] + assert perms == {"contents": "read", "pull-requests": "read"} + + def test_contracts_job_yields_to_the_full_backend_run(self): + tests = _load_workflow("tests.yml") + job = tests["jobs"]["test-backend-contracts"] + assert "inputs.run_backend_contracts" in job["if"] + assert "!inputs.run_backend" in job["if"] + + def test_backend_pytest_fans_out_over_logical_cpus(self): + tests = _load_workflow("tests.yml") + steps = tests["jobs"]["test-backend"]["steps"] + pytest_step = next(s for s in steps if s.get("name") == "Run pytest") + assert "-n logical" in pytest_step["run"] + assert "-n auto" not in pytest_step["run"] diff --git a/backend/tests/supply_chain/test_strands_agentcore_pairing.py b/backend/tests/supply_chain/test_strands_agentcore_pairing.py new file mode 100644 index 000000000..b21a9b7a5 --- /dev/null +++ b/backend/tests/supply_chain/test_strands_agentcore_pairing.py @@ -0,0 +1,207 @@ +"""Guard rails for the Strands / AgentCore / MCP pin set. + +Two rules that used to live only as prose in the kaizen queue, now asserted: + +1. **`strands-agents` >= 1.56 and `bedrock-agentcore` >= 1.23.1 move together.** + A half-bumped pair imports fine locally and in the suite, then fails on the + AgentCore Runtime as "initialization time exceeded (30s)", which reaches + the user as a 502. Nothing in a unit test would otherwise notice, so the + pairing is checked on the declared pins, on what `uv.lock` resolved, and + on the Lambda images that install `bedrock-agentcore` from their own + hand-maintained requirements files. + +2. **`mcp` stays below 2 because we declared it, not by accident.** + `strands-agents` itself allows `mcp<2.2`. Without a declared constraint the + only thing holding the 1.x line is `idna==3.15` (mcp 2.x -> httpx2 -> idna + >=3.18), so a routine security bump of `idna` would silently cross the + major version under `integrations/mcp_apps.py`'s `ClientSession` patch. + See the comment on `[tool.uv].constraint-dependencies` in pyproject.toml. +""" + +import re +import sys +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +from packaging.requirements import Requirement +from packaging.specifiers import SpecifierSet +from packaging.version import Version + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib # type: ignore[no-redef] + +BACKEND_ROOT = Path(__file__).resolve().parents[2] +PYPROJECT_PATH = BACKEND_ROOT / "pyproject.toml" +UV_LOCK_PATH = BACKEND_ROOT / "uv.lock" + +STRANDS_PAIRING_FLOOR = Version("1.56.0") +AGENTCORE_PAIRING_FLOOR = Version("1.23.1") + +# Lambda images that pip-install from their own requirements file rather +# than from uv.lock. Each says "versions match backend/uv.lock" in its header. +LAMBDA_REQUIREMENTS_FILES = [ + BACKEND_ROOT / "src" / "lambdas" / "scheduled_runs_worker" / "requirements.txt", + BACKEND_ROOT / "src" / "lambdas" / "scheduled_runs_dispatcher" / "requirements.txt", + BACKEND_ROOT / "src" / "apis" / "app_api" / "kb_sync" / "requirements.txt", + BACKEND_ROOT / "src" / "apis" / "app_api" / "kb_migration" / "requirements.txt", +] + +# Packages whose Lambda-side pins must equal the lockfile. boto3/botocore are +# included because bedrock-agentcore 1.23.1 raised its boto floor to 1.43.72: +# an agentcore bump without the boto bump fails `pip install` at image build. +LOCK_MATCHED_PACKAGES = ("bedrock-agentcore", "boto3", "botocore") + + +def _load_toml(path: Path) -> dict: + with open(path, "rb") as f: + return tomllib.load(f) + + +def _all_declared_requirements(pyproject: dict) -> List[Requirement]: + project = pyproject["project"] + reqs = [Requirement(dep) for dep in project.get("dependencies", [])] + for group, deps in project.get("optional-dependencies", {}).items(): + if group == "all": + continue + reqs.extend(Requirement(dep) for dep in deps) + return reqs + + +def _exact_pins(pyproject: dict, name: str) -> List[Tuple[Requirement, Version]]: + pins = [] + for req in _all_declared_requirements(pyproject): + if req.name != name: + continue + specs = list(req.specifier) + assert len(specs) == 1 and specs[0].operator == "==", f"{req} is not a single exact pin" + pins.append((req, Version(specs[0].version))) + return pins + + +def _declared_version(pyproject: dict, name: str) -> Version: + pins = _exact_pins(pyproject, name) + assert pins, f"{name} is not pinned in pyproject.toml" + versions = {v for _, v in pins} + assert len(versions) == 1, ( + f"{name} is pinned at more than one version across pyproject.toml " + f"groups: {sorted(str(r) for r, _ in pins)}. The base package and its " + "extras (e.g. strands-agents and strands-agents[bidi]) must move together." + ) + return versions.pop() + + +def _locked_versions() -> Dict[str, Version]: + lock = _load_toml(UV_LOCK_PATH) + return {pkg["name"]: Version(pkg["version"]) for pkg in lock["package"] if "version" in pkg} + + +def _requirements_pin(path: Path, name: str) -> Optional[Version]: + pattern = re.compile(rf"^\s*{re.escape(name)}\s*==\s*([^\s#;]+)", re.IGNORECASE) + for line in path.read_text().splitlines(): + match = pattern.match(line) + if match: + return Version(match.group(1)) + return None + + +def _paired(strands: Version, agentcore: Version) -> bool: + return (strands >= STRANDS_PAIRING_FLOOR) == (agentcore >= AGENTCORE_PAIRING_FLOOR) + + +# --------------------------------------------------------------------------- +# Guard rail 1: strands >= 1.56 <=> bedrock-agentcore >= 1.23.1 +# --------------------------------------------------------------------------- + + +def test_declared_strands_and_agentcore_pins_are_paired(): + pyproject = _load_toml(PYPROJECT_PATH) + strands = _declared_version(pyproject, "strands-agents") + agentcore = _declared_version(pyproject, "bedrock-agentcore") + + assert _paired(strands, agentcore), ( + f"strands-agents=={strands} with bedrock-agentcore=={agentcore} breaks the " + f"pairing: strands >= {STRANDS_PAIRING_FLOOR} requires bedrock-agentcore >= " + f"{AGENTCORE_PAIRING_FLOOR} and vice versa. A half-bumped pair presents on " + "the AgentCore Runtime as 'initialization time exceeded (30s)' -> 502." + ) + + +def test_locked_strands_and_agentcore_match_declared_pins(): + pyproject = _load_toml(PYPROJECT_PATH) + locked = _locked_versions() + + for name in ("strands-agents", "bedrock-agentcore"): + declared = _declared_version(pyproject, name) + assert name in locked, f"{name} missing from uv.lock" + assert locked[name] == declared, ( + f"uv.lock resolves {name}=={locked[name]} but pyproject.toml pins " + f"{declared}. Run `uv lock` in backend/." + ) + + assert _paired(locked["strands-agents"], locked["bedrock-agentcore"]) + + +def test_lambda_requirements_match_lock_for_agentcore_and_boto(): + locked = _locked_versions() + mismatches = [] + checked = 0 + + for path in LAMBDA_REQUIREMENTS_FILES: + assert path.exists(), f"Lambda requirements file not found: {path}" + for name in LOCK_MATCHED_PACKAGES: + pinned = _requirements_pin(path, name) + if pinned is None: + continue + checked += 1 + if pinned != locked[name]: + rel = path.relative_to(BACKEND_ROOT) + mismatches.append(f" {rel}: {name}=={pinned} (uv.lock has {locked[name]})") + + assert checked > 0, "No bedrock-agentcore/boto3/botocore pins found in Lambda requirements files" + assert not mismatches, "Lambda requirements drifted from backend/uv.lock:\n" + "\n".join(mismatches) + + +# --------------------------------------------------------------------------- +# Guard rail 2: mcp < 2 is held by a DECLARED constraint +# --------------------------------------------------------------------------- + + +def _declared_mcp_constraint(pyproject: dict) -> SpecifierSet: + constraints = pyproject.get("tool", {}).get("uv", {}).get("constraint-dependencies", []) + specs = [Requirement(c).specifier for c in constraints if Requirement(c).name == "mcp"] + assert specs, ( + "pyproject.toml [tool.uv].constraint-dependencies no longer constrains `mcp`. " + "strands-agents allows mcp<2.2, so nothing but an unrelated idna pin would " + "keep the resolver on mcp 1.x. Restore `mcp<2` or lift it deliberately " + "together with a test pass over integrations/mcp_apps.py." + ) + combined = SpecifierSet() + for spec in specs: + combined &= spec + return combined + + +def test_mcp_below_2_is_a_declared_constraint(): + combined = _declared_mcp_constraint(_load_toml(PYPROJECT_PATH)) + + for probe in ("2.0.0", "2.0.0rc1", "2.1.0", "2.2.0"): + assert not combined.contains(probe, prereleases=True), ( + f"[tool.uv] constraint on mcp ({combined}) admits {probe}; it must exclude every 2.x release." + ) + assert combined.contains("1.30.0"), f"[tool.uv] constraint on mcp ({combined}) excludes the 1.x line entirely" + + +def test_uv_lock_was_resolved_under_the_mcp_constraint(): + lock = _load_toml(UV_LOCK_PATH) + manifest_constraints = lock.get("manifest", {}).get("constraints", []) + mcp_entries = [c for c in manifest_constraints if c.get("name") == "mcp"] + + assert mcp_entries, "uv.lock [manifest] carries no mcp constraint; it was not resolved under pyproject's [tool.uv] constraint" + for entry in mcp_entries: + assert not SpecifierSet(entry["specifier"]).contains("2.0.0", prereleases=True) + + locked = _locked_versions() + assert "mcp" in locked, "mcp missing from uv.lock" + assert locked["mcp"] < Version("2"), f"uv.lock resolved mcp=={locked['mcp']}" diff --git a/backend/tests/test_repair_managed_kb_byte_counters.py b/backend/tests/test_repair_managed_kb_byte_counters.py index 6e376d4be..33016f036 100644 --- a/backend/tests/test_repair_managed_kb_byte_counters.py +++ b/backend/tests/test_repair_managed_kb_byte_counters.py @@ -238,3 +238,265 @@ def test_the_re_anchor_refuses_when_the_counters_moved(self, table): assert result["reanchor"].startswith("refused") assert _counters(table) == (98, 10, 108) + + +# ── --settle-unexplained ───────────────────────────────────────────────────── +#: The production knowledge base this mode was written for, with synthetic ids. +#: A three-document corpus migrated under the old code (reserved as a snapshot, +#: never settled), then two file-source imports the old consumer committed out of +#: reservedBytes they never added to. reservedBytes is short by exactly the +#: imports, which is why the adopt rule refused it. +CORPUS = {"DOC-mig1": 7_759, "DOC-mig2": 144_907, "DOC-mig3": 2_012_035} +IMPORTS = {"DOC-imp1": 140_270, "DOC-imp2": 359_565} +PROD_COUNTERS = (499_835, 1_664_866, 2_164_701) +TRUE_CORPUS = 2_664_536 + + +def _seed_prod_shape(table): + stored, reserved, total = PROD_COUNTERS + _kb(table, storedBytes=stored, reservedBytes=reserved, totalBytes=total) + for document_id, size in CORPUS.items(): + _doc(table, document_id, "complete", size, sourceAdapterKey="google-drive") + for document_id, size in IMPORTS.items(): + _doc( + table, document_id, "complete", size, + sourceAdapterKey="google-drive", byteCapSettled=True, committedBytes=Decimal(size), + retrievableAt="2026-09-26T01:00:00Z", + ) + + +def _settle(*extra): + return _run("--settle-unexplained", "--agent", AGENT, "--quiet-seconds", "0", *extra) + + +def _plan(table, settle=True): + record = table.get_item(Key={"PK": f"AST#{AGENT}", "SK": f"KB#{AGENT}"})["Item"] + return repair.plan_kb(AGENT, record, repair.document_rows(table, AGENT), settle_unexplained=settle) + + +class TestTheProdShape: + def test_the_counters_add_up_to_what_the_report_saw(self): + stored, reserved, total = PROD_COUNTERS + assert stored == sum(IMPORTS.values()) + assert reserved == sum(CORPUS.values()) - sum(IMPORTS.values()) + assert total == stored + reserved == sum(CORPUS.values()) + assert TRUE_CORPUS == sum(CORPUS.values()) + sum(IMPORTS.values()) + + def test_without_the_flag_it_is_left_for_review_with_the_diagnosis(self, table): + _seed_prod_shape(table) + + plan = _plan(table, settle=False) + assert _apply() == 0 + + assert not plan.has_work + assert any("left for review" in note for note in plan.notes) + assert any("explained exactly by 2 imported" in note and "499835 bytes" in note for note in plan.notes) + assert _counters(table) == PROD_COUNTERS + assert "byteCapSettled" not in _row(table, "DOC-mig1") + + def test_settling_it_counts_the_whole_corpus_once(self, table): + _seed_prod_shape(table) + + assert _settle("--apply", "--confirm-prefix", "test") == 0 + + assert _counters(table) == (TRUE_CORPUS, 0, TRUE_CORPUS) + for document_id, size in CORPUS.items(): + assert _row(table, document_id)["byteCapSettled"] is True + assert _row(table, document_id)["committedBytes"] == size + for document_id, size in IMPORTS.items(): + assert _row(table, document_id)["committedBytes"] == size + + def test_a_second_run_finds_nothing_to_do(self, table): + _seed_prod_shape(table) + _settle("--apply", "--confirm-prefix", "test") + + plan = _plan(table) + _settle("--apply", "--confirm-prefix", "test") + _apply() + + assert not plan.has_work + assert plan.notes == [] + assert _counters(table) == (TRUE_CORPUS, 0, TRUE_CORPUS) + + def test_the_reconciler_agrees_with_the_settled_state(self, table): + """The reconciler anchors storedBytes to S3 over the ledger's own members + (committedBytes set, not refunded). After the settle that is all five + documents, so its next pass is a no-op rather than a second correction.""" + from apis.app_api.kb_migration import reconciler + + _seed_prod_shape(table) + before = reconciler.counted_document_ids(AGENT, table=table) + _settle("--apply", "--confirm-prefix", "test") + after = reconciler.counted_document_ids(AGENT, table=table) + + assert before == set(IMPORTS) + assert after == set(IMPORTS) | set(CORPUS) + assert sum({**CORPUS, **IMPORTS}[d] for d in after) == _counters(table)[0] + + def test_a_reconciler_pass_that_changes_nothing_does_not_block_it(self, table): + """Today the reconciler's anchor for this shape is the two imports (its + ledger members), which is what storedBytes already holds, and totalBytes is + already stored + reserved. So a pass between the read and the write writes + the same numbers and the settle still lands.""" + from apis.app_api.kb_migration import reconciler + + _seed_prod_shape(table) + plan = _plan(table) + assert reconciler.refresh_stored_bytes(AGENT, AGENT, sum(IMPORTS.values())) + + result = repair.apply_plan(table, plan, quiet_seconds=0) + + assert result["settle"] == "done" + assert _counters(table) == (TRUE_CORPUS, 0, TRUE_CORPUS) + + def test_a_reconciler_pass_that_moves_a_counter_cancels_all_of_it(self, table): + """All or nothing: the counter condition fails, so none of the three + claims lands either, and a re-run starts from a clean read.""" + from apis.app_api.kb_migration import reconciler + + _seed_prod_shape(table) + plan = _plan(table) + assert reconciler.refresh_stored_bytes(AGENT, AGENT, 500_000) + + result = repair.apply_plan(table, plan, quiet_seconds=0) + + assert result["settle"].startswith("refused") + assert _counters(table) == (500_000, 1_664_866, 2_164_866) + for document_id in CORPUS: + assert "byteCapSettled" not in _row(table, document_id) + + +class TestSettleRefusals: + def test_an_upload_in_flight(self, table): + _seed_prod_shape(table) + _doc(table, "DOC-up", "uploading", 40) + + plan = _plan(table) + _settle("--apply", "--confirm-prefix", "test") + + assert not plan.settle + assert any("in flight" in note for note in plan.notes) + assert _counters(table) == PROD_COUNTERS + + def test_a_delete_between_deleting_and_its_refund(self, table): + """Its bytes are still in storedBytes, but not in a complete-only ledger; + settling now and refunding after would take them off twice.""" + _seed_prod_shape(table) + _doc(table, "DOC-going", "deleting", 10, byteCapSettled=True, committedBytes=Decimal(10)) + + plan = _plan(table) + + assert not plan.settle + assert any("part-way through" in note and "DOC-going" in note for note in plan.notes) + + def test_a_refunded_delete_does_not_block_it(self, table): + _seed_prod_shape(table) + _doc(table, "DOC-gone", "deleting", 10, byteCapSettled=True, committedBytes=Decimal(10), + byteCapRefunded=True) + + assert _plan(table).settle + + def test_a_document_that_changes_in_the_quiet_window(self, table): + """An upload reserves before its DOC# row exists, so a reservation can be + read with no row behind it yet. The re-read after the quiet window is what + catches the row arriving.""" + _seed_prod_shape(table) + plan = _plan(table) + _doc(table, "DOC-late", "uploading", 40) + + result = repair.apply_plan(table, plan, quiet_seconds=0) + + assert result["settle"].startswith("refused: the documents changed") + assert _counters(table) == PROD_COUNTERS + + def test_a_claimed_document_deleted_at_the_last_moment_cancels_all_of_it(self, table): + _seed_prod_shape(table) + plan = _plan(table) + table.update_item( + Key={"PK": f"AST#{AGENT}", "SK": "DOC#DOC-mig2"}, + UpdateExpression="SET #s = :d", + ExpressionAttributeNames={"#s": "status"}, + ExpressionAttributeValues={":d": "deleting"}, + ) + + with pytest.MonkeyPatch.context() as mp: + # Past the quiet-window re-read, so the transaction's own row condition + # is what has to catch it. + mp.setattr(repair, "plan_kb", lambda *a, **k: plan) + result = repair.apply_settle(table, plan, quiet_seconds=0) + + assert result["settle"].startswith("refused") + assert _counters(table) == PROD_COUNTERS + assert "byteCapSettled" not in _row(table, "DOC-mig1") + + def test_it_requires_naming_the_knowledge_base(self, table): + _seed_prod_shape(table) + + assert _run("--settle-unexplained", "--apply", "--confirm-prefix", "test") == 2 + + assert _counters(table) == PROD_COUNTERS + + def test_it_previews_without_apply(self, table): + _seed_prod_shape(table) + + assert _settle() == 0 + + assert _counters(table) == PROD_COUNTERS + assert "byteCapSettled" not in _row(table, "DOC-mig1") + + def test_a_knowledge_base_not_named_is_not_touched(self, table): + _seed_prod_shape(table) + other = "ast-repair0002" + _kb(table, agent=other, storedBytes=0, reservedBytes=42, totalBytes=42) + + _settle("--apply", "--confirm-prefix", "test") + + assert _counters(table, agent=other) == (0, 42, 42) + + def test_a_consistent_knowledge_base_is_left_alone(self, table): + """The flag only widens what counts as repairable; a knowledge base the + ordinary rules handle is handled by them.""" + _kb(table, storedBytes=0, reservedBytes=300, totalBytes=300) + _doc(table, "DOC-1", "complete", 300) + + plan = _plan(table) + + assert not plan.settle + assert plan.adopt == [("DOC-1", 300)] + + +class TestOtherShapesTheSettleRepairs: + def test_imports_on_a_knowledge_base_with_nothing_reserved_went_negative(self, table): + """A born-managed or already-adopted knowledge base: each pre-fix import + committed out of an empty reservedBytes.""" + _kb(table, storedBytes=500, reservedBytes=-500, totalBytes=0) + _doc(table, "DOC-imp", "complete", 500, sourceAdapterKey="google-drive", + byteCapSettled=True, committedBytes=Decimal(500)) + + plan = _plan(table) + _settle("--apply", "--confirm-prefix", "test") + + assert any("explained exactly by 1 imported" in note for note in plan.notes) + assert _counters(table) == (500, 0, 500) + + def test_a_leaked_reservation_on_a_quiet_knowledge_base(self, table): + """The dev shape the ordinary run reports and leaves: 42 bytes reserved, + nothing in flight. The quiet-window re-read is what makes zeroing it safe.""" + _kb(table, storedBytes=0, reservedBytes=42, totalBytes=42) + _doc(table, "DOC-fail", "failed", 42, byteCapSettled=True) + + _settle("--apply", "--confirm-prefix", "test") + + assert _counters(table) == (0, 0, 0) + + def test_a_pre_fix_row_without_committed_bytes_is_backfilled_in_the_same_transaction(self, table): + _kb(table, storedBytes=150, reservedBytes=100, totalBytes=200) + _doc(table, "DOC-mig", "complete", 250) + _doc(table, "DOC-imp", "complete", 150, sourceAdapterKey="http", byteCapSettled=True, + retrievableAt="2026-09-20T00:00:00Z") + + _settle("--apply", "--confirm-prefix", "test") + + assert _counters(table) == (400, 0, 400) + assert _row(table, "DOC-imp")["committedBytes"] == 150 + assert _row(table, "DOC-mig")["committedBytes"] == 250 diff --git a/backend/tests/test_seed_e2e_role.py b/backend/tests/test_seed_e2e_role.py new file mode 100644 index 000000000..dee79f29d --- /dev/null +++ b/backend/tests/test_seed_e2e_role.py @@ -0,0 +1,131 @@ +"""``scripts/seed_e2e_role.py`` — the RBAC role the nightly E2E user runs under. + +Pins what the turn-path smoke matrix depends on: the role maps to the Cognito +group ``seed-e2e-users.sh`` puts the user in, grants exactly the two tools the +matrix exercises plus every model, mirrors each grant into +``effective_permissions`` (the half the access checks actually read), and is +idempotent — a second run updates in place rather than failing on the +existing row. +""" + +import os +import sys + +import boto3 +import pytest +from moto import mock_aws + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts")) +from seed_e2e_role import ( # noqa: E402 + DEFAULT_GROUP, + DEFAULT_TOOLS, + ROLE_ID, + build_role, + parse_args, + upsert, +) + +TABLE_NAME = "test-app-roles" +REGION = "us-east-1" + + +@pytest.fixture +def repository(): + """A real AppRoleRepository over a moto table with the app-roles schema.""" + with mock_aws(): + dynamodb = boto3.resource("dynamodb", region_name=REGION) + table = dynamodb.create_table( + TableName=TABLE_NAME, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + {"AttributeName": "GSI1PK", "AttributeType": "S"}, + {"AttributeName": "GSI1SK", "AttributeType": "S"}, + ], + GlobalSecondaryIndexes=[ + { + "IndexName": "JwtRoleMappingIndex", + "KeySchema": [ + {"AttributeName": "GSI1PK", "KeyType": "HASH"}, + {"AttributeName": "GSI1SK", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + }, + ], + BillingMode="PAY_PER_REQUEST", + ) + table.meta.client.get_waiter("table_exists").wait(TableName=TABLE_NAME) + os.environ["AWS_DEFAULT_REGION"] = REGION + from apis.shared.rbac.repository import AppRoleRepository + + yield AppRoleRepository(table_name=TABLE_NAME), table + + +class TestBuildRole: + def test_defaults_match_what_the_smoke_matrix_needs(self): + role = build_role(group=DEFAULT_GROUP, tools=DEFAULT_TOOLS, models=["*"]) + assert role.role_id == ROLE_ID + assert role.jwt_role_mappings == ["e2e-users"] + assert role.granted_tools == ["ask_user_question", "calculator"] + assert role.granted_models == ["*"] + assert role.granted_skills == [] + assert role.enabled is True + assert role.is_system_role is False + + def test_effective_permissions_mirror_the_grants(self): + # The access checks read effective_permissions; a role with only the + # granted_* half grants nothing (CLAUDE.md, RBAC). + role = build_role(group="g", tools=["b_tool", "a_tool", "a_tool"], models=["m2", "m1"]) + assert role.effective_permissions.tools == role.granted_tools == ["a_tool", "b_tool"] + assert role.effective_permissions.models == role.granted_models == ["m1", "m2"] + assert role.effective_permissions.skills == [] + assert role.effective_permissions.quota_tier is None + + def test_cli_defaults(self): + args = parse_args([]) + assert args.group == DEFAULT_GROUP + assert args.tools.split(",") == DEFAULT_TOOLS + assert args.models == "*" + assert args.dry_run is False + + +class TestUpsert: + @pytest.mark.asyncio + async def test_first_run_creates_every_row_the_resolver_reads(self, repository): + repo, table = repository + role = build_role(group=DEFAULT_GROUP, tools=DEFAULT_TOOLS, models=["*"]) + + assert await upsert(role, repo) == "created" + + items = table.query(KeyConditionExpression=boto3.dynamodb.conditions.Key("PK").eq(f"ROLE#{ROLE_ID}"))["Items"] + sks = sorted(i["SK"] for i in items) + assert sks == [ + "DEFINITION", + "JWT_MAPPING#e2e-users", + "MODEL_GRANT#*", + "TOOL_GRANT#ask_user_question", + "TOOL_GRANT#calculator", + ] + definition = next(i for i in items if i["SK"] == "DEFINITION") + assert definition["effectivePermissions"]["tools"] == ["ask_user_question", "calculator"] + assert definition["effectivePermissions"]["models"] == ["*"] + # The JWT mapping row is what the login-time resolver's GSI query hits. + assert await repo.get_roles_for_jwt_role("e2e-users") == [ROLE_ID] + + @pytest.mark.asyncio + async def test_second_run_updates_in_place(self, repository): + repo, table = repository + assert await upsert(build_role(group=DEFAULT_GROUP, tools=["calculator"], models=["*"]), repo) == "created" + assert await upsert(build_role(group=DEFAULT_GROUP, tools=DEFAULT_TOOLS, models=["*"]), repo) == "updated" + + items = table.query(KeyConditionExpression=boto3.dynamodb.conditions.Key("PK").eq(f"ROLE#{ROLE_ID}"))["Items"] + tool_grants = sorted(i["SK"] for i in items if i["SK"].startswith("TOOL_GRANT#")) + assert tool_grants == ["TOOL_GRANT#ask_user_question", "TOOL_GRANT#calculator"] + stored = await repo.get_role(ROLE_ID) + assert stored is not None + assert stored.granted_tools == ["ask_user_question", "calculator"] + assert stored.created_at # preserved by update_role diff --git a/backend/uv.lock b/backend/uv.lock index 967c3d633..b94d624cd 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -15,7 +15,7 @@ constraints = [{ name = "mcp", specifier = "<2" }] [[package]] name = "agentcore-stack" -version = "1.25.1" +version = "1.26.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, @@ -104,9 +104,9 @@ requires-dist = [ { name = "aws-bedrock-token-generator", marker = "extra == 'agentcore'", specifier = "==1.1.0" }, { name = "aws-opentelemetry-distro", marker = "extra == 'agentcore'", specifier = "==0.19.0" }, { name = "beautifulsoup4", specifier = "==4.13.5" }, - { name = "bedrock-agentcore", marker = "extra == 'agentcore'", specifier = "==1.21.0" }, + { name = "bedrock-agentcore", marker = "extra == 'agentcore'", specifier = "==1.24.0" }, { name = "black", marker = "extra == 'dev'", specifier = "==26.3.1" }, - { name = "boto3", specifier = "==1.43.68" }, + { name = "boto3", specifier = "==1.43.103" }, { name = "cachetools", specifier = "==6.2.4" }, { name = "cryptography", specifier = "==50.0.1" }, { name = "fastapi", specifier = "==0.136.1" }, @@ -133,8 +133,8 @@ requires-dist = [ { name = "ruff", marker = "extra == 'dev'", specifier = "==0.15.12" }, { name = "soupsieve", specifier = "==2.9.0" }, { name = "starlette", specifier = "==1.3.1" }, - { name = "strands-agents", marker = "extra == 'agentcore'", specifier = "==1.55.0" }, - { name = "strands-agents", extras = ["bidi"], marker = "extra == 'bidi'", specifier = "==1.55.0" }, + { name = "strands-agents", marker = "extra == 'agentcore'", specifier = "==1.57.2" }, + { name = "strands-agents", extras = ["bidi"], marker = "extra == 'bidi'", specifier = "==1.57.2" }, { name = "strands-agents-tools", marker = "extra == 'agentcore'", specifier = "==0.8.8" }, { name = "tiktoken", marker = "extra == 'dev'", specifier = "==0.12.0" }, { name = "trafilatura", specifier = "==2.0.0" }, @@ -516,43 +516,43 @@ wheels = [ [[package]] name = "awscrt" -version = "0.32.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/4d/4d/c2aece4af7b5537c855548f53ee077d01216a1a4adbf0fd24f23dbac52bf/awscrt-0.32.0.tar.gz", hash = "sha256:92e749fce6c61da8db1af0baa6b7e96f7acf8a5574760b3d7880d190cedee8a0", size = 36832208, upload-time = "2026-03-27T01:19:18.147Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/93/9f/ded80b31ea4e30be66cf8dd35ecb0bce3e5a2fee39542bd877d7c2f2c2e4/awscrt-0.32.0-cp310-cp310-macosx_10_15_universal2.whl", hash = "sha256:211cee3a5335bbc96405afdbdebf9c17bf16f3a299e9d8b661d5c904634ade43", size = 3391016, upload-time = "2026-03-27T01:17:58.557Z" }, - { url = "https://files.pythonhosted.org/packages/a3/8e/81ad2ad2c90faba849f106f4ee8c2f00741b1137225e965ba2be4b36fd9d/awscrt-0.32.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:693f40a004270dfc7e0f92f7d36ef5fb8e40de686fa87a0603ade74a479bbf40", size = 3930641, upload-time = "2026-03-27T01:18:01.454Z" }, - { url = "https://files.pythonhosted.org/packages/b8/09/193d7da3033e0c2f283f0c1a017a6e0918c91e0f57c3e4a18314ba7c97ad/awscrt-0.32.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3b3014bd320ce95a3698f8b00312a06accc6222505ace4e6cbb524aa10ab8654", size = 4217980, upload-time = "2026-03-27T01:18:02.909Z" }, - { url = "https://files.pythonhosted.org/packages/3c/fa/63fc0f4573bf29135fde0b5a168c8d2c003344a342e7a342ab6e13ded795/awscrt-0.32.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:43ea8a3bb994259b1c2cb6fc400374cc6c84a2b235eada3370d5f012420427dd", size = 3861043, upload-time = "2026-03-27T01:18:04.424Z" }, - { url = "https://files.pythonhosted.org/packages/95/57/e64e365c9112cbe4b04982effb22d62d57728513555c9516879598ebfea9/awscrt-0.32.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:7cec6175c9616840a599050a0e6157846b509e7b3a7f2ab9f8e9a64bcb342e68", size = 4101797, upload-time = "2026-03-27T01:18:05.758Z" }, - { url = "https://files.pythonhosted.org/packages/7c/2f/2a3decaa3111c47fbec0a051767838741b52c61e37c215cb9c905d74ed5b/awscrt-0.32.0-cp310-cp310-win32.whl", hash = "sha256:0a585e9089140f55a8452c2f1aefe17712d39eff5413c1c0c7aa393e749fd5b7", size = 4041085, upload-time = "2026-03-27T01:18:07.28Z" }, - { url = "https://files.pythonhosted.org/packages/b3/63/72f38c224bd38e383b1e08713c655c0756a41c6d0e7e59616e7d47b6ec35/awscrt-0.32.0-cp310-cp310-win_amd64.whl", hash = "sha256:d4423dbdb75ed37bbd9697a672bfd75a02a7354ac9b818fa2d38059726171928", size = 4204956, upload-time = "2026-03-27T01:18:08.947Z" }, - { url = "https://files.pythonhosted.org/packages/9f/74/1e63af11b71ca90e6bcc70affea6400078d4cf6605f0593fe0a91a1daeb5/awscrt-0.32.0-cp311-abi3-macosx_10_15_universal2.whl", hash = "sha256:4ca7040b279cf6014c06de93be7a29a164c9c92469eb79c70143853873e81949", size = 3391226, upload-time = "2026-03-27T01:18:10.231Z" }, - { url = "https://files.pythonhosted.org/packages/43/62/d1383a31d32b9963a3a646d926f77a46d88cee1a86536186ad0ac0c44aea/awscrt-0.32.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:15a96559710e559bc4131b7af55b93c0c79505d4b9c4c4511b3b825bbb4f82a9", size = 3889898, upload-time = "2026-03-27T01:18:11.484Z" }, - { url = "https://files.pythonhosted.org/packages/2c/92/37c25e283ed4ebe21117fb183c1dbcd947d72fb770f05f9f1e8c2b63541e/awscrt-0.32.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:11a8cf1902c35ad784ceedbe2d5b44956a34bb2b7c3c818511ce93bdb21bc386", size = 4178874, upload-time = "2026-03-27T01:18:13.128Z" }, - { url = "https://files.pythonhosted.org/packages/eb/23/6dbefd6efbe0914c73f633ea6702aa4701425337f4e0c1059ec99aedde23/awscrt-0.32.0-cp311-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:f1eea556144c2999e105966bbe97a0460d10d656e331bc2c875f15ece3315c3b", size = 3800811, upload-time = "2026-03-27T01:18:14.848Z" }, - { url = "https://files.pythonhosted.org/packages/5d/2e/0af9a203fea97504e0bd11261b12422ff555745699c176dee1767f09c9f6/awscrt-0.32.0-cp311-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:d7ef6f0e728c1b7a0a4b771d4c677a00bc1bf377b8b2dd59a1dd626b920efc3b", size = 4039356, upload-time = "2026-03-27T01:18:16.684Z" }, - { url = "https://files.pythonhosted.org/packages/c4/8a/6371dc9dc7b4a8dfafe7ed1b4f30500cc22e239413a6fdcaab72f8b80b8e/awscrt-0.32.0-cp311-abi3-win32.whl", hash = "sha256:8cee2fea902452a36f67f9d79e6eb406d4359854dad6df439b3c671f07059763", size = 4039614, upload-time = "2026-03-27T01:18:17.979Z" }, - { url = "https://files.pythonhosted.org/packages/75/c2/0bd9346f22ced5f11dac7039876ea2824cc3e268b6b681e2a8a29b1e8701/awscrt-0.32.0-cp311-abi3-win_amd64.whl", hash = "sha256:8213ee3b3c1adb5364a48a87420cde4426688f0438a88f6381595586be7ffc17", size = 4201989, upload-time = "2026-03-27T01:18:19.332Z" }, - { url = "https://files.pythonhosted.org/packages/d3/67/2093978f8496dad5e90d77c3f59f07d3f040e32eda60f3592f1b45d48d65/awscrt-0.32.0-cp313-abi3-macosx_10_15_universal2.whl", hash = "sha256:d1efd89302eeee14878ca2067de6525c85d4973cd4473bc537e8807ecca660d3", size = 3390414, upload-time = "2026-03-27T01:18:21.006Z" }, - { url = "https://files.pythonhosted.org/packages/1f/4d/c2f8a732fef457ce58e7d798b2775deea16a3a840d8a9dd43bb21b80ec63/awscrt-0.32.0-cp313-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:18e36af2cfec50b0a2d270921217245c36f6722b9aae756394ca050dee535883", size = 3881094, upload-time = "2026-03-27T01:18:22.602Z" }, - { url = "https://files.pythonhosted.org/packages/b9/be/3cd7ad30fbc65eed95c1df2d50f9f1facb82335137f1842ec4e2152d3a51/awscrt-0.32.0-cp313-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a94d7c85486908adf07693519b3a8ec7c61b86cc0901fed266ff2239babef6ce", size = 4172552, upload-time = "2026-03-27T01:18:23.857Z" }, - { url = "https://files.pythonhosted.org/packages/eb/af/e299192ae380cb688ff505aa9145a9b4e9c31bf12275d67074bfd6aff899/awscrt-0.32.0-cp313-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:68e36b744ed8089be80a2f7c8ed9bd46573f00870d1429707c1c847f3dc99a6f", size = 3791182, upload-time = "2026-03-27T01:18:25.144Z" }, - { url = "https://files.pythonhosted.org/packages/96/ab/ffa769df6417720a4f9bddd9f8cc077f671d0a970d88d2c80e0e06eb0890/awscrt-0.32.0-cp313-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:b1a9f3952f382feee264614e727d78e7fa12bad138a025e832affc84780fdc89", size = 4035286, upload-time = "2026-03-27T01:18:26.444Z" }, - { url = "https://files.pythonhosted.org/packages/50/f2/cf80de14e9735750af9c4d11173150da997b8333ca6e68dc4850b6768a7c/awscrt-0.32.0-cp313-abi3-win32.whl", hash = "sha256:56c418ac23102e34ad95ea68ad5527ce534b7c79bb3ec3b908b647d90ee3ce41", size = 4034631, upload-time = "2026-03-27T01:18:28.247Z" }, - { url = "https://files.pythonhosted.org/packages/96/d6/d95df41f0e3a9434a750a33fbe465c8638212a46ec469f301511fd4791f7/awscrt-0.32.0-cp313-abi3-win_amd64.whl", hash = "sha256:7a3a464b0c23d1c2cca23b210035da203b1ead7cafcda2cebb87af3de20cc2b2", size = 4197293, upload-time = "2026-03-27T01:18:29.666Z" }, - { url = "https://files.pythonhosted.org/packages/9a/aa/5d85dc363c2269a205d33305e13b07298beaf582f653f10f6fc70531dc29/awscrt-0.32.0-cp313-cp313t-macosx_10_15_universal2.whl", hash = "sha256:f61cc36f645444d4e27cdf2c8fc5d3fdca77de35f341ad0e6c65f6c097ef5afd", size = 3401086, upload-time = "2026-03-27T01:18:31.388Z" }, - { url = "https://files.pythonhosted.org/packages/52/9e/fac5cab0cd7a94a4978daade9ef6d77c3b1037f470476d16ae822b54c97e/awscrt-0.32.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c51510e2171a3cab33612b98333a4baca391ed76a1185e9f6ec5433196e646af", size = 4006884, upload-time = "2026-03-27T01:18:32.696Z" }, - { url = "https://files.pythonhosted.org/packages/e1/cc/abd847148100a62616abaa5bdf9731686646e2a6f73e44bb63a718d6fd1e/awscrt-0.32.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f22723fc84ec31ed13591105083bea57fcbb7a9b20600dcfda8b5c28fa8047f8", size = 4293086, upload-time = "2026-03-27T01:18:34.092Z" }, - { url = "https://files.pythonhosted.org/packages/3d/8e/05572eaab9ff7a0a40f17c084d61389602c5018c7f4b7bb7ad9e58b4bea9/awscrt-0.32.0-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:4bc6492b7622cbad46d65492fb12239d946e890797c5da6c30a878a04c694623", size = 3934220, upload-time = "2026-03-27T01:18:35.438Z" }, - { url = "https://files.pythonhosted.org/packages/8f/4d/6e2dc94c69b32aadf037e5057f550d8e1bf3271573e5e95cc8934499a579/awscrt-0.32.0-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:dd6df896ae0650977de05c8e83fc5f3f7472b4d8de7744560526c13a63da8fd0", size = 4168581, upload-time = "2026-03-27T01:18:36.885Z" }, - { url = "https://files.pythonhosted.org/packages/42/4d/099e4fc39839ff130716713401d36ed9fa8b78feb5dcbf273e1e1aa71dd2/awscrt-0.32.0-cp313-cp313t-win32.whl", hash = "sha256:5fb05ab256b90c2d39386702d20419159b605a1f0e95d0fade715ccc9a76856a", size = 4091134, upload-time = "2026-03-27T01:18:38.532Z" }, - { url = "https://files.pythonhosted.org/packages/c6/a6/e0c63b8b73424f91a9fab52f100f412864ddd47e01ce84a6aeae35a12b7b/awscrt-0.32.0-cp313-cp313t-win_amd64.whl", hash = "sha256:c1c69543cdeab10f7fcbd3f238996ee1ed73fb8f88dd9701fffc872d73bd256d", size = 4247510, upload-time = "2026-03-27T01:18:40.02Z" }, - { url = "https://files.pythonhosted.org/packages/b7/64/e7d5eac410e305b5d62da268d4c486dd003b065b3119031679a6cf242861/awscrt-0.32.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:f82a7969c025875fa419fc6349c8013bc88359ce264cb6c2399d03f42fbae0e3", size = 3401104, upload-time = "2026-03-27T01:18:41.354Z" }, - { url = "https://files.pythonhosted.org/packages/27/36/20d11e4b2a32337b712f1aa683a7d4bed777d1bfdff5d7803c7b952556c2/awscrt-0.32.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:877e6061448abc91b1926f3f8c73808ce2a170a404065ba185a407fbfd2eb8e7", size = 4007601, upload-time = "2026-03-27T01:18:43.005Z" }, - { url = "https://files.pythonhosted.org/packages/47/b6/d1d21aaa4c3affc82ff3b4ae75bad10d80234ea01d8d239a578efa23646b/awscrt-0.32.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b83427cf90f1606a34dbbe29b1544f945899b3abdb8608a57f004c6f459fd1fd", size = 4293839, upload-time = "2026-03-27T01:18:44.628Z" }, - { url = "https://files.pythonhosted.org/packages/97/2b/bb17205da426d175991ca9f9a5873c8c50a0620ec12c299ee3d80941552a/awscrt-0.32.0-cp314-cp314t-win32.whl", hash = "sha256:de44db7677361a05a1cdce9a1c29b6628542094599cb33105b99b97b4b9580ea", size = 4171775, upload-time = "2026-03-27T01:18:46.184Z" }, - { url = "https://files.pythonhosted.org/packages/39/2c/65ac451a08b57d9d66c8ffc2cbdf3c32e7da84d0a1887bdf6a3da4877585/awscrt-0.32.0-cp314-cp314t-win_amd64.whl", hash = "sha256:7b9eb088e4e17539d3c5ec8f40f04363fabb807f9d509653d2443056d22b3506", size = 4347754, upload-time = "2026-03-27T01:18:47.641Z" }, +version = "0.32.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/cb/980fe60c4209af71d036276217f8b9f372f958e290c15d2849a3de4dcd23/awscrt-0.32.2.tar.gz", hash = "sha256:a4f48805e8a66237923f03b7b692d213994cff42d1ff08125d1d60c74fcaf872", size = 36862073, upload-time = "2026-04-24T22:59:55.835Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e6/93/08c8dde6e10c2aa5e50c08730e7913fd68cc861536260115bdf109f20bb8/awscrt-0.32.2-cp310-cp310-macosx_10_15_universal2.whl", hash = "sha256:a9ed98e20ca6fcad8ef32e3c6779aaa3526a5ff3f0aa99d59e0deeff59640375", size = 3405893, upload-time = "2026-04-24T22:58:34.276Z" }, + { url = "https://files.pythonhosted.org/packages/b8/d8/f838299003df9c77b9d582411f009bf2e5bc07fc68566685b434de249755/awscrt-0.32.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acb707df280079d21d20ad1735b38a80d4939133cbaecfc2e4927dd8fc0190bc", size = 3946562, upload-time = "2026-04-24T22:58:37.467Z" }, + { url = "https://files.pythonhosted.org/packages/34/95/a9d9ff694e15550a1fa2f83ac9b3b50cc5f809d66dacc57af4d84cc01c7f/awscrt-0.32.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ab06479bcdcb42b2956f59c0e4138049e5b44c885b7584ea05e39cc4d71b1f99", size = 4236332, upload-time = "2026-04-24T22:58:38.789Z" }, + { url = "https://files.pythonhosted.org/packages/24/bd/92fcf514966e7a139b146282396608493b99e25f745b332075ebe18e129b/awscrt-0.32.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:2792fc2877335673058d0b4e8249ef73dc36b22689fda939506aea6eceb42054", size = 3883087, upload-time = "2026-04-24T22:58:40.292Z" }, + { url = "https://files.pythonhosted.org/packages/5c/7d/cf7ec60d4997ee966ca003a3c7bfc556ee34db10f230f03e5608edf022ca/awscrt-0.32.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:0d2e8a13063a3d9b61eee0e2885d133a75b38de3600f6b62437d8559f7d664e3", size = 4118974, upload-time = "2026-04-24T22:58:41.913Z" }, + { url = "https://files.pythonhosted.org/packages/65/bb/769f8923d7d7986e762898fa98a9d16130681a47da2ff668e6fb07bae0b4/awscrt-0.32.2-cp310-cp310-win32.whl", hash = "sha256:4e975223f3af4faa581c733f0fdd316514b987c42ce85ef3bcd6d9c02eb48f77", size = 4067395, upload-time = "2026-04-24T22:58:43.308Z" }, + { url = "https://files.pythonhosted.org/packages/83/42/17ab1d701f2628adf7b1b993cae9993111ed760aaf08f5772c2bf37c6b7a/awscrt-0.32.2-cp310-cp310-win_amd64.whl", hash = "sha256:a13c0a555bd930c829c72e6b2b2df70442b3b414037fd488984495b5beff5ddc", size = 4224532, upload-time = "2026-04-24T22:58:45.039Z" }, + { url = "https://files.pythonhosted.org/packages/03/99/a6c712a2f638c766b0879da0eacd9fe5695c64deb89c0357bcc4f1f4df9b/awscrt-0.32.2-cp311-abi3-macosx_10_15_universal2.whl", hash = "sha256:32785f54d0786e07b6491b51f9c2f75ea9e17decd39bb6b66fdc60cd871a49ef", size = 3406247, upload-time = "2026-04-24T22:58:46.268Z" }, + { url = "https://files.pythonhosted.org/packages/a7/03/665c81f3b9d3c56fcdfb8f353c22501bc538d192c431cfaaf307f867d404/awscrt-0.32.2-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f8a586c52f41ff14c2f1b8afeb764e231ad3d66acfd42a6b9fb6c8afd8da8fe2", size = 3906890, upload-time = "2026-04-24T22:58:47.96Z" }, + { url = "https://files.pythonhosted.org/packages/31/ad/5db0691a0c72d55a17c03c47149cf15d4602b83b470355c322c9a7f115e8/awscrt-0.32.2-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3ef1664588d35dcad2115377120667e689cd7a517da52a481373c9536811ed96", size = 4196631, upload-time = "2026-04-24T22:58:49.244Z" }, + { url = "https://files.pythonhosted.org/packages/9a/66/63a4654b2996158f33e88d74d79178a5812d94a7e0d6c94ec475115dff18/awscrt-0.32.2-cp311-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:7ed7c209136650fe25659436bb4150e5af6eb43d71a0bf294f0bf414428736ee", size = 3818090, upload-time = "2026-04-24T22:58:50.657Z" }, + { url = "https://files.pythonhosted.org/packages/cd/26/b8fee227918465830a0bbba48f54ebf0a5029c3a7d11d4fa973838a79262/awscrt-0.32.2-cp311-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:cfd437122d5a2ec7eb9fffaf2cd8b96543d4a0d7e906b9515b79672005a1607a", size = 4056453, upload-time = "2026-04-24T22:58:52.373Z" }, + { url = "https://files.pythonhosted.org/packages/e3/26/f5b9e9677bf90d1840d4db1513ba92e73601ef82e61a16e747a83493d35c/awscrt-0.32.2-cp311-abi3-win32.whl", hash = "sha256:5197d1e4e780d755632f79f8d32a09a30a9101cccb51ca1694fe25c711b2e801", size = 4066027, upload-time = "2026-04-24T22:58:53.752Z" }, + { url = "https://files.pythonhosted.org/packages/57/0b/fd1798551ecdc8a28d61ecdeda248f99faa3457f83aefa10ab797f466889/awscrt-0.32.2-cp311-abi3-win_amd64.whl", hash = "sha256:80420aa19c074a4c0335f2bd0e4aee3381fa452328d937795a1e0c779f0c052d", size = 4221984, upload-time = "2026-04-24T22:58:55.115Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a3/075e29b39945f398adfb5d5ee4d533e00d37c0f5c68d79b250a1bdb087ba/awscrt-0.32.2-cp313-abi3-macosx_10_15_universal2.whl", hash = "sha256:d2f7aee3bce261ab1ceba1fac404de4d496aa866237161d4257cef92bff9d828", size = 3404901, upload-time = "2026-04-24T22:58:56.492Z" }, + { url = "https://files.pythonhosted.org/packages/e3/41/1c4783b32bf4ec7383156787570ea1221c95c037c2c0f11cbc9e9529ba48/awscrt-0.32.2-cp313-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ff0fff9c2b613d7fabc298b0fd81f0d7056353f3d20271a852a719c5b2f7ccf4", size = 3897627, upload-time = "2026-04-24T22:58:58.144Z" }, + { url = "https://files.pythonhosted.org/packages/af/2d/5736128847ff4a877d50720ea7a48d7e50a56b78741919e4b6ffabffb1ad/awscrt-0.32.2-cp313-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:79bb11d5d1dfdcfd867aac4a026bee11afbd2154279e12b66588442d8c14bdf7", size = 4189579, upload-time = "2026-04-24T22:58:59.56Z" }, + { url = "https://files.pythonhosted.org/packages/90/f5/064b23d4c927e9b63725ee60c88edb75a1e8f5fd1e97d56d4d6a63fe954b/awscrt-0.32.2-cp313-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:47330f421948207a122092042f235cf82d48fa145c446ff4db12cc8cd3a418b6", size = 3809647, upload-time = "2026-04-24T22:59:00.884Z" }, + { url = "https://files.pythonhosted.org/packages/f5/36/f14ea531cccb6ff85c4d50c9e79e61030675520a393b4af23685d24ea018/awscrt-0.32.2-cp313-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:5ec4d8200eb0158b60e35fbb65faa96ef1eb7763f6ce1192827886ee24c6df99", size = 4051532, upload-time = "2026-04-24T22:59:02.593Z" }, + { url = "https://files.pythonhosted.org/packages/64/aa/b12e721c8148a1bbc58a22cbd204d88d0a6263331049d40c057f9dcd3e11/awscrt-0.32.2-cp313-abi3-win32.whl", hash = "sha256:a81f30a501d2eb6ba52c769cd6ecb3f7005512fba4a533211dc717e1115b0d94", size = 4062484, upload-time = "2026-04-24T22:59:04.251Z" }, + { url = "https://files.pythonhosted.org/packages/5c/26/9f5f23647465a4fe1b28afce334e54a4264e23b69365024c28955f0c4119/awscrt-0.32.2-cp313-abi3-win_amd64.whl", hash = "sha256:8d731edda20dce5afc15a04731d91136a31779a244672d1f0a292a8b04aa0fd3", size = 4220425, upload-time = "2026-04-24T22:59:05.627Z" }, + { url = "https://files.pythonhosted.org/packages/dd/76/18077410c1d7b524a7a7486550bc969218f6575288f66e285157dc78e143/awscrt-0.32.2-cp313-cp313t-macosx_10_15_universal2.whl", hash = "sha256:f2a085d0dd4eef974f2ae5467ae3717d2fc08dd8cd508c4fd7a5acb658c68616", size = 3414715, upload-time = "2026-04-24T22:59:07.01Z" }, + { url = "https://files.pythonhosted.org/packages/d3/12/465d72ea6afffc7829f7f48f408a707d4dab9e3f2b694f4e0e63e011478a/awscrt-0.32.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d8a4e52f7312e5e435461119aa903f6424e9996d93a040101fb1eb7b9c4e58dc", size = 4028634, upload-time = "2026-04-24T22:59:08.303Z" }, + { url = "https://files.pythonhosted.org/packages/e0/ee/2e9d3716cf6a24f30b85af24691d473c913c119dc996bcd8c9b2b3fe2c17/awscrt-0.32.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6cb3c858e96ba023de691a3b6478bed9fb59085433042dff78c42e59eed19cf2", size = 4311846, upload-time = "2026-04-24T22:59:10.02Z" }, + { url = "https://files.pythonhosted.org/packages/d0/ce/aa08aad92e48929cfe243ed7da5cf039fbb137c391e84af79e88c848a37a/awscrt-0.32.2-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:7f92b8f40a104a2a87ea5f428b3799220666ec1450b3a90665867d3715749e91", size = 3952242, upload-time = "2026-04-24T22:59:11.632Z" }, + { url = "https://files.pythonhosted.org/packages/42/e3/09ad98aa7dae9cd3ad578c7721b64e9da03f9a5ed444e518c63e82db05a9/awscrt-0.32.2-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:9fd2dbca02f4e22fb5c02d1505327e6e6e9320dbe8ca80fc033cbbb29ed8631a", size = 4187982, upload-time = "2026-04-24T22:59:14.477Z" }, + { url = "https://files.pythonhosted.org/packages/26/be/897b1ad519d83a837d721f4e8a87d98e6f36e5b985fa697f3ffed512a8eb/awscrt-0.32.2-cp313-cp313t-win32.whl", hash = "sha256:023a2f4595804a0f1d61ab49b64dda5612be9bfbe9b13759331e8e31658dda3f", size = 4111958, upload-time = "2026-04-24T22:59:15.857Z" }, + { url = "https://files.pythonhosted.org/packages/71/54/6cca298e8acb6769c8078b39bedbc507f17a3f7fe6ea768d8256d584d4ed/awscrt-0.32.2-cp313-cp313t-win_amd64.whl", hash = "sha256:a2f513f0fb3aafdf7cdf29d7f6f0c46bf4cc7880380c86e88635b7818565e76d", size = 4271679, upload-time = "2026-04-24T22:59:17.425Z" }, + { url = "https://files.pythonhosted.org/packages/c7/09/51fca333d42b53ddb04881f53e3cc0f4462872ac81426fbb34f5d0b1d1fb/awscrt-0.32.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:7404cb551046bbe7cd454ea75f88b4a1b26f018d1b9bea83dbe46c174789d835", size = 3414716, upload-time = "2026-04-24T22:59:18.918Z" }, + { url = "https://files.pythonhosted.org/packages/39/3a/0ae52a8dfb0e3c37f0129232d79307c65e6ee1ea13a3cbdcf301aaad0a6a/awscrt-0.32.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:51d6132e9d70de40da07dfad5f17780a652dd4b351c35ca97c79d0fa0186d645", size = 4028980, upload-time = "2026-04-24T22:59:20.316Z" }, + { url = "https://files.pythonhosted.org/packages/2d/7f/7f52020bfcf29122cad77e936d82abaa1f6857a5a70453e8cc734119f0e2/awscrt-0.32.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:193cc3ecf03a1dd6f989853b6c21549a0a9750c855520fedc8c3c8fbcd32e1bc", size = 4312564, upload-time = "2026-04-24T22:59:21.851Z" }, + { url = "https://files.pythonhosted.org/packages/25/86/2c9b5479e08b8198459d2a781df5524e45d4d0f9c6329bad26979a4d1e85/awscrt-0.32.2-cp314-cp314t-win32.whl", hash = "sha256:25c7e7e6535cb2d2a4d22fd6264f621672d3903491318364dbc59066b63c7186", size = 4195784, upload-time = "2026-04-24T22:59:23.769Z" }, + { url = "https://files.pythonhosted.org/packages/ab/85/a12515514de8969b9e7fa40bb782501a336a8a985cc3093309120a80b627/awscrt-0.32.2-cp314-cp314t-win_amd64.whl", hash = "sha256:a6782c19a00f354c7b232b675f09cde94d1ca37bcf29009b8779b3f6395b27b4", size = 4366435, upload-time = "2026-04-24T22:59:25.53Z" }, ] [[package]] @@ -588,7 +588,7 @@ wheels = [ [[package]] name = "bedrock-agentcore" -version = "1.21.0" +version = "1.24.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "boto3" }, @@ -600,9 +600,9 @@ dependencies = [ { name = "uvicorn" }, { name = "websockets" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/6c/86/1923fb5c5249768d4bf70ecd145b68f7e5fc8c8ef5de524d05d9744f7a4a/bedrock_agentcore-1.21.0.tar.gz", hash = "sha256:429174e42e2ebd2b82c691862caf0a6f2ad9c3182f1f6c48912570691491b7ce", size = 1106118, upload-time = "2026-08-06T14:47:42.461Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/b7/7f9eb61f380a78f7ffa8506a37cd3ded03848d32a39744e59b3f68c3d9da/bedrock_agentcore-1.24.0.tar.gz", hash = "sha256:f3999e45668343c753424a8886b1f95cf1015c7d4b27788a93099c8de233c6ae", size = 1273535, upload-time = "2026-09-28T17:05:42.693Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/7d/11/e917fd7ef38a612f7f955d6abb47978d860531f24f9b702ea20d1fd28802/bedrock_agentcore-1.21.0-py3-none-any.whl", hash = "sha256:be1ce7440455b994e9c5bbbc6329cce8528512441590240aa1a6f25cadbe1dcf", size = 482831, upload-time = "2026-08-06T14:47:41.006Z" }, + { url = "https://files.pythonhosted.org/packages/98/a8/8a7a2bbc9cb6c574c80c595909ae7340cb0d5517201fc813cc080e13e847/bedrock_agentcore-1.24.0-py3-none-any.whl", hash = "sha256:ecc3edcf4b0090736c23123356a7498ce1ac762b172301a1775a931c4d99e1dd", size = 522093, upload-time = "2026-09-28T17:05:41.165Z" }, ] [[package]] @@ -651,30 +651,30 @@ wheels = [ [[package]] name = "boto3" -version = "1.43.68" +version = "1.43.103" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "botocore" }, { name = "jmespath" }, { name = "s3transfer" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/35/40/95db6388539e6194b7d8863e4228263e7a778fff0164b17d0e4530d44ce2/boto3-1.43.68.tar.gz", hash = "sha256:4be7531c45fbf8eb145ecd0f385c7b8f9d66ba2fd0c256522717260e67fca89d", size = 112664, upload-time = "2026-08-10T19:22:07.864Z" } +sdist = { url = "https://files.pythonhosted.org/packages/46/59/012898d78087105e9c20fe31605d3f1999921745e1890ee0f0d993846e2e/boto3-1.43.103.tar.gz", hash = "sha256:524821052527f6446d249bf710847b032d9b12135e346c9751777a0a2811cf04", size = 112687, upload-time = "2026-09-25T19:26:30.427Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/96/bd/36dd3f5718160ea7f8ef5a81b461cfb9d0f0b289bb31ed416ae6a1cbfee3/boto3-1.43.68-py3-none-any.whl", hash = "sha256:4be071482312d05ea345ae8a2ea307637d3e2f4fdce44b8ebeb0192929aa2644", size = 140027, upload-time = "2026-08-10T19:22:05.592Z" }, + { url = "https://files.pythonhosted.org/packages/f2/66/6eb2e92c57b30e54c87fe14c8cd71de3aed9184f3cec95c39132a3fcc61f/boto3-1.43.103-py3-none-any.whl", hash = "sha256:574796187eb4e4fcaaf5c1b88c1724a1ba5a95bc27aeae5573676178666acc71", size = 140044, upload-time = "2026-09-25T19:26:29.085Z" }, ] [[package]] name = "botocore" -version = "1.43.68" +version = "1.43.103" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jmespath" }, { name = "python-dateutil" }, { name = "urllib3" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/e1/dc/ff7e35ecb25e2584e22ce8d5a13c5f991687597fd77d7cc388f2e4768cf3/botocore-1.43.68.tar.gz", hash = "sha256:a6c7ac88724c96f2ad5a7657f87d545d8218a74a1b219ba4397e3d37481941cd", size = 15894825, upload-time = "2026-08-10T19:22:02.501Z" } +sdist = { url = "https://files.pythonhosted.org/packages/3c/83/acbee6f2e3b1f02de935fef0c48a387511bf979149dc6b44466aaafba6a5/botocore-1.43.103.tar.gz", hash = "sha256:8c7f220e09f3b7ec99c59c716aa66cf6f5b2b57fd8731544054f3778aa744c6a", size = 16243609, upload-time = "2026-09-25T19:26:25.884Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/32/48/18dc095a908da94e2389d8e4127f4438b185dfd2dd9190fe6f946b9f41ac/botocore-1.43.68-py3-none-any.whl", hash = "sha256:9985c6eb9b7896f88bd89c07d543accd6985ea772af9d796087012ad40f78420", size = 15579433, upload-time = "2026-08-10T19:21:58.157Z" }, + { url = "https://files.pythonhosted.org/packages/8c/47/790ba88ec849d1e07b5866458b67e79b291164f3de7f23429b4dcb7e2ced/botocore-1.43.103-py3-none-any.whl", hash = "sha256:6a6a561a6de43845ae42bc7ff7e2c798742bfa277f5215398f9c15ed9b1bd67c", size = 15938608, upload-time = "2026-09-25T19:26:23.162Z" }, ] [[package]] @@ -2179,7 +2179,7 @@ wheels = [ [[package]] name = "mcp" -version = "1.28.1" +version = "1.30.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "anyio" }, @@ -2197,9 +2197,9 @@ dependencies = [ { name = "typing-inspection" }, { name = "uvicorn", marker = "sys_platform != 'emscripten'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/6e/77/9450b8f251a13affb6281997d0523c4615f8a8b35d0b21ff30db3a5aac9d/mcp-1.28.1.tar.gz", hash = "sha256:d51e36a5f5644faea4f85ea649bfffa6bc6c26770d42798ad6a3de3d2ba69683", size = 638501, upload-time = "2026-06-26T12:57:29.093Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/93/0142dc84a666daf8ad51a34268f34c12fd6fda4f3810c4be2504eecc8212/mcp-1.30.0.tar.gz", hash = "sha256:445414625fce5c295faa505bb11bacece661ab6f4028d57c935db57820b7a3e4", size = 680511, upload-time = "2026-09-07T14:34:15.845Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/e2/5e/d118fce19f87a2e7d8101c35c8ae0ec289098a4df0ff244cec23e415aca0/mcp-1.28.1-py3-none-any.whl", hash = "sha256:2726bca5e7193f61c5dde8b12500a6de2d9acf6d1a1c0be9e8c2e706437991df", size = 222620, upload-time = "2026-06-26T12:57:27.218Z" }, + { url = "https://files.pythonhosted.org/packages/f5/f4/e58bc33317c92a0203664daaf00bf6f41166cc0149e5d6870a03f7cd004a/mcp-1.30.0-py3-none-any.whl", hash = "sha256:666edb5009503e1047c9d60346a756f94b261f05cc2625f23d41c728ffc484d0", size = 234581, upload-time = "2026-09-07T14:34:14.266Z" }, ] [[package]] @@ -4747,7 +4747,7 @@ wheels = [ [[package]] name = "strands-agents" -version = "1.55.0" +version = "1.57.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "boto3" }, @@ -4764,17 +4764,18 @@ dependencies = [ { name = "typing-extensions" }, { name = "watchdog" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/60/37/04f19549c24bd7dbdbf5010317ed9b090f454aae550ed188b8b030b1be50/strands_agents-1.55.0.tar.gz", hash = "sha256:8ad2ad0b5306e1e0419cde03534d3a725d8fb80aa19ed7065c844bb2ffbd7dc1", size = 1599495, upload-time = "2026-09-08T16:55:24.132Z" } +sdist = { url = "https://files.pythonhosted.org/packages/91/87/050fa17fb5d9115e96b7332bea61a4a46ed9eb6c788b9486e807789640e6/strands_agents-1.57.2.tar.gz", hash = "sha256:6df1e6617b81c07d258631ef6dcf5e4388a93b561729d3696cd44113f2c530d7", size = 1726882, upload-time = "2026-10-01T18:55:51.078Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/33/90/6db60946626fb7f83249ac9f8e3037947f3aeac9b93bbbf116a4edd7be6c/strands_agents-1.55.0-py3-none-any.whl", hash = "sha256:01cd3ebd0606042b0e0c099aa6e549080a945ed3f4e063c223bf34f11373cd75", size = 807217, upload-time = "2026-09-08T16:55:22.37Z" }, + { url = "https://files.pythonhosted.org/packages/df/40/1df02c89427ab165a95cda83149159e09b655846c1d44589cda8ba79ccbd/strands_agents-1.57.2-py3-none-any.whl", hash = "sha256:c0d4726e63cfbb75cbe381c40d76029aaefb7f29555476e8080f626376c9fe00", size = 866480, upload-time = "2026-10-01T18:55:49.154Z" }, ] [package.optional-dependencies] bidi = [ { name = "aws-sdk-bedrock-runtime", extra = ["awscrt"], marker = "python_full_version >= '3.12'" }, - { name = "prompt-toolkit" }, + { name = "awscrt", marker = "python_full_version >= '3.12'" }, { name = "smithy-aws-core", marker = "python_full_version >= '3.12'" }, { name = "smithy-core", marker = "python_full_version >= '3.12'" }, + { name = "smithy-http", extra = ["awscrt"], marker = "python_full_version >= '3.12'" }, ] [[package]] diff --git a/docs-site/src/content/docs/configuration/environment-variables.md b/docs-site/src/content/docs/configuration/environment-variables.md index 92705d08b..37cad81de 100644 --- a/docs-site/src/content/docs/configuration/environment-variables.md +++ b/docs-site/src/content/docs/configuration/environment-variables.md @@ -1,26 +1,42 @@ --- title: Environment Variables -description: Key environment variables across services. +description: Where configuration lives, and how a value reaches a running container. sidebar: order: 1 --- -:::caution[Draft] -This page is a scaffolded placeholder — content to be written. -::: +Configuration reaches a running service through a fixed chain: -Document feature flags such as AGENTCORE_MCP_APPS_HOST_ENABLED, CDK_MCP_SANDBOX_ENABLED, and SKIP_AUTH. +``` +GitHub Actions variable → platform.yml workflow → scripts/common/load-env.sh → +cdk synth/deploy (infrastructure/lib/config.ts) → container env var → backend reader +``` -## Shared Projects +Almost every operator-facing knob is a `CDK_*` GitHub Actions variable (or the +matching `cdk.context.json` key). `config.ts` reads it at synth time, validates +it, and writes the plain env var (e.g. `SKILLS_ENABLED`, `CORS_ORIGINS`) onto the +Fargate task / AgentCore Runtime. -| Variable | Service | Default | Notes | -| --- | --- | --- | --- | -| `CDK_PROJECTS_ENABLED` | GitHub environment variable → CDK | off | Set to `true` to turn Projects on in that environment | -| `PROJECTS_ENABLED` | app-api, inference-api | off | Set by CDK from the above; only `true` enables | -| `DYNAMODB_PROJECTS_TABLE_NAME` | app-api, inference-api | — | `{prefix}-projects`, set by CDK | -| `DYNAMODB_AUDIT_LOG_TABLE_NAME` | app-api | — | Required for the project Activity trail | -| `PROJECTS_MAX_MEMBERS` | app-api | `200` | Not set by CDK | -| `PROJECTS_EDITORS_MANAGE_MEMBERS_DEFAULT` | app-api | on | Not set by CDK | -| `DIRECTORY_PROVIDER` | app-api | `users_table` | Not set by CDK | +## Feature flags -The SPA has a matching switch, `features.projects`, in its environment files. Both sides, and what the switch stops: [Admin › Projects](/agentcore-public-stack/admin/projects/#turning-projects-on). +Every on/off feature flag — what it does, its default on a fresh fork, and +whether it costs money — has its own page: + +**→ [Feature Flags](/agentcore-public-stack/configuration/feature-flags/)** + +## Other configuration + +- **CORS origins** — see [CORS](/agentcore-public-stack/configuration/cors/). +- **Authentication** (Cognito, federated IdPs, `SKIP_AUTH` local-dev bypass) — + see [Authentication](/agentcore-public-stack/configuration/authentication/). +- **AgentCore services** (Memory, Gateway, Code Interpreter, Browser) — see + [AgentCore Services](/agentcore-public-stack/configuration/agentcore-services/). +- **Sizing, observability, and tuning knobs** — the authoritative list with + defaults lives in + [`infrastructure/lib/config.ts`](https://github.com/Boise-State-Development/agentcore-public-stack/blob/main/infrastructure/lib/config.ts). + +## Local development + +For local backend/frontend runs, values come from `backend/src/.env` (see +`backend/src/.env.example`) and the frontend `environment.ts`, not from CDK. See +[Local Development](/agentcore-public-stack/local-development/). diff --git a/docs-site/src/content/docs/configuration/feature-flags.md b/docs-site/src/content/docs/configuration/feature-flags.md new file mode 100644 index 000000000..403aa7b57 --- /dev/null +++ b/docs-site/src/content/docs/configuration/feature-flags.md @@ -0,0 +1,234 @@ +--- +title: Feature Flags +description: Every feature flag in the stack — what it does, its default on a fresh fork, and whether it costs money. +sidebar: + order: 2 +--- + +This is the complete list of feature flags in the stack: what each one does, +whether it is **on or off on a fresh fork**, and whether it **incurs recurring +cost**. If you are standing up your own deployment and want to know "what am I +paying for by default, and what can I turn off?", this page is the answer. + +:::tip[The one-minute version] +- **Most feature flags default ON** ("default on with a kill switch"). You opt + *out* by setting the variable to `false`. +- **A few default OFF** and must be opted *into* — these are the ones with a + real cost or a wide blast radius (Managed KB, feedback evaluations, MCP token + enrichment). +- **Almost nothing costs extra just by being on.** The cost-bearing flags are + called out explicitly in [Flags that cost money](#flags-that-cost-money). +- **Some runtime-only flags are temporary.** A flag marked *Rollout* exists + only to undo a recent change quickly, and it will be removed. Don't build on + its off position. See + [Feature switches and rollout switches](#feature-switches-and-rollout-switches). +::: + +## How flags work + +A flag lives in one of two layers, and it matters which: + +1. **CDK-exposed flags** — set with a `CDK_*` GitHub Actions variable (or a + `cdk.context.json` key). CDK reads it at synth/deploy time and writes a + plain env var (e.g. `SKILLS_ENABLED`) onto the container. **These are the + flags you configure.** +2. **Runtime-only flags** — read directly from the container environment by the + backend, with **no `CDK_*` variable wired up**. They are all ON by default + and can only be flipped by hand-editing the task definition environment. + See [Runtime-only flags](#runtime-only-flags-on-by-default-no-cdk_-variable). + +The delivery chain for a CDK-exposed flag is: + +``` +GitHub Actions variable → platform.yml workflow → load-env.sh → +cdk synth/deploy (config.ts) → container env var → backend reader +``` + +### Two idioms, and the empty-string rule + +- **Default ON with a kill switch** (most flags): unset or empty resolves to + **enabled**; only the literal string `false` disables it. This is deliberate + — an *unset* GitHub Actions variable is forwarded as an **empty string**, and + the empty string must mean "use the default (on)", never "off". +- **Default OFF, opt-in** (cost/blast-radius flags): only the literal string + `true` enables it. An unset/empty variable stays off. + +:::caution[Reading the code default can mislead you] +For several flags the Python reader's built-in default and the **deployed** +default differ, because **CDK sets the env var explicitly**. Example: +`memory_spaces_enabled()` in `backend/src/apis/shared/feature_flags.py` defaults +to *off* if the env var is unset — but every CDK deploy writes +`MEMORY_SPACES_ENABLED=true`, so a real deployment gets it **on**. **Trust the +"Default on a fork" column below, not the code default.** (The gap only exists +for local dev with no env var set.) +::: + +## CDK-exposed flags + +Set these with the named `CDK_*` GitHub Actions variable. "Default on a fork" is +what you get if you set nothing. + +| Feature | `CDK_*` variable | Default on a fork | Recurring cost | What it gates | +|---|---|:---:|:---:|---| +| Skills (catalog, picker, authoring) | `CDK_SKILLS_ENABLED` | **ON** | none | Skills routes + runtime resolution | +| Scheduled / headless runs | `CDK_SCHEDULED_RUNS_ENABLED` | **ON** | when runs fire | "Run now" + headless-grant routes (dispatcher when Phase B lands) | +| Memory Spaces | `CDK_MEMORY_SPACES_ENABLED` | **ON** | none | `/memory/spaces` CRUD, runtime tools, SPA panel | +| Agent Designer | `CDK_AGENTS_API_ENABLED` | **ON** | none | The governed `/agents/*` authoring surface | +| Agent Marketplace | `CDK_AGENT_MARKETPLACE_ENABLED` | **ON** | none | Listing lifecycle, publisher profiles, admin Review/Listings (app-api only) | +| Fine-tuning | `CDK_FINE_TUNING_ENABLED` | **ON** | **only when a job runs** | Mounts `/fine-tuning` routes. SageMaker training bills per job; `defaultQuotaHours=0` keeps it whitelist-only | +| Artifact share inbox | `CDK_ARTIFACT_SHARE_INBOX_ENABLED` | **ON** | none | "Shared with you" tab (`GET /shared-artifacts`) | +| KB sync (scheduled re-index) | `CDK_KB_SYNC_ENABLED` | **ON** | **yes — scheduled** | EventBridge rule + kb-sync Lambdas that re-embed assistant KB sources | +| Managed KB — new default | `CDK_MANAGED_KB_NEW_DEFAULT` | **OFF** | **yes — storage** | New KBs provisioned on managed backend (~$5/GB-mo vs ~$0.15) | +| Managed KB — migration worker | `CDK_MANAGED_KB_MIGRATION_ENABLED` | **OFF** | **yes — storage** | Whether the background Migration_Worker runs at all | +| Managed KB — reconciler armed | `CDK_MANAGED_KB_RECONCILER_ARMED` | **OFF** | none (report-only until armed) | Reconciler *deletes* orphans vs only logging them | +| Managed KB — doc reconciler armed | `CDK_MANAGED_KB_DOC_RECONCILER_ARMED` | **OFF** | none (report-only until armed) | Dead-letter doc reconciler *corrects* rows vs only reporting | +| Feedback → Evaluations sampling | `CDK_FEEDBACK_EVAL_SAMPLING_ENABLED` | **OFF** | **yes — per eval run + PII** | Admin batch sends down-thumbed conversations to AgentCore Evaluations | +| MCP token enrichment | `CDK_MCP_TOKEN_ENRICHMENT_ENABLED` | **OFF** | **yes — Cognito plan** | Pre-token Lambda that copies pool attributes into access-token claims; forces Cognito **Essentials** feature plan (per-MAU cost) | +| MCP Apps host renderer | `AGENTCORE_MCP_APPS_HOST_ENABLED` | **ON** | none | Renders third-party MCP-server UI in the sandbox iframe (needs `mcp-sandbox` deployed) | +| Token exchange (RFC 8693) | `CDK_TOKEN_EXCHANGE_URL` (+ `_CLIENT_ID`) | **absent** | none | Optional external token-service exchange. Unset ⇒ no resources created | + +## Runtime-only flags (on by default, no `CDK_*` variable) + +These have **no CDK variable**. They are **all ON** and can only be turned off by +adding the env var directly to the container task definition. They are grouped +here because a fork operator will never find them by looking at `CDK_*` +variables. Two of them cost extra against the model: tool summaries, and the +extra call compaction makes to pin facts. The **Kind** column says whether a flag +is permanent (*Feature*) or will be removed (*Rollout*, with the condition for +removing it). + +| Env var | Default | Kind | Cost | What it gates | +|---|:---:|---|:---:|---| +| `TOOL_SUMMARIES_ENABLED` | **ON** | Feature | **small — Nova Micro** | Model-generated one-line summary per tool batch. Off ⇒ falls back to free client-side formatters | +| `AGENT_STATUS_ENABLED` | ON | Feature | none | Live "what the agent is doing" status line + per-tool durations | +| `AGENT_STATUS_LIVE_DRAIN_ENABLED` | ON | Rollout — eligible now (shipped ≤ 1.24.0) | none | Drains status during tool-execution silence (fixes bundled-late narration) | +| `AGENT_PREPARING_PHASE_ENABLED` | ON | Rollout — eligible now (shipped ≤ 1.24.0) | none | Builds the agent inside the stream so a cold start is narrated, not dead air | +| `MID_TURN_STEERING_ENABLED` | ON | Feature | none | Inject a follow-up into a running turn at tool boundaries | +| `ANNOUNCEMENTS_ENABLED` | ON | Feature | none | Feature-announcement authoring + user-facing surfaces | +| `RESPONSE_FEEDBACK_ENABLED` | ON | Feature | none | Thumbs up/down on assistant messages | +| `ASK_USER_QUESTION_ENABLED` | ON | Feature | none | `ask_user_question` built-in tool (pauses a turn for structured input) | +| `BROWSER_TAKEOVER_ENABLED` | ON | Feature | none | `request_user_login` — hand the browser to the user to sign in | +| `BROWSER_TOOL_ENABLED` | ON | Feature | usage-based | The `browse_web` tool itself (AgentCore Browser sessions bill while active) | +| `WORKSPACE_TOOLS_ENABLED` | ON | Feature | none | `workspace_list` / `read` / `write` file tools | +| `DOCUMENT_READ_ENABLED` | ON | Feature | none | `document_read` tool for sessions carrying a readable attachment | +| `ATTACHMENT_TOOL_AUTOENABLE_ENABLED` | ON | Feature | none | Auto-injects spreadsheet-analysis tools when a spreadsheet is attached | +| `ATTACHMENT_TURN_GUARD_ENABLED` | ON | Rollout — eligible now (shipped ≤ 1.24.0) | none | Enforces per-message file count + inline-byte budget | +| `ADMIN_ALWAYS_ON_TOOLS_ENABLED` | ON | Feature | none | Unions admin-flagged `alwaysOn` tools into every turn (inert with no data) | +| `COST_DIAGNOSTICS_ENABLED` | ON | Feature | none | Content-free behavioral counters for the admin session profile | +| `CONFIG_CACHE_ENABLED` | ON | Rollout — eligible now (shipped ≤ 1.24.0) | **saves** money | In-process cache of tenant-global catalogs (fewer DynamoDB reads) | +| `AGENT_BUILD_SHARED_SESSION_ENABLED` | ON | Rollout — retires after the Runtime V2 migration, once a V2 snapshot restore is clean | none (**saves** ~0.5s on a cold first turn) | One process-wide boto3 session for the agent build's SDK clients (Memory session manager, strategy-id discovery, Bedrock model), built at container warm-up. Off ⇒ each SDK builds its own session, as before | +| `MCP_PARALLEL_PREFLIGHT_ENABLED` | ON | Rollout — retires after one prod release + 2 weeks clean | none (**saves** all but the slowest server's startup on a first turn with several external MCP servers — ~9.5s measured on dev with two cold servers) | Pre-flights an agent's external MCP servers concurrently when the agent is built, instead of one after another; clients are still registered in catalog order, so tool order (and the prompt-cache prefix) is unchanged. Off ⇒ servers load one at a time, as before | +| `KB_SEARCH_AHEAD_ENABLED` | ON | Rollout — retires after one prod release + 2 weeks clean | none (**saves** up to the knowledge-base search's round trips — ~0.4s measured on dev — on an agent turn with a knowledge base) | Starts an agent turn's knowledge-base search where it always ran but awaits it after the agent build, ahead of the citation frames; the augmented message and citations are byte-identical. Off ⇒ the search is awaited before the build, as before | +| `MEMORY_RETRIEVAL_PREFETCH_ENABLED` | ON | Rollout — retires after one prod release + 2 weeks clean | none (**saves** ~200ms on every turn with long-term memory) | Starts the long-term-memory lookup as soon as the user's message is added, overlapping the two Memory writes the SDK awaits first; the result is applied at the same point as before, so persisted and live messages are unchanged. Off ⇒ the lookup runs after the writes, as before | +| `HISTORY_COUNT_PREFETCH_ENABLED` | ON | Rollout — retires after one prod release + 2 weeks clean | none (**saves** ~100–400ms on every warm turn, growing with conversation length) | Reads the per-turn count of stored messages (the base every per-message metadata index is computed from) on a worker thread started at the head of the turn, instead of before the model call; counts only messages created before the turn began, so the index is the one the inline read gave. Off ⇒ the count is read at the head of the turn, as before | +| `INLINE_ATTACHMENT_PERSIST_ENABLED` | ON | Rollout — retires after one prod release + 2 weeks clean | none | Writes a headless caller's inline spreadsheet or deck bytes to S3 as session files before the turn runs, so the tools the guidance note names can reach them. The SPA uploads first, so it never takes this path. Off ⇒ diverted inline attachments are dropped, as before | +| `COMPACTION_SUMMARY_EXTRACT_ENABLED` | ON | Rollout — retires after a production quality readout of extract-then-compress | one extra side-channel call per compaction cut | Pins standing instructions, decisions, identifiers and latest values verbatim ahead of the compaction summary (two concurrent calls). Off ⇒ the single plain compression call | +| `DOCUMENT_OFFLOAD_ENABLED` | ON | Rollout — retires once the quality check waived in `docs/specs/document-offload-evaluation.md` has run | none | Document-context-offload pipeline (see spec) | +| `DOCUMENT_REHYDRATE_ENABLED` | ON | Rollout — retires with `DOCUMENT_OFFLOAD_ENABLED` | none | Re-injects offloaded document context on demand | +| `DOCUMENT_DIGEST_ENABLED` | ON | Rollout — retires with `DOCUMENT_OFFLOAD_ENABLED` | possible side-channel | Document digest step of the offload pipeline | + +:::note +`SCHEDULED_RUNS_ENABLED`, `KB_SYNC_ENABLED`, `MEMORY_SPACES_ENABLED`, +`FINE_TUNING_ENABLED` and the Managed-KB vars also appear as raw env vars read by +background Lambdas/dispatchers — but you configure them through their `CDK_*` +variable in the table above, not by hand. The Lambda-side readers default to +*off* when unset, which is why CDK always sets them explicitly. +::: + +## Feature switches and rollout switches + +Every default-on flag is one of two kinds: + +- **Feature switch**: turns off a capability you may reasonably not want, + because it costs money, widens what users can do, or is a product choice. + These are configuration and are permanent. Every CDK-exposed flag is a + feature switch or an opt-in. +- **Rollout switch**: guards a change that is meant to be invisible to users, + such as a reordering on the chat path, a fix or a cache. It exists so the + maintainers can undo a bad deploy without reverting code. Nobody needs the off + position once the change is proven, and an off path that nobody runs stops + working without anyone noticing. So rollout switches **are removed**. + +A rollout switch normally retires after it has shipped in a production release +and run there for at least two weeks with no one needing to flip it. Some +switches wait for a stricter condition, which the Kind column states. +Retirements are batched into one cleanup per release cycle, and each removed +variable is listed under **Removed** in the +[CHANGELOG](https://github.com/Boise-State-Development/agentcore-public-stack/blob/main/CHANGELOG.md). + +**If you've set a rollout switch to `false`**, treat it as temporary and tell +the maintainers why, through an issue. A rollout switch someone actually needs +turned off is a bug report, and it stays until that bug is fixed. + +## Flags that cost money + +If you care about the bill, these are the only flags that move it: + +- **`CDK_MANAGED_KB_NEW_DEFAULT` / `CDK_MANAGED_KB_MIGRATION_ENABLED`** — managed + KB storage bills ~**$5/GB-month** (vs ~$0.15 for the legacy backend). Both + default **OFF** for exactly this reason. Leave them off unless you have + decided to pay for managed retrieval. +- **`CDK_FEEDBACK_EVAL_SAMPLING_ENABLED`** — each admin-triggered batch sends + conversation spans (system prompt + user messages) to an AWS-managed + evaluator: a per-run cost **and** a PII exposure decision. Default **OFF**. +- **`CDK_MCP_TOKEN_ENRICHMENT_ENABLED`** — turning it on forces the Cognito user + pool onto the **Essentials** feature plan, which bills per monthly active + user. Default **OFF**. +- **`CDK_KB_SYNC_ENABLED`** — default **ON**. Runs a scheduled Lambda that + re-embeds assistant KB sources; cost is the embedding calls + Lambda time on + the schedule. Turn off if you do not use assistant knowledge bases. +- **`TOOL_SUMMARIES_ENABLED`** — default **ON**, with a recurring model cost: + one bounded **Nova Micro** call per tool batch (a side-channel that never + touches the cacheable prompt prefix). Cheap, but real. Set to `false` to fall + back to free client-side tool formatters. +- **`COMPACTION_SUMMARY_EXTRACT_ENABLED`** — default **ON**. Each compaction cut + makes one extra summary-model call to pin facts verbatim. Cuts are infrequent, + so the extra cost is small. This is a rollout switch, so plan on it going + away rather than turning it off to save money. +- **`CDK_FINE_TUNING_ENABLED`** — default **ON**, but the routes are free; only a + SageMaker training job actually launched by a user bills, and + `CDK_FINE_TUNING_DEFAULT_QUOTA_HOURS=0` keeps it admin-whitelist-only. +- **Observability** (`CDK_OBSERVABILITY_XRAY_SAMPLING_RATE`, + `CDK_OBSERVABILITY_AGENTCORE_APPLICATION_LOGS_ENABLED`) — X-Ray bills per trace + recorded (default sampling 1%); AgentCore application logs are **off by + default** because they are high-volume and carry full prompts/responses. + +## Access-control settings (not feature rollout) + +These look like flags but govern *access*, not feature availability. They +deliberately **default to the closed/safe posture**, inverting the "default on" +rule. + +| Setting | `CDK_*` variable | Default | Notes | +|---|---|:---:|---| +| Cognito self-signup | `CDK_COGNITO_SELF_SIGNUP_ENABLED` | **OFF (closed)** | Opens the Hosted UI "Sign up" link to the public internet. Federated sign-in and first-boot admin are unaffected | +| Gateway inbound auth | `CDK_GATEWAY_INBOUND_AUTH` | `iam` | `iam` or `jwt`. **Immutable after Gateway creation** — changing it needs a new Gateway | +| Browser URL blocklist | `CDK_BROWSER_URL_BLOCKLIST` | **empty** | Hosts Chromium refuses during a browser takeover. Empty ⇒ RBAC on `browse_web`/`request_user_login` is the only control (the deploy log warns) | +| Local-dev auth bypass | `SKIP_AUTH` | **OFF** | `SKIP_AUTH=true` returns a fake admin. **Local dev only** — a CI guard refuses any PR that puts it in deployable config, and app-api refuses to boot if it is paired with a non-localhost CORS origin | + +## Non-boolean tuning knobs + +Beyond on/off flags, these numeric/string knobs shape cost and behavior. They +follow the same env → context → default precedence. See +[`infrastructure/lib/config.ts`](https://github.com/Boise-State-Development/agentcore-public-stack/blob/main/infrastructure/lib/config.ts) +for the authoritative list and defaults: + +- **Managed KB byte caps**: `CDK_MANAGED_KB_PER_OWNER_BYTES` (100 MB), + `CDK_MANAGED_KB_PER_OWNER_ELEVATED_BYTES` (1 GB), + `CDK_MANAGED_KB_PER_KB_CEILING_BYTES` (500 MB), plus fleet alarms + `CDK_MANAGED_KB_STORAGE_ALARM_GB` / `CDK_MANAGED_KB_DAILY_COST_ALARM_USD`. +- **Observability thresholds**: log retention, X-Ray sampling, ALB/Lambda/ECS + alarm thresholds, per-model Bedrock TPM quotas — all `CDK_OBSERVABILITY_*`. +- **App-API sizing**: `CDK_APP_API_CPU` / `_MEMORY` / `_DESIRED_COUNT` / + `_MAX_CAPACITY`. + +## Source of truth + +- **Runtime flag readers + rationale**: `backend/src/apis/shared/feature_flags.py` + (each flag documents its own default and why). +- **CDK-exposed flags + env wiring**: `infrastructure/lib/config.ts` and the + `infrastructure/lib/constructs/*/*-environment.ts` maps. + +If you add a flag, update this page in the same PR — that is the whole point of +having one canonical list. Give a rollout switch its retirement condition in the +Kind column when you add it. diff --git a/docs-site/src/content/docs/development/testing.md b/docs-site/src/content/docs/development/testing.md index d6731b0d3..a359fd5f3 100644 --- a/docs-site/src/content/docs/development/testing.md +++ b/docs-site/src/content/docs/development/testing.md @@ -1,12 +1,38 @@ --- title: Testing -description: Backend and frontend tests. +description: What each test suite covers, how to run it locally, and what runs on a pull request. sidebar: order: 2 --- -:::caution[Draft] -This page is a scaffolded placeholder — content to be written. -::: +## Suites -pytest (not run in CI), Vitest, jest for infra, and import-boundary tests. +| Suite | Where | Run locally | +|---|---|---| +| Backend (pytest, ~11k tests) | `backend/tests/` | `cd backend && uv run pytest tests/ -n logical` | +| Root supply-chain contracts | `tests/supply_chain/` | `uv run --project backend --no-sync pytest tests/supply_chain/` | +| Frontend (Vitest via Analog) | `frontend/ai.client/src/**/*.spec.ts` | `cd frontend/ai.client && npm test` | +| Infrastructure (jest) | `infrastructure/test/` | `cd infrastructure && npx jest` | +| Load-test unit tests | `tests/load/tests/` | `cd tests/load && uv run pytest tests/` | +| Operational scripts | `scripts/restore-data/test_restore.py` | `cd scripts/restore-data && uv run --frozen pytest test_restore.py` | + +The backend suite is hermetic: a conftest guard fails any test that opens a socket to a non-local host, so a partially mocked test cannot quietly reach AWS. Run it with `-n logical` rather than `-n auto`; with psutil installed, `auto` counts physical cores and starts half as many workers as the machine has. + +## What runs on a pull request + +`.github/workflows/ci.yml` runs on every pull request into `develop` or `main`, but it does not run every suite every time. A `changes` job reads the PR's file list and maps each path to the suites that can observe it (`scripts/ci/classify-changes.sh`): + +| Changed paths | Suites | +|---|---| +| `docs/`, `docs-site/`, `tui/`, root markdown, `LICENSE` | none | +| `backend/` | backend, load-test unit | +| `backend/scripts/` | backend, load-test unit, infrastructure (jest asserts the backfill scripts exist) | +| `frontend/` | frontend, frontend under `--coverage` | +| `infrastructure/`, `RELEASE_NOTES.md` | infrastructure, plus the backend supply-chain and architecture suites that read the CDK tree | +| `.claude/` | infrastructure (one jest test reads the release skill) | +| `tests/load/` | load-test unit | +| `.github/`, `scripts/`, anything unrecognised | everything | + +The mapping is fail-open. A new top-level directory, an empty file list, or a `workflow_dispatch` run turns every suite on; only the explicit list in the first row turns none on. The rules are pinned by `backend/tests/supply_chain/test_ci_path_filter.py`, so extend the script and the test together. + +The deploy workflows (`backend.yml`, `platform.yml`, `frontend-deploy.yml`) are unaffected: on push to `develop` and `main` they still run their full suites before deploying, so the PR gate is the first line, not the last. diff --git a/docs/kaizen/decisions.md b/docs/kaizen/decisions.md index 711d34504..cf5058f9e 100644 --- a/docs/kaizen/decisions.md +++ b/docs/kaizen/decisions.md @@ -49,3 +49,27 @@ the new context would have to be to re-open it. - **Decision**: **Keep both skills. Stop flagging them.** - **Reasoning**: age is not dormancy for a reference skill. Both encode conventions that are still correct and still load-bearing — which is exactly why neither has needed an edit. The listing was a metric finding a fact about the file system, not about the repo. - **Re-open only if**: a skill's *content* is contradicted by the code it describes. Do not re-flag either skill on days-since-modified. + +### [2026-10-03] Declined — docling #405 and Guardrails #480 as kaizen items +- **Origin**: review-queue.md ▸ [2026-06-05] docling bump, [2026-06-19] configurable Guardrails, [2026-07-03] gateway-level Guardrails; recommended Decline in reviews/2026-09-25.md ▸ Carried Over (7th carry) and reviews/2026-10-02.md ▸ Carried Over (8th). +- **Decision**: **Decline as kaizen items.** Issues #405 and #480 stay open as product backlog; only the queue entries close. +- **Reasoning**: eight carries without a surface anyone picked up. #480 is a product decision (in-agent `guardrail_id` vs. a gateway-level AgentCore Policy, and who owns the alerting it implies), not an improvement a weekly review can ship. The docling bump is the opposite: small and well-specified, yet it carried anyway, because the queue is not where a backlog bug gets picked up. A kaizen item needs an entry point; neither entry had one the forum would act on. +- **Re-open only if**: someone proposes a PR-sized slice. The docling bump alone qualifies (move `requirements.lock` off `docling==2.81.0`, verify a `.txt` upload, close #405). Guardrails returns as product work on #480, not as a queue entry. + +### [2026-10-03] Declined — the standing "do derived execution contexts re-evaluate RBAC" audit +- **Origin**: carried in reviews since it first fell due on 2026-09-11 (no queue entry of its own); recommended Decline in reviews/2026-09-25.md ▸ Carried Over (3rd time due) and reviews/2026-10-02.md ▸ Carried Over. +- **Decision**: **Decline.** +- **Reasoning**: it never had a surface. The concern is real, but it is answered per context, not by a standing audit: Shared Projects (#1257/#1258) built a new derived execution context and gave it its own named check (`apis/shared/projects/access.py`), which is the pattern that works. Same shape as the [2026-09-21] `oauth_required` decline. +- **Re-open only if**: a specific derived context is named together with the RBAC check it skips. That is ordinary bug or design work and welcome; the standing audit is not. + +### [2026-10-03] Accepted — `bedrock-agentcore` #629, last-turn telemetry under-reports +- **Origin**: review-queue.md ▸ [2026-08-14] "Guard against `bedrock-agentcore` #564" (named there as a related upstream risk); recommended Accept in reviews/2026-09-25.md ▸ Carried Over (5th cycle) and reviews/2026-10-02.md ▸ Carried Over (6th). +- **Decision**: **Accept** the known under-report. No local workaround. +- **Reasoning**: the SDK never flushes its TracerProvider before the microVM freezes, so end-of-invocation spans are dropped and span-based telemetry under-reports the final turn of each session. Upstream has been silent since August, and six cycles of carrying it changed nothing. A local flush shim would be patching SDK lifecycle we do not own, for a bounded and known gap. +- **Re-open only if**: the upstream issue moves (a fix or a flush hook ships in a version we can pin), or a cost readout is visibly short by one turn per session. + +### [2026-10-03] Declined as scoped — reverse the Astra 272K decision +- **Origin**: review-queue.md ▸ [2026-09-18] "Reverse the Astra decision — register at the full 1M window and make the price tier its own catalog field"; recommended Decline in reviews/2026-09-25.md ▸ Retirement Candidates and reviews/2026-10-02.md ▸ Carried Over. +- **Decision**: **Decline as scoped.** GPT-6 Astra stays registered at `maxInputTokens: 272_000`. +- **Reasoning**: the cap is load-bearing pricing, and `curated-models.ts` says so in the comment above the Mantle Responses rows. AWS prices these models on two cards (above 272K input tokens, input costs 2× and output 1.5×), while `CuratedModel` holds one flat rate per bucket. Raising the cap without a tier field silently opens the second price card and under-charges every long turn. The tier-boundary field half of the entry never shipped (see the resolved [2026-09-11] Astra entry). +- **Re-open only if**: it comes together with a per-tier rate field on `CuratedModel` that the cost calculator reads. With that in place, the 1M window is the natural follow-on. diff --git a/docs/kaizen/research/2026-10-02.md b/docs/kaizen/research/2026-10-02.md new file mode 100644 index 000000000..c3bbc0570 --- /dev/null +++ b/docs/kaizen/research/2026-10-02.md @@ -0,0 +1,416 @@ +# Kaizen Research — Friday, October 2, 2026 + +> Scan window: **September 25 – October 2, 2026 (7 days)**. +> Web budget: **13 / 50** used. Seven subagents. Five of them ran almost entirely through `gh`, `curl` and registry APIs, which cost nothing against the budget. + +## TL;DR + +**The week's most consequential external item is the price of the V2 AgentCore Runtime, and it changes the shape of last week's #1.** AWS published the V2 rates: **$0.0169/GB-hour of memory against V1's $0.00945, which is 1.79×**, and $0.1276/vCPU-hour against $0.0895 (1.43×). Idle memory is reclaimed after 120 s, and a committed-baseline tier ($0.0132/GB-hour) is "launching by October 2026". V2 therefore saves money only if our billed GB-hours fall **below ~56% of V1's**. That is plausible, because V1 bills an idle microVM at its peak for the whole 900 s idle timeout. But it is a measurement to make, not a given. In the same week `aws-cdk-lib` 2.272.0 added a typed `CfnRuntime.platformVersion`, which retires the plan's `addPropertyOverride`. Step 1 of the merged V2 plan, the deploy-script allow-list fix (B1), has still not been started. + +**The most pressing internal signal is that the nightly has been red for two nights and will stay red until someone acts.** On 10-01 the teardown of the ephemeral `nightly-develop-PlatformStack` failed because the AgentCore Runtime timed out while deleting (`NotStabilized`). The stack is now in `DELETE_FAILED`, so the 10-02 deploy could not update it. No upstream report matches. + +**Recommended #1: re-base the V2 plan on the published price, and land its B1 deploy-script fix.** Without that fix the dev A/B cannot be run cleanly, and the price now decides whether the A/B is worth running at all. + +## External Scan + +### What's moving this week + +The week's shape: **the model layer moved again, and two of its moves are about history, not capability.** Claude Sonnet 5.5 shipped (09-28, Bedrock `global.*`), and Sonnet 4.5 was deprecated for 2026-11-30. GPT-6.1 Sol, Grok 4.7 and a GPT-6 Astra "UltraFast" tier landed on Bedrock. The more important change is in the guidance. Anthropic's cookbook added a caveat (09-28): on models with **preserved thinking**, rewriting or compacting earlier history, including collapsing a tool result, invalidates every later thinking block and **can get the next request rejected**. Opus 5.5, which we curated on 09-26, has thinking that cannot be turned off. Our compaction and tool-result offloader both rewrite history. Nobody has run a compaction turn on Opus 5.5. + +The second theme is **append-only context changes reaching Bedrock, partially.** The Bedrock user guide now documents both `mid-conversation-tool-changes-2026-07-01` and a new `inline-tools-2026-09-15` beta for Claude. Each adds or removes tools from a point in the conversation onward without invalidating the cache. Support is limited to Opus 5.5, Fable 5.1, Mythos and Opus 5/4.8. Sonnet 5, Sonnet 4.6 and the 4.5 models return 400, and the examples use the InvokeModel body, not Converse. That rules it out for our default Haiku path and probably for Strands today. It is still the first time the structural fix for "tools that appear mid-session re-write the prefix" has been reachable on Bedrock at all. + +The third: **Strands kept shipping into a bump we haven't landed.** `python/v1.57.2` (10-01) carries our own #4193, so the cache-write half of `usage_normalization.py` can retire. It also **removes `session_manager` from `BidiAgent`**, which our voice agent passes, including on PR #1367's branch. The paired-bump PR has been open for five days, has no review, and is now `CONFLICTING`. The harness-sdk#4618 hazard did not move: there is no design doc and no adapter folds cache tokens, though a commenter's 96-model matrix found that the usage convention follows the *API*, not the model family. + +### Notable items by source + +> **Annotation conventions:** `*relevance*:` = impact on existing code. `*unlocks*:` = capability-unlock lens. + +#### AWS Bedrock / AgentCore + +- **AgentCore Runtime V2 pricing published** (on the pricing page, undated; verified against the page source 2026-10-02). Consumption rates are $0.1276/vCPU-hour and **$0.0169/GB-hour**; V1 is $0.0895 and $0.00945. The committed baseline is $0.0997 vCPU / $0.0132 GB, "launching by October 2026". Idle memory is reclaimed after **120 s**. Billing is per second on actual consumption, with a 1 s minimum — https://aws.amazon.com/bedrock/agentcore/pricing/ + - *relevance*: `docs/specs/agentcore-runtime-v2.md` still says "No V2 price" (§1) and "Unknown rate" (§4). The break-even is now arithmetic: on memory, which is ~73% of the AICC bill, V2 wins only if billed GB-hours fall below **1 / 1.79 ≈ 56%** of V1's. CPU, at 1.43×, needs used vCPU-hours below ~70%. + - *unlocks*: a go/no-go threshold for the dev A/B, stated before the A/B runs. The committed baseline is the V2-native replacement for the W5 instance-SKU arithmetic, as the spec anticipated. +- **`bedrock-agentcore` SDK v1.24.0** (09-28). Changes: graceful empty payloads (#683), `WaitConfig` with region aliases (#675), the aws-cn partition, and payments user-agent attribution. No breaking change seen — https://github.com/aws/bedrock-agentcore-sdk-python/releases/tag/v1.24.0 + - *relevance*: PR #1367 targets 1.23.1. 1.24.0's `[strands-agents]` extra still needs strands ≥1.56 and `mcp<2`, so it is a drop-in for the same pairing. +- **SDK #681 (open, updated 09-30)**: `AgentCoreMemorySessionManager` builds and discards two boto3 clients per construction, because `boto_session` never reaches `MemoryClient`. The reporter measured 37–51 ms per turn — https://github.com/aws/bedrock-agentcore-sdk-python/issues/681 + - *relevance*: the agent path is already worked around (`session/session_factory.py:99`, `_SharedSessionMemoryClient`, #1395). Two app-api call sites construct the manager directly with no workaround: `apis/shared/sessions/messages.py:389` and `apis/app_api/shares/service.py:502`. They are not on the TTFT path, so this is low priority. +- **Amazon Bedrock Managed Agents (powered by OpenAI), preview** (09-29). AWS hosts the agent loop: durable sessions, tools, code execution and skills, on OpenAI's Agents API — https://aws.amazon.com/about-aws/whats-new/2026/09/bedrock-managed-agents-preview/ + - *relevance*: a managed alternative to our harness, for OpenAI models only. **Watch**; it is not a candidate. +- **New models**: Claude Sonnet 5.5 (09-28), Grok 4.7 (09-28), GPT-6.1 Sol GA (09-29), GPT-6 Astra UltraFast tier (09-30). Claude in-region availability also expanded (London, India, Seoul, Singapore) — https://aws.amazon.com/about-aws/whats-new/2026/09/claude-sonnet-5-5-aws/ · https://aws.amazon.com/about-aws/whats-new/2026/09/openai-gpt-6-1-sol-on-amazon-bedrock/ · https://aws.amazon.com/about-aws/whats-new/2026/09/amazon-bedrock-grok-4-7/ · https://aws.amazon.com/about-aws/whats-new/2026/09/openai-gpt-6-astra-ultrafast-on-amazon-bedrock/ + - *relevance*: `curated-models.ts` on `develop` has **no** Sonnet 5.5, GPT-6.1 Sol or Grok 4.7 row. See the friction section: Sonnet 5.5 *was* curated on 09-29, on a branch that was never pushed. +- **The Runtime deletion timeout behind the nightly failure: no public match.** Nothing turned up in the SDK or starter-toolkit trackers, or in a GitHub-wide search for the error string. The nearest is terraform-provider-aws #47399 (VPC ENIs block delete), which doesn't fit, because our Runtime is `networkMode: 'PUBLIC'` (`inference-agentcore-construct.ts:313`) — https://github.com/hashicorp/terraform-provider-aws/issues/47399 + +#### Strands Agents + +- **`python/v1.57.2` (10-01)**. We are pinned at 1.55.0; PR #1367 targets 1.57.1 — https://github.com/strands-agents/harness-sdk/releases/tag/python/v1.57.2 + - **#4698 (breaking): `BidiAgent.__init__` drops `session_manager` and `**kwargs`**, so an unknown keyword raises `TypeError` — https://github.com/strands-agents/harness-sdk/pull/4698 + - *relevance*: **verified on disk.** `voice_agent.py` passes `session_manager=self.session_manager` on `develop` (`:192-199`) **and on PR #1367's branch (`:255`)**. #1367's fix 2 depends on `TurnBasedSessionManager` seeing the `BidiAgent`. On 1.57.2 every voice session fails at construction. The import still succeeds, so the `BIDI_AVAILABLE` guard won't catch it. #4813 (merged 10-02, unreleased) points voice persistence at `SnapshotSessionManager` checkpoints instead. + - **#4707: Bidi graduates to `strands.bidi`.** `strands.experimental.bidi` now warns and is **removed in v1.60.0** — https://github.com/strands-agents/harness-sdk/pull/4707 + - *relevance*: the imports at `voice_agent.py:37-39`. This is a deadline, not a choice. + - **#4193 (ours) shipped**, so cache writes from Responses are now mapped upstream (commit `15399af09`). + - *relevance*: the `cache_write_tokens` half of `apis/shared/models/usage_normalization.py` retires on a ≥1.57.2 pin. That is the subtraction the standing watch has been waiting for. + - #4625: Bedrock replays a non-dict `toolUse.input` as `{}` instead of failing the whole request. It changes bytes only for history that used to fail outright, so it is not a new cache miss. + - #4694: context-window limits for Sonnet 5.5, Opus 5.5 and Fable 5.1, which feed our model-relative compaction thresholds. +- **Merged, unreleased**: + - **#4499**: no hang when the connection drops mid-tool-call. That is our "a stalled turn holds the lease" class. + - #4658: `Agent.aux_model`. + - #4693: no `guardContent` wrap on blank text. + - https://github.com/strands-agents/harness-sdk/pull/4499 +- **Tracked items**: + - #4330 (retry backoff ignores cancellation): reproduced on main 10-02, with a fix proposed. + - #4407, #4340, #4367, #4176: quiet. + - #4201 was reworked to emit OpenAI's native `prompt_cache_breakpoint`, which is relevant to our OFF-by-default `apply_explicit_prompt_cache()`. +- **New bugs to watch**: + - #4713: `ConcurrentToolExecutor` swallows `BaseException`. + - #4795: an early stream close leaks the agent-as-tool `_busy` lock. + - #4622: an interrupted tool span has no output event, which breaks AgentCore Evaluations parsing. + +**Caching watch: answers** + +1. **#4618: no movement that matters yet.** It is still open, with no design doc and no PR. + - Two comments: ours (09-26) and a reply (09-28) with a 96-model matrix. It found that the convention **follows the API, not the model family**: Converse is disjoint, Mantle Responses/Chat is subset. It also found a ConverseStream double-count bug on GPT-5.4/5.5 and Grok 4.6. + - In v1.57.2, `models/bedrock.py` passes `usage` through untouched. `_total_prompt_tokens` lives in `telemetry/metrics.py` and is used only at `event_loop.py:151`, as a read-side heuristic. + - **No adapter folds cache tokens into `inputTokens`.** A 1.57.x bump is safe on this axis. +2. **#4193: released in 1.57.2.** +3. **#3546: closed 09-28 by #4617** (shipped in 1.57.2). The broader normalization work moved to #4618. +4. **`CacheConfig`: no new field.** `_openai_cache.py` is unchanged. `_cache_strategy` still resolves `"auto"` only for `claude`/`anthropic` ids, so the **Nova coverage gap persists** upstream as well as here. +5. **No new cacheable Bedrock family upstream.** The probe was not run this week. + +#### Reference repo (aws-samples/sample-strands-agent-with-agentcore) + +- **PR #305 (09-25)**: Opus 5 → 5.5, GPT-5.6 → GPT-6, with Sol as the default. Saved legacy selections are migrated, and AG-UI is pinned — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/305 + - *applicability*: we already curated the same models (#1358). Their saved-selection migration is our retirement redirect, which our own notes say has not been verified end to end. +- **PR #311 (10-01)**: the Agent Registry client moved from `bedrock-agentcore-control` to a new `agent-registry-control` client, with renamed fields (`descriptorType` → `recordType`, and others) and renamed IAM actions. The default model became Sonnet 5.5 — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/311 + - *applicability*: **none today.** A grep of `backend/src` and `infrastructure/lib` for registry calls finds nothing. Note it for any future marketplace/registry integration. +- Dependabot #303 (strands + agentcore) is still open, and now also carries uvicorn 0.53 (opt-in HTTP/2). AG-UI reached 1.0 (#299/#308/#310). +- ✅ Their A2A `AgentCard` still has `streaming=True` (`code-agent/src/main.py:959`). + +#### MCP ecosystem + +- **python-sdk: no release** (v2.2.0 / v1.30.0 remain latest). Skills extension PR #3485 is still open, with changes requested 09-24. +- **SEP-3392 Extension Lifecycle (opened 10-01)**: `experimental-ext-*` / `beta-ext-*` (may break) → `ext-*` (stable, changes only with a protocol revision) — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/3392 + - *relevance*: it would formally set the stability of the `ext-apps` contract our MCP Apps host implements. +- **SEP-3371 Consistent SDK extension points (updated 10-02)**: standard hooks so Apps and Skills ship as separate packages — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/3371 + - *relevance*: long run, a candidate to retire the `ClientSession` substitution in `integrations/mcp_apps.py`. +- **#3364**: clarifies extension negotiation in `initialize` for pre-2026-07-28 versions. That is the flow our `mcp<2` client uses. +- **SEP-2643** (structured authorization denials) and **SEP-2848** (async approval for tool calls, draft) are the protocol-level shapes of our `oauth_required` and tool-approval interrupts. Watch only — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2848 +- Governance: #3336 (merged 09-28) requires a WG/IG discussion before a SEP. #3389 consolidates the file-uploads and filesystem WGs, taking on SEP-2631. +- ext-apps v2.0.2/v2.0.3: example-server security pins only (`pdfjs-dist`). #778 (title precedence) has had no activity since 09-18. + +#### FastMCP + +- **v4.0.10 (09-25)**: task-enabled tools behind search transforms and CodeMode, and nested tool calls run in the foreground — https://github.com/PrefectHQ/fastmcp/releases/tag/v4.0.10 + - *implications*: minor for our Lambda servers. **The upgrade floor stays ≥4.0.9.** +- **New, auth-relevant: #5362.** Verifier-only auth with `base_url` advertises a `resource_metadata` URL FastMCP doesn't serve. That could break OAuth discovery for a server behind the Gateway — https://github.com/PrefectHQ/fastmcp/issues/5362 +- **New, correctness**: + - #5304: `ResponseLimitingMiddleware` turns an error into success when it truncates. + - #5378: a sync tool that exceeds its timeout returns success. + - Both mean an MCP server can report success for a failed call. Our tool rail's "failed" count would undercount. +- Still open: #5134 (`session_id` per call, modern era only), #5237 (step-up 403), #5192 (`clientInfo` needed for extension detection; we always send it on 1.x). + +#### Agentic UI/UX patterns + +- **"Preparing X" vs "Running X" (LibreChat #16619 / #16455).** A tool card says "Preparing" while the model streams its arguments, and "Running" only once dispatched — https://github.com/danny-avila/LibreChat/pull/16619 + - *fit*: **direct port, and we have the gap (verified).** `describeToolCall` (`tool-rail/tool-summary.ts:170`) treats any `status === 'pending'` as running. That covers the whole argument-streaming window, which on a large tool input (an MCP App, a document write) is most of the wait. Both signals already exist: `agent_status` `tool_start` (consumed in `tool-insight.service.ts:207`) and `inputComplete` (`assistant-message.component.ts:134`). + - *where it'd land*: `tool-summary.ts` + the tool rail. No SSE change. +- **Key message parts by identity (assistant-ui #8684)**: index-keyed parts keep the previous occupant's state when parts reorder — https://github.com/assistant-ui/assistant-ui/pull/8684 + - *fit*: pattern-only. **Gap verified**: `assistant-message.component.ts:161` is `@for (block of displayBlocks(); track $index)`. If a block is regrouped mid-stream, a tool rail's expanded state, or an MCP App iframe, can attach to the wrong block, and an iframe can be torn down and remounted. + - *where it'd land*: track on `toolUseId` for tool blocks, type+ordinal otherwise. +- **"N failed" opens the failed rows (LibreChat #16390)**: partial gap. `tool-rail.component.html:51-56` renders the count as text inside the toggle — https://github.com/danny-avila/LibreChat/pull/16390 +- **MCP Apps "split" display mode (ext-apps #739, draft)** and **WebMCP tools from a View (#798, breaking prototype)**. Spec watch only. We offer `['inline','fullscreen']` (`mcp-app-bridge.ts:577-579`). +- Smaller: + - Vercel #21879 clears stale pending/denied approvals once a tool has output. Unverified against `tool-approval.service.ts`. + - Vercel #21862: two clients refreshing one MCP OAuth token can delete the new token. Backend; worth a read against `oauth_token_cache`. + +#### Frontier model announcements + +- **Claude Sonnet 5.5 (09-28)**. The release notes list five breaking changes from Sonnet 5: + - Thinking blocks are account-bound. + - Effort levels are recalibrated. + - Thinking is turned off with `thinking: {"type": "between_tools"}`, not `"disabled"`. + - https://platform.claude.com/docs/en/release-notes/overview.md · https://platform.claude.com/docs/en/build-with-claude/effort + - *relevance*: the unpushed curation branch should be checked against all three before it is pushed. +- **Prompting Opus 5.5** (doc, HN 09-28): the default effort is `medium`, thinking cannot be disabled, and **`tool_choice` `any`/`tool` returns 400** — https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5 + - *relevance*: any path that forces a tool on the *user-selected* model would fail on Opus 5.5. Strands `structured_output` forces tool choice. Unverified whether any of our structured-output calls run on the user's model rather than a fixed side-channel model. +- **Sonnet 4.5 deprecated (09-30), retiring 2026-11-30.** Verified: no `claude-sonnet-4-5` row in `curated-models.ts`. Prod catalog rows are unchecked. +- **Per-message effort beta (`mid-conversation-output-config-2026-07-01`)**: changes effort without busting the cache. Opus 5.5, Sonnet 5.5, Fable 5.1. Bedrock availability is **unverified**. It addresses our "effort switch busts the prompt cache" note directly. +- **Gemini 4 Argon (09-30)**: cached input is 95% off. Not on Bedrock; low relevance — https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/ +- **OpenAI is withholding a model over safety concerns** (NYT, 09-28; the URL slug says "astra"). Paywalled and unread. Whether it is the same as our curated `us.openai.gpt-6-astra` is **UNVERIFIED** — https://www.nytimes.com/2026/09/28/technology/openai-astra-safety.html + +#### Agent harness patterns + +- **Claude Code 2.1.284: compact again when the compacted request is still too long**, keeping less recent history — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md + - *relevance*: our only overflow recovery is Strands' `SlidingWindowConversationManager.reduce_context` (`core/agent_factory.py:384-385`). It slides `agent.messages` behind the compaction anchor's back. Routing an overflow through a forced `TurnBasedSessionManager` compaction would keep the anchor intact. Note opencode reverted its own attempt last week (#51273) as "hard to get right". **Monitor.** +- **Claude Code 2.1.287**: a CLAUDE.md attached twice after resume/compaction, a reminder landing inside a tool result after compaction, and model switches rewriting earlier MCP tool announcements. + - *relevance*: the same class as our byte-stability contract. One regression test is worth having: injected guidance (e.g. `ask_user_question` `SYSTEM_PROMPT_GUIDANCE`, `chat_agent.py:144`) never lands in history. +- **2.1.283–2.1.286**: + - A tool returning a non-string caused 400s. + - A brief 404 from a stateless remote MCP server no longer disables it for the session. + - Headless mode doesn't wait on the slowest MCP server. + - These are ideas to check against our offloader and MCP pre-flight, not confirmed gaps. +- **pydantic-ai v2.53.0**: `AskUser` gains a deferred mode and an answer timeout; on timeout the model gets `TIMED_OUT` and carries on (#9509). It also adds `ToolCallJudge` (#9041) — https://github.com/pydantic/pydantic-ai/pull/9509 +- **Cookbook bf24d45 (09-28)**: model-agnostic caveats in `cost_optimization`. + - A top-level effort change invalidates the cache. + - On preserved-thinking models, rewriting or compacting earlier history invalidates later thinking blocks **and can get the next request rejected**. + - Some newer models price cache reads below 0.1×. + - https://github.com/anthropics/claude-cookbooks/commit/bf24d45 + - *relevance*: Top 5 #2. +- Anthropic engineering blog: nothing in window. + +#### opencode (anomalyco/opencode) + +v1.18.33 and v1.18.34 carried namespaced session headers and macOS signing only. Merged PRs: + +- **[Tooling] #52740: keep going when a question can't be shown.** The tool returns "continue without an answer: make reasonable assumptions and state them" — https://github.com/anomalyco/opencode/pull/52740 + - *relevance*: **gap verified in our headless harness.** + - `RunStatus` is `completed | error | timeout | oauth_required` (`apis/shared/harness/models.py:20`). + - The accumulator tracks only `oauth_required`. + - Any stream that reaches `done` is `completed` (`harness/runner.py:266-272`). + - A scheduled run whose model calls `ask_user_question` (registered whenever the flag is on, `tool_registry.py:117`) would record `completed` with a question nobody will answer. + - ⚠️ Unverified: whether a scheduled run's `enabled_tools` ever includes it. Top 5 #5. +- **[Tooling] #52747**: honour the `disable-model-invocation` skill frontmatter. The skill stays user-invocable but leaves ``. That is a smaller cacheable prefix, and Claude Code-authored skills import cleanly. +- **[Context] #52664**: report a compaction refusal with its category. We already fall back (`compaction_summary.py:257-260`, shipped this week); logging the category is the only addition. +- **[Cost] #52528**: Bedrock tool-result images are placed by model family. They stay inline for Claude and Nova and move beside the text for others. Relevant if a non-Claude Bedrock model gets an image-returning MCP tool. + +#### LibreChat + +- **v0.8.8 stable (10-01)** — https://www.librechat.ai/changelog/v0.8.8 + - The agent-control headline (interrupt, steer, queue, ask, approve, live activity, context usage, costs) matches what we have shipped. + - Three things we don't have: **manual compaction**; **background tasks as tool cards** with a stop control (#16348, #16407), relevant to the background-handoffs spec on this branch's parent; and **scheduled chats with "Bound Scheduled MCP Consent"** (#16652, 10-02). The last bears on the open "scheduled runs ungated" note. + - Small layout patterns: #16621 (pending steering stays in the chat column), #16637 (collapse folds from their rails). + +#### Pricing / quota + +- **AgentCore Runtime V2**: see above. Verified against the page source. +- **Bedrock model pricing page**: the tables are rendered by JavaScript, and no rates came back through the fetch. Nothing could be verified for Sonnet 5.5, GPT-6.1 or Grok 4.7. Per `CLAUDE.md`, the Price List API (the right offer file) plus the AWS model card is the source for any curation. + +#### Community + GitHub issues + +- **"You said no MCP"** (HN 676 pts, 09-30), an MCP-skeptic debate (unread) — https://news.ycombinator.com/item?id=49906637 +- **Figma limits MCP access to allow-listed clients** (HN 183 pts, 10-01) — https://news.ycombinator.com/item?id=49922729 + - *relevance*: a third-party MCP server can lock out custom clients like ours. A risk for the external MCP catalog, not an action. +- AgentCore SDK / starter toolkit: see AWS above. Reddit skipped (domain-blocked, decisions.md 2026-05-18). + +#### Cookbook / courses + +- See bf24d45 under harness patterns. The repo is now `anthropics/claude-cookbooks`; the old name redirects. + +#### Seasonal + +- Out of window. No seasonal sources scanned. re:Invent is late Nov / early Dec. + +### Patterns worth considering + +- **Say "preparing" until a tool is dispatched.** LibreChat separates argument streaming from execution. We already receive both signals and collapse them into one label. + - **Where**: LibreChat #16619/#16455. + - **Fit**: direct; one function plus a spec. It pairs naturally with the `track $index` fix in the same component tree. + - **Verdict**: **Worth trying.** Not in the Top 5 only because it is cosmetic next to the five below; it is a good PR to bundle. +- **Append-only tool changes, now partially on Bedrock.** The structural answer to mid-session `toolConfig` rewrites. + - **Fit**: blocked on our default models (Haiku 4.5 and Sonnet 4.6 return 400) and on Converse support. + - **Verdict**: **Monitor.** Re-check when Converse or Haiku gains it. +- **A headless run with no human never pauses for one** (opencode #52740, pydantic-ai #9509). See Top 5 #5. + +## Internal Audit + +> Every number below comes from the command quoted next to it. + +### Activity (last 7 days) + +- **PRs merged into `develop`: 100** — `gh pr list --base develop --state merged --search "merged:>=2026-09-25" --limit 200 --json number | jq length` +- **Non-merge commits on `develop`: 121; reverts: 0** — `git log origin/develop --since=2026-09-25 --oneline --no-merges | wc -l`; `… | grep -ci revert` +- **Releases**: **1.25.0** (09-25) and **1.25.1** (09-26) — `grep -n "^## \[" CHANGELOG.md` +- **Open PRs against `develop`: 1**: #1367, the paired bump. 0 comments, opened 09-27, **mergeable state `CONFLICTING`**, 7/7 checks green at last run — `gh pr list --base develop --state open`; `gh pr view 1367 --json mergeable` +- **Issues created in window: 15, all open** (9 are enhancement requests #1305–#1308 and #1379–#1389; #1390/#1391 are test-infra; #1372 is a KB reconciler bug; #1340 is an admin-selectable summarizer model). **Closed in window: 0. Open repo-wide: 41** — `gh issue list --state all --search "created:>=2026-09-25"`; `--state closed --search "closed:>=2026-09-25"` +- **CI failures since 09-25**: `Nightly Build & Test` → **2** (10-01, 10-02, consecutive); `CI` (pull_request) → 1 (09-26, resolved pre-merge) — `gh run list --status=failure --limit 40` +- **`develop`-push deploys**: Backend 23/23, Frontend 14/14, Platform Stack 3 success / 1 cancelled. **Zero failures** — `gh run list --branch develop --event push --limit 80` + +**Shape of the week** (commit scopes): compaction (extract-then-compress on by default, salvage, concurrency), KB byte accounting and sync, memory (LTM prefetch overlap, relevance calibration), the TTFT turn-path programme (shared agent-build session default-on, sub-stage timing, first-token clock), observability, voice (voice + mic picker, Nova Sonic metering), CI path filtering. + +### Last week's Top 5: outcome, read against disk + +| # (research/2026-09-25) | Outcome | Evidence | +|---|---|---| +| 1. A/B V2 Runtime in dev | **Planned, not started** | `docs/specs/agentcore-runtime-v2.md` (#1376). Plan step 1 (B1 allow-list) not done: `grep platformVersion scripts/build/deploy-runtime-image-if-changed.sh` finds nothing | +| 2. Non-`end_turn` = failed side-channel call | ✅ **Shipped** | `7f1e8fd4` | +| 3. Guard Stop against a finished turn | ✅ **Shipped** | `41dd4471` (+ `5956eb20` page-hide) | +| 4. Paired Strands/agentcore/boto bump | ⏳ **PR open, stalled** | #1367: 5 days, 0 reviews, now `CONFLICTING`, and already one release behind on all three packages | +| 5. Curate Opus 5.5 (+ measure Luna) | ✅ (a) **Shipped**; (b) not started | `f3c90799` (#1358); no Luna-vs-Nova-Micro A/B commit | + +**3 of 5 shipped, 1 planned, 1 stalled in review.** That is a sharp turn from last week's 0 of 5, and it confirms the review's diagnosis: research ideas convert once a forum ranks them. + +### Repeated friction signals + +- **Finished work stranded off `develop`** (2 occurrences). + - **Sonnet 5.5 curation** (`d708b358`, 09-29) exists only on the local branch `claude/sonnet-5-5-bedrock-dev-e9a13e`, in a worktree. It was never pushed and has no PR. The auto-memory index describes Sonnet 5.5 as "curated in code", which is true only of that worktree. + - **#1367**: written, tested and mutation-checked, then left unreviewed until `develop` moved under it. + - *Hypothesis*: work started from a task chip or worktree finishes in that session and has no hand-off step that ends in a PR. That is the "orphaned worktree diff" pattern already in feedback memory. + - *Fix candidate*: push and PR the Sonnet 5.5 branch after checking it against the 5.5 breaking changes (Top 5 #2). Rebase #1367 (Top 5 #3). +- **The nightly's ephemeral stack cannot tear down an AgentCore Runtime reliably** (1 occurrence, 2 failed nights). + - 10-01 teardown: `InferenceApiAgentCoreRuntime… Request timed out while deleting AWS::BedrockAgentCore::Runtime (NotStabilized)`, which left the stack in `DELETE_FAILED`. + - 10-02 deploy: `nightly-develop-PlatformStack … is in DELETE_FAILED state and can not be updated`. + - The teardown already treats S3 vector buckets as best-effort ("Failed to delete vector bucket …, skipping"). It has no equivalent for the Runtime. + - *Fix candidate*: Top 5 #4. +- **Backend pin drift** (4th consecutive week). This week the PR exists, which is progress, but it aged out in review. + +### Version-pin lag + +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| `strands-agents` | 1.55.0 | **1.57.2** 🆕 (10-01) | 5 / 24d | #1367 targets 1.57.1. ⚠️ **1.57.2 removes `BidiAgent(session_manager=)`**, which our voice agent passes. 1.57.2 also ships #4193 (cache writes). `strands.experimental.bidi` is removed in 1.60 | +| `strands-agents-tools` | 0.8.8 | 0.8.9 | 1 / 28d | ⛔ 0.8.9 assumes `mcp` 2.x | +| `bedrock-agentcore` | 1.21.0 | **1.24.0** 🆕 (09-28) | 4 / 57d | Extra needs strands ≥1.56 and `mcp<2`, the same pairing as 1.23.1 | +| `boto3` / `botocore` | 1.43.68 | **1.43.107** 🆕 | 39 / 53d | Patch-only. #1367 targets 1.43.103 | +| `fastapi` | 0.136.1 | **0.142.2** 🆕 (09-30) | 29 / 162d | ⚠️ crosses the declared 0.137 break. 0.136.3 is the last patch in our line. Oldest active backend pin | +| `pydantic` (lock) | 2.12.5 | 2.13.5 | 6 / 310d | Minor | +| `mcp` (lock, `<2`) | 1.28.1 | 2.2.0 (1.x tail 1.30.0) | 3 in 1.x / 98d | #1367 targets 1.30.0 | +| `openpyxl` | 3.1.5 | 3.1.5 | 0 | Current | +| `@angular/core` | 21.2.20 | 22.2.1 🆕 (21 tail **21.2.25** 🆕, `v21-lts`) | 5 on tail / 50d | 22 needs TS 6.0.x | +| `@analogjs/vite-plugin-angular` | 3.0.0-alpha.53 | alpha.90 | 37 / 156d | — | +| `@analogjs/vitest-angular` | 3.0.0-alpha.30 | alpha.90 | 60 / 172d | Drifted from vite-plugin (alpha.30 vs alpha.53) | +| `vitest` | 4.1.11 | **5.0.3** 🆕 | major / 45d | Analog no longer blocks 5 (alpha.89) | +| `typescript` | 5.9.3 | 7.0.2 | 3 / 367d | — | +| `tailwindcss` | 4.2.4 | 4.3.3 | 4 / 164d | Lowest-risk bump on the board | +| `aws-cdk-lib` | 2.265.0 | **2.272.0** 🆕 (09-30) | 7 / 50d | 🆕 **2.272.0 is the first release with a typed `CfnRuntime.platformVersion`** (verified in `bedrockagentcore.generated.d.ts`) | +| `constructs` | 10.6.0 | 10.8.1 | 5 / 193d | Move with `aws-cdk-lib` | +| `aws-cdk` (CLI) | 2.1128.0 | **2.1144.0** 🆕 | 16 / 107d | CLI-only | + +Method: PyPI JSON, `npm view` and `npm pack`, no WebFetch. + +### Retirement candidates + +- **The `addPropertyOverride('PlatformVersion', …)` step in the V2 plan** (`agentcore-runtime-v2.md` §5 step 2). `aws-cdk-lib` 2.272.0 types the property. Bump CDK and set `platformVersion` directly, keeping the "always set it explicitly, V1 or V2" rule. +- **The `cache_write_tokens` half of `usage_normalization.py`**, on any pin ≥1.57.2. Its docstring should name #4193 as the reason it exists, so the deletion is mechanical. +- **The W5 "instance-based Runtime arithmetic" queue entry ([2026-09-04]).** The V2 spec already says to hold it for committed-baseline pricing. That pricing is now published as a rate ($0.0132/GB-hour) with a launch window. Fold the entry into the V2 item rather than carrying it separately. +- **The [2026-09-25] "Strands 1.57 + agentcore 1.23.1 paired bump" queue entry**, once #1367 lands. Its retarget is Top 5 #3. + +### Risks introduced this week + +- **Strands 1.57.2 breaks voice at runtime, not at import** — https://github.com/strands-agents/harness-sdk/pull/4698 — *what breaks if ignored*: a "just bump to latest" edit to #1367 ships a voice mode that raises `TypeError` on every session start. The import-only CI check passes. +- **Preserved thinking vs history rewrites on Opus 5.5** — https://github.com/anthropics/claude-cookbooks/commit/bf24d45 — *what breaks if ignored*: the first compaction, or tool-result offload, on a long Opus 5.5 conversation may invalidate its thinking blocks. That costs a full re-write at best and a rejected request at worst. Opus 5.5 is curated in prod. +- **Opus 5.5 rejects forced `tool_choice`** — https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5 — *what breaks if ignored*: any structured-output call that runs on the user-selected model returns 400 on Opus 5.5. Unverified whether one exists. +- **The nightly stays red until the stuck stack is cleared by hand** — *what breaks if ignored*: the E2E signal goes dark, and the "14+ consecutive greens" baseline the reviews lean on stops meaning anything. +- **FastMCP servers can report success for failed calls** (#5304, #5378) — *what breaks if ignored*: tool-rail failure counts and `agent_status ok=false` undercount for servers that use `ResponseLimitingMiddleware` or sync timeouts. + +## Ideas — Top 5 (ranked) + +| # | Idea | Surface | Effort | Impact | Subtracts? | Unlocks? | +|---|---|---|---|---|---|---| +| 1 | Re-base the V2 Runtime plan on the published price, land B1, and use CDK's typed `platformVersion` | infra + docs | L | H | yes: the untyped property override; the separate W5 instance-SKU entry | a pre-registered go/no-go threshold (≤56% GB-hours) before the A/B | +| 2 | Verify compaction and offload on preserved-thinking models (Opus 5.5) before more of them ship, then push the stranded Sonnet 5.5 curation | backend + frontend | L | H | no: addition, justified because it is a correctness check on a model already in prod | — | +| 3 | Land #1367 now at 1.57.1/1.24.0; take 1.57.2 as its own PR with the voice change and the `usage_normalization` deletion | backend | M | M | yes: the cache-write shim; the `strands.experimental.bidi` import before 1.60 | Bidi resilience across reconnects; #4499 when released | +| 4 | Make the nightly teardown survive a Runtime that won't delete | CI / scripts | L | M | no: addition, justified because the nightly is the only E2E signal | — | +| 5 | Headless runs never pause for a human: withhold the interactive tools or classify the pause | backend | L | M | yes: removes `ask_user_question` (and `request_user_login`) from the headless `toolConfig` | — | + +### 1. Re-base the V2 Runtime plan on the published price, land B1, and use CDK's typed `platformVersion` + +- **Source**: https://aws.amazon.com/bedrock/agentcore/pricing/ (verified 2026-10-02); `aws-cdk-lib` 2.272.0; queue ▸ [2026-09-25] V2 A/B; `docs/specs/agentcore-runtime-v2.md`. +- **Surface area**: `docs/specs/agentcore-runtime-v2.md` §1/§4; `scripts/build/deploy-runtime-image-if-changed.sh:215` (`ALLOWED`); `infrastructure/package.json` (aws-cdk-lib 2.265.0 → 2.272.0, constructs alongside); `infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts:297`. +- **Change**: + - (a) Write the rates into §4, with the break-even stated **before** the A/B: memory ≤56% of V1 GB-hours, CPU ≤70% of V1 vCPU-hours. Also record that idle memory is reclaimed after 120 s. That is far below our 900 s `idleRuntimeSessionTimeout`, so the idle-tail term of the cost model mostly vanishes. + - (b) Land plan step 1 (B1): add `platformVersion` to `ALLOWED`, assert it after the update, and fail fast if the runner's CLI lacks the field. It is harmless on V1. + - (c) Make the CDK bump the vehicle for step 2 and set the typed property, not an override. +- **Subtracts**: the `addPropertyOverride` workaround, and the separately-tracked W5 instance-SKU entry (the committed baseline is now a published rate). +- **Unlocks**: a decision rule, written down before the data arrives, which makes the dev A/B decisive instead of interpretive. +- **Effort × Impact**: Low × High +- **Verdict**: **Worth trying.** ⚠️ 1.79× on memory means V2 is **not** an obvious win for a long, busy session. The bet is on the idle tail. The Cost Explorer sync (#1235) must show whether V2 bills under a distinct usage type, or the dev week can only be compared before/after. + +### 2. Verify compaction and offload on preserved-thinking models, then push the stranded Sonnet 5.5 curation + +- **Source**: https://github.com/anthropics/claude-cookbooks/commit/bf24d45 (09-28); https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5; Sonnet 5.5 release notes; local commit `d708b358`. +- **Surface area**: `agents/main_agent/session/turn_based_session_manager.py` (checkpoint application, truncation anchor); `core/tool_result_offload.py`; the curated Opus 5.5 row (`curated-models.ts`); the unpushed Sonnet 5.5 row. +- **Change**: + - (a) In dev, drive one Opus 5.5 conversation (`us.*`, which the dev SCP allows) past the compaction threshold with at least one offloaded tool result. + - Record whether the next call succeeds, and its `cacheStatus` on the `C#` rows. + - If Bedrock rejects it, compaction on preserved-thinking models must keep earlier thinking blocks byte-identical or drop whole assistant turns, never rewrite inside one. + - (b) Grep for forced `tool_choice` / `structured_output` on the user-selected model. Opus 5.5 returns 400 on `any`/`tool`. + - (c) Check the Sonnet 5.5 branch against the 5.5 breaking changes (`between_tools` replaces `disabled`, recalibrated effort, account-bound thinking), then push and open a PR. +- **Subtracts**: no. Justified because this is a correctness check on a model already curated in prod, and the failure mode, a rejected request deep in a long conversation, is the costliest place to find it. +- **Effort × Impact**: Low × High +- **Verdict**: **Worth trying.** ⚠️ "Can get the next request rejected" is the cookbook's wording for the Claude API. Bedrock behaviour is **unverified**, and (a) is the cheapest way to find out. + +### 3. Land #1367 now at 1.57.1 / agentcore 1.24.0; take 1.57.2 as its own PR + +- **Source**: https://github.com/strands-agents/harness-sdk/releases/tag/python/v1.57.2; https://github.com/strands-agents/harness-sdk/pull/4698; #1367 (`CONFLICTING`, 0 reviews). +- **Surface area**: `backend/pyproject.toml`, `backend/uv.lock`, the Lambda requirements files; `agents/main_agent/voice_agent.py:37-39,192-199`; `apis/shared/models/usage_normalization.py`. +- **Change**: + - PR A (now): rebase #1367. Optionally move agentcore to 1.24.0 and boto to latest; the pairing test already guards both. Dev-validate one text turn **and one voice turn**, then merge. **Don't** move strands past 1.57.1 in this PR. + - PR B (next): + - strands 1.57.2. + - Remove `session_manager=` from `BidiAgent` and decide where voice history persists. #4813 points at `SnapshotSessionManager` checkpoints; until then, persist from the `VoiceWireAdapter` stream #1367 already introduces. + - Switch imports to `strands.bidi` before 1.60. + - Delete the `cache_write_tokens` half of `usage_normalization.py`, checking one cached Responses call prices identically before and after. +- **Subtracts**: yes. The cache-write shim, the experimental-bidi import, and the stale queue entry. +- **Unlocks**: 1.57.2's Bidi reconnect resilience (#4677); #4499 (no hang on a dropped MCP connection) on the next release after it. +- **Effort × Impact**: Med × Med +- **Verdict**: **Worth trying.** The pairing work is done and verified. The risk now is only that it keeps aging. #4618 did not move, so neither PR inherits the cost-convention hazard. + +### 4. Make the nightly teardown survive a Runtime that won't delete + +- **Source**: Nightly runs on 10-01 (teardown `DELETE_FAILED`, `NotStabilized` on `AWS::BedrockAgentCore::Runtime`) and 10-02 (deploy refused on the stuck stack). +- **Surface area**: the nightly teardown script (the one that already skips undeletable vector buckets) and the nightly deploy step. +- **Change**: + - (a) One-off: delete the Runtime out of band, then `delete-stack --retain-resources` or a plain retry, to unstick `nightly-develop-PlatformStack`. + - (b) Durable: + - Before `delete-stack`, delete the Runtime via `bedrock-agentcore-control delete-agent-runtime` and poll until it is gone. + - On `DELETE_FAILED` naming only the Runtime, retry the delete once and then retain it. + - Make the deploy step detect a `DELETE_FAILED` predecessor and clear it before deploying, so one bad teardown costs one night, not every night after. +- **Subtracts**: no. Justified because the nightly is the only end-to-end signal, and the reviews lean on its green streak. +- **Effort × Impact**: Low × Med +- **Verdict**: **Worth trying.** (a) is urgent ops and doesn't need a review slot. ⚠️ Root cause is unknown, and nothing matches upstream. If it recurs, file it with AWS with the request token from the log. + +### 5. Headless runs never pause for a human + +- **Source**: https://github.com/anomalyco/opencode/pull/52740; https://github.com/pydantic/pydantic-ai/pull/9509; verified on disk. +- **Surface area**: `apis/shared/harness/models.py:20` (`RunStatus`), `apis/shared/harness/sse.py`, `apis/shared/harness/runner.py:266-272`; the scheduled-runs request that sets `enabled_tools`; `agents/main_agent/tools/tool_registry.py:117`. +- **Change**: preferred: **don't offer interactive tools on the headless path.** Strip `ask_user_question` and `request_user_login` from `enabled_tools` whenever the harness builds a run. Belt and braces: teach the accumulator `user_question_required`, `browser_login_required` and `tool_approval_required`, and classify a `done` that carries one as a new `needs_input` status instead of `completed`. +- **Subtracts**: yes. Two tool specs leave the headless `toolConfig`, so each scheduled run's prefix is smaller. +- **Effort × Impact**: Low × Med +- **Verdict**: **Worth trying.** ⚠️ Unverified end to end: whether any scheduled run's `enabled_tools` actually includes `ask_user_question` today. The classification gap is verified either way, because a paused turn reaches `done`. + +## Take + +**The ecosystem handed us two price tags this week, and both argue for measuring before moving.** V2's memory rate is 1.79× V1's, so the A/B now has a threshold it must clear rather than a story to confirm. Opus 5.5's always-on thinking may not tolerate the history rewrites our whole cost strategy is built on. Internally the forum worked: 3 of last week's 5 shipped, against 0 the week before. The failure mode has moved downstream, to finished work that never reaches `develop` (the Sonnet 5.5 branch, #1367 in review). **What Phil would notice first**: the nightly going green again, which takes one manual delete (Top 5 #4a), and voice mode surviving the next Strands bump because it was split from the safe one (#3). + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS What's New (RSS, curl) | https://aws.amazon.com/about-aws/whats-new/recent/feed/ | 2026-10-02 | 6 | +| 2 | AWS ML blog (RSS, curl) | https://aws.amazon.com/blogs/machine-learning/feed/ | 2026-10-02 | case studies only | +| 3 | AgentCore pricing (WebFetch + curl verify) | https://aws.amazon.com/bedrock/agentcore/pricing/ | 2026-10-02 | V2 rates | +| 4 | Bedrock pricing | https://aws.amazon.com/bedrock/pricing/ | 2026-10-02 | 0 (JS-rendered) | +| 5 | Bedrock Managed Agents preview | https://aws.amazon.com/about-aws/whats-new/2026/09/bedrock-managed-agents-preview/ | 2026-10-02 | 1 | +| 6 | Model launches (What's New) | https://aws.amazon.com/about-aws/whats-new/2026/09/claude-sonnet-5-5-aws/ · /openai-gpt-6-1-sol-on-amazon-bedrock/ · /amazon-bedrock-grok-4-7/ · /openai-gpt-6-astra-ultrafast-on-amazon-bedrock/ | 2026-10-02 | 4 | +| 7 | AgentCore SDK (gh) | https://github.com/aws/bedrock-agentcore-sdk-python/releases/tag/v1.24.0 · /issues/681 | 2026-10-02 | 2 | +| 8 | Starter toolkit + TF provider (gh) | https://github.com/hashicorp/terraform-provider-aws/issues/47399 | 2026-10-02 | 0 matches for the nightly error | +| 9 | Strands 1.57.2 + PRs (gh) | https://github.com/strands-agents/harness-sdk/releases/tag/python/v1.57.2 · /pull/4698 · /pull/4707 · /pull/4499 · /pull/4625 · /issues/4618 · /issues/4330 | 2026-10-02 | 1 release, ~15 PRs/issues | +| 10 | Reference repo (gh) | https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/305 · /pull/311 · /pull/303 | 2026-10-02 | 2 merged, 1 open | +| 11 | MCP spec (gh) | https://github.com/modelcontextprotocol/modelcontextprotocol/pull/3392 · /pull/3371 · /pull/3364 · /pull/2643 · /pull/2848 · /pull/3336 · /pull/3389 | 2026-10-02 | 7 | +| 12 | python-sdk / ext-apps / ext-skills (gh) | https://github.com/modelcontextprotocol/python-sdk/pull/3485 · https://github.com/modelcontextprotocol/ext-apps/releases/tag/v2.0.3 · https://github.com/modelcontextprotocol/ext-skills/pull/136 | 2026-10-02 | 0 releases; 2 example-server releases | +| 13 | MCP blog (RSS, curl) | https://blog.modelcontextprotocol.io | 2026-10-02 | 0 in window | +| 14 | FastMCP (gh) | https://github.com/PrefectHQ/fastmcp/releases/tag/v4.0.10 · /issues/5362 · /issues/5304 · /issues/5378 · /issues/5134 | 2026-10-02 | 1 release, 4 issues | +| 15 | LibreChat (gh + changelog) | https://www.librechat.ai/changelog/v0.8.8 · https://github.com/danny-avila/LibreChat/pull/16619 · /pull/16390 · /pull/16652 | 2026-10-02 | v0.8.8 + 5 PRs | +| 16 | assistant-ui (gh) | https://github.com/assistant-ui/assistant-ui/pull/8684 · /pull/8723 | 2026-10-02 | 2 | +| 17 | Vercel AI SDK (gh) | https://github.com/vercel/ai/pull/21879 · /pull/21862 | 2026-10-02 | 2 | +| 18 | ext-apps PRs (gh) | https://github.com/modelcontextprotocol/ext-apps/pull/739 · /pull/798 | 2026-10-02 | 2 drafts | +| 19 | NN/g | https://www.nngroup.com/articles/disclose-ai-paced/ | 2026-10-02 | 2 titles, index only | +| 20 | Claude release notes / docs | https://platform.claude.com/docs/en/release-notes/overview.md · /build-with-claude/effort · /prompt-engineering/prompting-claude-opus-5-5 | 2026-10-02 | 4 | +| 21 | Bedrock Claude tool-use guide | https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages-tool-use.html | 2026-10-02 | 2 betas | +| 22 | Anthropic news (Sonnet 5.5 page) | https://www.anthropic.com/news/claude-sonnet-5-5 | 2026-10-02 | 404 | +| 23 | Google / OpenAI / Meta (search) | https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/ · https://www.nytimes.com/2026/09/28/technology/openai-astra-safety.html | 2026-10-02 | 2 (NYT unread) | +| 24 | HN (Algolia API, curl) | https://news.ycombinator.com/item?id=49906637 · /item?id=49922729 · /item?id=49913571 | 2026-10-02 | 4 | +| 25 | Claude cookbooks (gh) | https://github.com/anthropics/claude-cookbooks/commit/bf24d45 | 2026-10-02 | 1 | +| 26 | Claude Code CHANGELOG (gh) | https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md | 2026-10-02 | 2.1.283–2.1.287 | +| 27 | Anthropic engineering | https://www.anthropic.com/engineering | 2026-10-02 | 0 in window | +| 28 | pydantic-ai / LangChain (gh) | https://github.com/pydantic/pydantic-ai/pull/9509 · https://github.com/langchain-ai/langchain/pull/40886 · /pull/40864 | 2026-10-02 | 3 | +| 29 | opencode (gh) | https://github.com/anomalyco/opencode/releases/tag/v1.18.34 · /pull/52740 · /pull/52747 · /pull/52664 · /pull/52528 | 2026-10-02 | 2 releases, 4 PRs | +| 30 | PyPI / npm registries (curl, npm) | https://pypi.org/pypi//json · https://registry.npmjs.org/ | 2026-10-02 | 17 deps | + +## Web Budget + +Used: **13 / 50** requests (target). AWS: 3 WebFetch, plus 1 curl to verify the V2 rates (counted). Frontier + community + cookbook: 7 (one 404). UI/UX: 1. Harness + opencode: 1. Strands, reference repo, MCP/FastMCP and version pins: 0 (all `gh`, `curl` or registry). +Skipped (unreachable / rate-limited): openai.com (blocks WebFetch); anthropic.com/news/claude-sonnet-5-5 (404); the Bedrock pricing tables (JS-rendered); NYT (paywall). +Skipped (other): Reddit (domain-blocked, decisions.md 2026-05-18); Cursor changelog and Linear blog (UI budget); the optional CloudFormation `describe-type` check; seasonal sources (out of window). +Notes: under target. No overage. diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index 15214c1c5..643299dd3 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,6 +5,44 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open +### [2026-10-02] Re-base the V2 Runtime plan on the published price, land B1, and use CDK's typed `platformVersion` +- **Source**: research/2026-10-02.md +- **Surface**: infrastructure (+ `scripts/build/deploy-runtime-image-if-changed.sh`, `docs/specs/agentcore-runtime-v2.md`) +- **Effort × Impact**: L × H +- **Subtracts**: yes. The plan's untyped `addPropertyOverride` (`aws-cdk-lib` 2.272.0 types `CfnRuntime.platformVersion`), and the separate [2026-09-04] W5 instance-SKU arithmetic entry (the committed-baseline rate of $0.0132/GB-hour is now published). +- **Unlocks**: a go/no-go threshold written down before the dev A/B. V2 memory is $0.0169/GB-hour against V1's $0.00945 (1.79×), so V2 wins only if billed GB-hours fall below ~56% of V1's. Idle memory is reclaimed after 120 s. +- **Status**: open. Plan step 1 (B1 allow-list) is still not started; it is harmless on V1. + +### [2026-10-02] Verify compaction and offload on preserved-thinking models (Opus 5.5), then push the stranded Sonnet 5.5 curation +- **Source**: research/2026-10-02.md +- **Surface**: backend + frontend +- **Effort × Impact**: L × H +- **Subtracts**: no. Addition, justified because it is a correctness check on a model already curated in prod. The cookbook (bf24d45) says rewriting or compacting history on preserved-thinking models invalidates later thinking blocks and can get the next request rejected. Opus 5.5 also returns 400 on forced `tool_choice`. +- **Status**: open. The Sonnet 5.5 curation (`d708b358`, 09-29) exists only on an unpushed local worktree branch. Check it against the 5.5 breaking changes (`between_tools`, recalibrated effort) before opening a PR. + +### [2026-10-02] Make the nightly teardown survive an AgentCore Runtime that won't delete +- **Source**: research/2026-10-02.md +- **Surface**: CI / scripts +- **Effort × Impact**: L × M +- **Subtracts**: no. Addition, justified because the nightly is the only E2E signal and it has been red since 10-01. The Runtime delete hit `NotStabilized`, and the stack has stayed in `DELETE_FAILED` since. +- **Status**: open. (a) One-off unstick of `nightly-develop-PlatformStack` (ops, no review needed). (b) Delete the Runtime out of band before `delete-stack`, retry and then retain on a Runtime-only `DELETE_FAILED`, and have the deploy step clear a `DELETE_FAILED` predecessor. + +### [2026-10-02] Headless runs never pause for a human: withhold interactive tools or classify the pause +- **Source**: research/2026-10-02.md +- **Surface**: backend +- **Effort × Impact**: L × M +- **Subtracts**: yes. `ask_user_question` and `request_user_login` leave the headless `toolConfig`. +- **Status**: open. Verified: `RunStatus` (`apis/shared/harness/models.py:20`) has no paused state, and `runner.py:266-272` marks any `done` as `completed`. Unverified: whether any scheduled run's `enabled_tools` includes `ask_user_question` today. + +### [2026-09-30] Backend test suite: shard across runners (#1390) and fix fixture scope (#1391) +- **Source**: Phil-initiated. The PR gate ran every suite on every pull request: a two-file docs PR paid the same 7–8 minutes as a backend change, and the backend pytest job alone was 7:02 with every other job under 2 minutes. Two things landed together in the path-filter PR: the `changes` job in `ci.yml` now runs only the suites a PR's paths can reach (`scripts/ci/classify-changes.sh`, fail-open, pinned by `test_ci_path_filter.py`), and the backend job runs `-n logical` — `-n auto` counts physical cores when psutil is installed, so the 4-vCPU runner started 2 workers. + - **What the profile showed.** Of the time measured across the suite (every phase ≥50 ms), fixture setup was 1,846 s against 399 s of test body. 77 files enter `mock_aws()` and create tables per test function; 5 files use any broader scope. One file alone: 21.8 s setup, 0.9 s test code. The top 25 files are ~950 s of ~2,245 s and are listed in #1391. + - **Two follow-ups, complementary.** #1390 shards the backend job across 3 runners with `pytest-split` (new dev dep, needs approval) — a CI-only win, floor ~2–2.5 min. #1391 scopes the moto fixtures at module level file by file — the bigger lever, and the only one that speeds up local runs. +- **Surface**: CI only for #1390; test files only for #1391. No production code. +- **Effort × Impact**: #1390 M × M (one workflow change plus a durations file with an owner). #1391 M × H, incremental (one PR per handful of files; track the setup:call ratio). +- **Subtracts**: no. Both reduce PR wall clock without dropping a test. +- **Status**: open; issues carry the acceptance criteria. Also worth a decision: `develop` has no required status checks, so the PR gate is advisory today. The job-level filter shape was chosen so that adding them later does not strand a skipped suite as "Expected". + ### [2026-09-26] Verify native token counts for `global.*` models in production — after #1343 reaches `main` - **Source**: Phil-initiated, from #1343 and its dev validation. - **The gap.** `base_foundation_model_id` never stripped `global.`, so every prod model counted with the heuristic. Since #1337, that means prod records no `prefixTokens` and no `contextBreakdown` at all. @@ -31,6 +69,16 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. 5. **Runtime logs.** No new error family. The OpenTelemetry "Token was created in a different Context" errors predate #1343 (about 4 per turn in dev) and have their own investigation. - **Done when**: steps 2–4 pass on at least one `global.*` Haiku 4.5 session. Record the counts here (no ids) and close the entry. +### [2026-09-26] Chore: settle the production knowledge base the byte-counter repair left for review — after the import-reservation fix reaches `main` +- **Source**: the prod `--apply` of the entry below adopted 12 of 13 knowledge bases. The 13th reported `reservedBytes 1664866 is not explained by 3 unsettled complete document(s) of 2164701 bytes`. Its two settled documents are Google Drive imports. Before the fix, the ingestion consumer read an import's `sizeBytes` as a request-time reservation, but imports never make one. So it committed 499,835 bytes out of the migrated corpus's reservation, and the corpus reservation came up short by exactly that amount. The same consumer path still runs on `develop` until the fix (`byte_cap.reserved_at_request`) ships. On the 12 adopted knowledge bases, which now reserve 0, every import or crawl drives `reservedBytes` negative. The fix and the `--settle-unexplained` mode of `backend/scripts/repair_managed_kb_byte_counters.py` land in the same PR. +- **Surface**: ops only, after the release. +- **Status**: blocked on release. ⛔ Settle only after the release that carries the fix is deployed. If you settle earlier, the next import breaks the counters again. A human runs this; production is read-only from agents. Dev was settled on 2026-09-26 with these same steps: 2 knowledge bases settled, and a re-run found nothing to do. + 1. Report the whole fleet: `AWS_PROFILE= backend/.venv/bin/python backend/scripts/repair_managed_kb_byte_counters.py --project-prefix --region us-west-2`. Any knowledge base that had an import between the adoption and the deploy now shows a negative `reservedBytes`. Its note says `explained exactly by N imported/crawled/synced document(s)`. + 2. Preview each flagged one: `... --settle-unexplained --agent `. For the known case, expect the `explained exactly by 2 imported` note and a settle to `storedBytes = totalBytes = 2664536` with `reservedBytes = 0`. A `not settled:` note means it isn't quiet (upload in flight, a delete mid-refund), so wait and retry. + 3. Apply: `... --settle-unexplained --agent --apply --confirm-prefix `. The script waits `--quiet-seconds` (10) and re-reads, then writes once in a single transaction. `refused` means something moved; re-run. + 4. Re-run step 1. The knowledge base should have no notes. +- **Done when**: step 4 is clean for every flagged knowledge base. Record the counts here (no ids) and close the entry. + ### [2026-09-26] Chore: repair managed-KB byte counters in production — after #1347 and #1348 reach `main` - **Source**: Phil-initiated, from the byte-cap accounting fix (#1347) and the reconciler read grant (#1348). A read-only look at production on 2026-09-25 found every promoted knowledge base (12) with `storedBytes=0` and its whole corpus still in `reservedBytes`. One of them also carries 9 unsettled legacy `failed` rows (about 12.4 MB). #1347 fixes the code but not the counters it already wrote. `backend/scripts/repair_managed_kb_byte_counters.py` fixes those. #1348 turns on the reconciler's daily `storedBytes` refresh, which never ran before because the Lambda had no read access to the documents bucket. - **Surface**: ops only. No code change. @@ -160,29 +208,12 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: - Pay-for-used Runtime memory — the first lever on the 73%-of-AICC line that is neither a token change nor a session-lifetime change. - P75 cold start ~1.9–2.0 s (vs 5.4–30 s on V1) — first-turn TTFT. -- **Status**: open. ⚠️ `aws-cdk-lib` 2.270.0 has no typed `platformVersion`; needs `addPropertyOverride('PlatformVersion', 'V2')` behind a dev-only config flag. Gate 1: does CFN accept the key today. Gate 2: does V2 change the `/ping`/`/invocations` contract, idle reaper, or 30 s init budget. Measure with the turn-latency EMF (#1184) and Cost Explorer sync (#1235). - -### [2026-09-25] Treat any non-`end_turn` stop reason as a failed side-channel call -- **Source**: research/2026-09-25.md ▸ Top 5 #2 — Claude Code 2.1.282 (refused compaction retries on fallback); verified on disk. -- **Surface**: backend — `agents/main_agent/session/compaction_summary.py:152`, `apis/shared/tool_summaries/summarizer.py:188` (both test only `stopReason == "max_tokens"`); check the session-title generator too. -- **Effort × Impact**: L × M -- **Subtracts**: no — addition, justified: a refusal or guardrail stop is currently accepted as a compaction summary and persists into the cacheable history until the next cut. The fallbacks already exist. -- **Status**: open. One predicate change per site + a stubbed `guardrail_intervened` test. Does not depend on knowing Bedrock's exact refusal stop reason. - -### [2026-09-25] Guard Stop against a turn that already finished -- **Source**: research/2026-09-25.md ▸ Top 5 #3 — assistant-ui #8282 (merged 2026-09-24); verified on disk. -- **Surface**: frontend — `session/services/chat/chat-http.service.ts:346` (`cancelChatRequest` — no completion check before `signalInterrupt` / `setLastTurnInterrupted`); loading only clears in `finalizeStream` from `onclose` (`:134`, `:283`); `stream-parser.service.ts:252` `isStreamCompleteFor()` has no external caller. -- **Effort × Impact**: L × M -- **Subtracts**: yes — one source of the false-"interrupted" marker (#988 lineage); gives an unused public method its caller (or delete it). -- **Status**: open — not reproduced live; widest window is a first turn where `session_title` arrives after `done`. Needs a spec for Stop-after-`done`-before-`onclose`. - -### [2026-09-25] Strands 1.57 + `bedrock-agentcore` 1.23.1 + boto — one paired bump (supersedes the [2026-09-18] 1.56 entry) -- **Source**: research/2026-09-25.md ▸ Top 5 #4 — https://github.com/strands-agents/harness-sdk/releases/tag/python%2Fv1.57.0 -- **Surface**: backend — `pyproject.toml` (strands + `[bidi]`, agentcore, boto3 ≥1.43.72), `uv.lock`; read first: `session/turn_based_session_manager.py` (Generic `SessionManager`), any `except EventLoopException` pause detection, `bedrock_responses.py:313`. -- **Effort × Impact**: M × M -- **Subtracts**: yes — Chat Completions cache-write gap in `usage_normalization.py:106` (reads no `prompt_tokens_details`) closes upstream via #4361; picks up #4371 (interventions honor interrupts) and #4426 (schema normalization no longer mutates caller specs); retires the [2026-09-18] 1.56 paired-pin entry. -- **Unlocks**: `handoff_to_user` vended tool (read before building another interrupt tool); Bedrock `requestTimeout`. -- **Status**: open. ⛔ Keep `strands-agents-tools` at 0.8.8 (0.8.9 needs mcp 2.x); `mcp` stays `<2`, target 1.30.0. ⚠️ A missed pairing presents on Runtime as "initialization time exceeded (30s)" → 502 — dev-validate a real turn. Run `probe_bedrock_cache_point_support.py --offline-only` and diff `strands/_context_manager/` per the standing watch. +- **Status**: open. The plan and pre-work findings are in **`docs/specs/agentcore-runtime-v2.md`**. + - ✅ **Gate 1 answered (2026-09-27).** The CFN schema has `PlatformVersion`, and it is **not** create-only, so flipping it is an in-place update and the runtime ID stays stable. The dev runtime reports `V1` today. + - ⚠️ **Blocker B1.** `scripts/build/deploy-runtime-image-if-changed.sh` rebuilds a full-replacement `update-agent-runtime` payload from an allow-list that omits `platformVersion`. Every `backend.yml` deploy could therefore revert the runtime to V1. Fix it before the flag goes on. + - ⚠️ **Gate 2 reframed.** V2 restores a snapshot of the running environment. The risk is less a contract change than import-time state cloned into every session. The concrete case is `runtime_health.py`, which stamps its idle clock at import (B2). + - ⚠️ **The turn-latency EMF (#1184) can't see a Runtime cold start**, because it starts at handler entry. Measure cold starts from the client side (`tests/load`) (B3). +- **Status 2026-10-02**: plan merged as `docs/specs/agentcore-runtime-v2.md` (#1376, 2026-09-28). **Re-based by the [2026-10-02] entry above** on the published V2 price (memory 1.79× V1 → wins only below ~56% of V1 GB-hours) and on `aws-cdk-lib` 2.272.0's typed `platformVersion`. B1 still unstarted. Second week as the review's #1 (reviews/2026-10-02.md ▸ #1). ### [2026-09-25] Curate Claude Opus 5.5 — and measure GPT-6 Luna as a side-channel candidate - **Source**: research/2026-09-25.md ▸ Top 5 #5 — https://aws.amazon.com/blogs/machine-learning/claude-opus-5-5-is-now-available-on-aws/ ; https://aws.amazon.com/about-aws/whats-new/2026/09/openai-gpt-6-sol-luna-on-amazon-bedrock/ @@ -193,6 +224,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - A cheaper top-tier Claude. - Possibly a Nova Micro successor for titles/summaries (measure, don't swap). - **Status**: open. ⚠️ Rates from the AWS **model card** only; `us.*` id in dev (SCP denies `global.*`); thinking cannot be disabled — check against the effort selector (an effort switch already busts the cache); bracket the real cache minimum before any caching A/B. +- **Status 2026-10-02**: (a) **SHIPPED** — #1358 (`f3c90799`, 2026-09-27); prod rows added 2026-09-26 with the retirement redirect verified on one retired-model agent. Opus 5.5 cache read is 0.05× input on the Price List API, not the 0.1× default. (b) Luna side-channel A/B **deferred to 2026-10-23** (reviews/2026-09-25.md ▸ #6); fold with issue #1340 (admin-selectable summarizer model) when due. ### [2026-09-21] Sweep session anatomies at fleet scale — one session already refuted a queued decision, and 1.23.0 may have moved the write:read ratio - **Source**: Phil-initiated, from a single prod session anatomy (`7f5f207f`, $21.84 / 48 Opus 5 calls, 2026-09-16→21). One session produced two results that no aggregate on the dashboard surfaces today, which is the argument for doing this at scale rather than one link at a time. @@ -216,6 +248,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Effort × Impact**: S × H for the one-off sweep (a read-only script over `C#` rows; the projection is already content-free). M × H if it earns a standing cohort panel. - **Subtracts**: no on its own — but it is the thing that lets us *close* cost items on evidence instead of leaving them open, and it already closed one. - **Status**: open — **run the sweep before the next cost decision.** The honest framing for the review: most of `7f5f207f` was an expensive conversation, not a bug, and its two real regressions were already fixed. The value was not the verdict, it was that one anatomy overturned a queued decision. Do that across a month of sessions. +- **Status 2026-10-02**: **question 2 ran on 2026-09-27, off-repo** (34,891 rows; see the PR-5 entry below for the result, which reversed that item's recommendation). Questions 1, 3, 4 and 5 and the `missCause` labelling did not run, and the script was not committed. reviews/2026-10-02.md ▸ #5 asks for the script in `backend/scripts/`, the remaining four answers, and says this is the last carry before `missCause` is declined. ### [2026-09-21] Standing watch — Strands' first-party `ContextManager` as it converges on what our compaction does - **Source**: conversation with Phil, 2026-09-21 — *"keep eyes on the compaction features delivered by Strands."* Prompted by research/2026-09-18 ▸ Strands, which reported the context-manager/offloading stack "fully released in 1.56.0" (#4118/#4146/#4187/#4231/#4254/#4282). **That framing is misleading and the correction is the headline here.** - **⚠️ It is already in our pin. Read this before planning a bump around it.** Everything below was read on disk from `backend/.venv/.../strands/_context_manager/` at **1.55.0**, our current pin — `context_manager.py`, `stash.py`, `strategies/offload/{base,drop,summarize,truncate}.py`, `methods/`, `modes/agentic/`, `retrieval_tool.py`. 1.56.0 did not *release* the stack; it finished it. The 1.56 additions are strategy presets + agent rewire (#4282) and **session-manager integration (#4254)** — the last being the one that matters, since our compaction lives inside `TurnBasedSessionManager`. **Evaluating this is not gated on the 1.56 bump**, which is separately blocked as a paired pin with `bedrock-agentcore>=1.23.1`. @@ -243,17 +276,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - ⏳ `strands/storage/s3_storage.py` vs. our own S3 backend in `tool_result_offload.py` — if upstream's is equivalent, that is a deletion. - ⏳ Diff `_context_manager/` on **every** Strands bump. It is marked *"Internal implementation. Public experimental API is re-exported from `strands.experimental.context_manager`"* — an experimental surface we would be building on, so churn is expected and is itself the signal. -### [2026-09-21] Build the quality-veto harness — scoped, and smaller than both specs assume -- **Source**: `docs/kaizen/scoping/2026-09-21-quality-veto-harness.md`, written 2026-09-21. It is **trigger 1** on both waivers recorded the same day, and the only path to an answer that does not wait on user volume (the other two triggers do). -- **The finding that changes it**: the specs and the spike describe *one* harness driving live multi-turn sessions through the deployed runtime, for both epics. That is right for offload and **substantially wrong for compaction** — `CompactionPolicy.resolve()` and `choose_checkpoint()` (`compaction_policy.py:77,199`) take a `CompactionConfig` **object** and a message list and read no environment at call time, so both arms can be built in-process and applied to the same authored transcript. The cut is deterministic; only the bounded summary (one Nova Micro call per arm) and the answers need a model. A 60-turn session does not have to be *run*, twice, k=3. -- **⚠️ Correction to the spike's "extend `experiment_agent_cache_arms.py`, don't rebuild it"**: half right, and the wrong half matters. `run_arm()` and `attach_cost_rows()` generalize directly. **The arm definition does not** — `ARMS` is a dict of `enabled_tools` lists, chosen because that script's docstring wanted arms with *no redeploy*, and `run_agent_headless` (`harness/runner.py:136`) exposes only the `InvocationRequest` surface (`model_id`, `rag_assistant_id`, `enabled_tools`, `agent_type`, `inference_params`). There is no compaction or document knob on it. Compaction config is `from_env()` on the runtime, so **two compaction arms on one runtime is not possible through the live path** — which is what makes the offline route above the answer rather than a shortcut. Offload B-vs-C *is* per-session selectable (`offload_enabled_for` buckets on `crc32(session_id) % 100`), so that arm needs no deploy either; only A-vs-B does. -- **The slice that needs no judge**: two of three families in *each* spec are programmatically generatable **and** scorable — constraint retention + reference lookup (spiral §4.3), lookup + citation (offload eval §2.1). Exact match against facts we planted. So slice 1 needs **no blinded judge, no scrubber, no AgentCore Evaluations, no LLM-judging spend** — and it is where the statistical power lives anyway, since ≥100 tasks is affordable only in the programmatic families (~40 paired binary tasks detects only a ~25pt regression). -- **Surface**: new — `backend/scripts/` + a fixtures dir; reuses `compaction_policy.py`, `compaction_summary.bound_summary`, and (slice 2) `experiment_agent_cache_arms.py`'s transport. -- **Effort × Impact**: M × H for slice 1 (was L × H as an undifferentiated epic, which is why it never started). Slices 2 and 3 are separately sized in the doc. -- **Subtracts**: no — addition. Justified because it is the **only** instrument that can falsify a cost change on quality, and `CLAUDE.md`'s tenet ("when cost and answer quality genuinely conflict, quality wins") is unexercisable without one. -- **Unlocks**: the counterweight the cost arc has never had — and with it, the ability to move `FLOOR_RATIO` / `CEILING_CAP_TOKENS` / the digest budget on evidence rather than taste, which both specs' tuning sections already assume exists. -- **Status**: open, scoped, not started. ⚠️ Three open questions in §6 gate slice 1 and are cheap to answer: does an authored transcript reproduce the cut the real path takes (`choose_checkpoint` rescales to `history_tokens`, so the transcript must supply a realistic measured size, not just realistic text); is `bound_summary` reachable offline; and does `model_relative_enabled=False` reproduce pre-#1125 behaviour *exactly* rather than approximately — if not, the control is a third policy and the comparison means something narrower than the spec claims. - ### [2026-09-21] The cheap quality veto does not exist — the query was already built, and the substrate is empty - **Source**: measured today against prod and dev. reviews/2026-09-18.md ▸ Proposal #3 recommended "**Ship the cheap version this week**… #1142/#1145 just shipped `F#` down-thumb rows, so a comparison across turn classes is available today with a **query, not a build**." Both halves of that sentence are wrong, in opposite directions. - **The query was already built.** `GET /admin/feedback/fleet?days=N` (`admin/feedback/fleet.py:197`) already emits `arms.turnClass`, bucketed by `costs/service.py:88` `turn_class()` with exactly the precedence the offload spec §6.1 asks for — **full > retrieved > digestOnly**, chosen so a call that pulled pages back does not fall into the digest arm and leave `retrieved` permanently empty. Nothing needed building. The review proposed writing a query that had shipped in the same epic it was reviewing. @@ -317,17 +339,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **✅ DIAGNOSED + GUARDED 2026-09-18 (`e196a00e`)** — not an off-by-one. `toolTokens` is a **residual** between two independently sourced counts, `full` (Strands' projection) minus `no_tools` (our own CountTokens call), so any disagreement about how a content block is counted lands wholly in it. The arithmetic is conclusive: 106,756 − 12,516 = **94,240**, against a document measured at ~94,485 — the entire document attributed to tools, because Bedrock counts a PDF page as an image *and* a text layer and the two sources did not agree. The counting disagreement is Bedrock/Strands behaviour we do not control, so the split is simply **not computed while inline document or image bytes are in context** and is taken on a later clean turn; absent reads "not tracked", which a wrong number does not. A digest is text, so attachment sessions still get the field from turn 2. **Residual question for the prod readout:** whether `full` and `no_tools` disagree on *images* too — if `prefixTokens` goes missing on image-only sessions for more than the attach turn, that is the tell. - **`compaction_policy._block_tokens` has the same PDF blind spot** #1147 fixed elsewhere, and was deliberately left alone: it rescales to measured history and drives cut thresholds, so changing it moves when compaction fires. Its own decision. -### [2026-09-18] Reverse the Astra decision — register at the full 1M window and make the price tier its own catalog field -- **Source**: research/2026-09-18.md ▸ Idea #1 — https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html. ⚠️ **Directly supersedes the open `[2026-09-11] Add GPT-6 Astra at the 272K tier` entry**, whose blocking question this answers in the *opposite* direction: the model card's own headings ("short context (272K input tokens or fewer)" / "long context (more than 272K input tokens)") show the tier is selected by **actual input tokens per request**, not by the declared window. Merge the two at review. -- **Surface**: frontend (`admin/manage-models/models/curated-models.ts`, model picker) + backend (Mantle Responses path, `admin/services/model_access.py`) -- **Effort × Impact**: L × H -- **Subtracts**: yes — a queued recommendation (`maxInputTokens: 272000`) that would have shipped a capability regression; the `long = short × 2` assumption (output is **1.5×**: $55.00 → $82.50, not $110.00); and the ambiguity about which API surface reaches Astra -- **Unlocks**: - - A genuine **1M-token frontier tier** — the first catalog model where a whole corpus fits in one turn — with no artificial cap - - A **tier-aware catalog**: the admin page can show the boundary, cost projections can respect it, and Astra's 10× output-token quota burn can be accounted for against the known campus TPM ceiling. The 272K cap survives as an optional *policy* lever -- **Notes**: ⚠️ Take the **Mantle Responses** path, not Converse — the card lists prompt caching only under `bedrock-mantle`/Responses, so `bedrock-runtime` Converse would re-pay a 30k–150k prefix at $11/MTok every turn. ⚠️ Astra is **not** a third `cacheRead × 0.1` counterexample (it is exactly 0.100× in both tiers); Fable 5.1 and Grok 4.6 remain the only two. ⚠️ The ref repo's `rejectsTemperature: true` is **still not on the model card** — verify, don't copy. ⚠️ Astra's cache-write SKU is a **30-minute** TTL — do not fold into `cache_ttl_seconds_for()`'s 5m/1h assumptions. -- **Status**: open - ### [2026-09-18] Sweep the cacheable prefix for derived and relative values — and close the last daily boundary - **Source**: research/2026-09-18.md ▸ Idea #3 — Claude Code 2.1.275 (*"a restored memory file's age note changing between requests after a compaction or resume, which caused prompt cache misses"*), 2.1.269, 2.1.273 — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md — plus our own `9f246cb7` - **Surface**: backend — `agents/main_agent/utils/` (`get_current_date_pacific()` + the `SystemPromptBuilder` tail), `session/turn_based_session_manager.py` (truncation anchor), `session/document_rehydration.py`, `apis/shared/sessions/messages.py`, and the resume path in `inference_api/chat/routes.py` @@ -367,6 +378,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - ⚠️ **The intuition is backwards.** "Our tool prefix is huge, so a 1h TTL must pay" is wrong — a bigger static prefix scales *both* sides of the trade, making the arm more negative. Even a hypothetical 1h TTL on the **whole** prefix (not what is built) saved only $0.68 of $20.47 (3%). - **Decide it for free, on data we already hold**: count `cacheStatus` starting with `miss` and `cacheGapSeconds < 3600` over rows with `cacheWriteInputTokens > 0`, per session, over a month. If that share is not north of 65% for a meaningful cohort, close this item and keep the flag off — no dev week, no flag flip. Folded into the fleet sweep item above. - Keep the code and the flag: it is a correct implementation of a lever that pays under a usage pattern we do not currently have. +- **⚠️ Status 2026-10-02 — the DECLINE above was REVERSED on 2026-09-27 by a fleet pass that was never recorded here.** The 65% rule used the wrong denominator (all cache-writing calls). The 1h premium is paid only when the static prefix itself is re-written (`calculator.py:102` bills `min(cw, S − cr)`), so the test is avoided-TTL-misses over *remaining prefix writes* against 0.65. Prod fleet, 34,891 `C#` rows, 2026-09-06 → 27, load tests excluded: **0.78**, net **about +$22 over 21 days (~2% of spend)**, 476 of 1,350 sessions positive; `7f5f207f` flips to +$2.23. TTL-expired gaps: 32% at 5–10 min, 31% at 10–30 min, so most 5m misses fall inside 1h. Caveat: only 18% of rows carried `prefixTokens`; the rest were imputed at the 15.8k median. **Recommendation now: decline on SIZE (~$30/month), never on sign — or flag it on in prod for one measured week.** The thresholds spec §6 PR-5 still carries the old number (reviews/2026-10-02.md ▸ Retirement Candidates). ### [2026-09-16] The 28% "neither" cohort — content-class ordering inside the cut is the next compaction lever, if the metrics say the floor is still missed - **Source**: 2026-09-15 prod cost audit (content-free): of 95 September sessions over 100k, **52 (55%) had a ≥4k-token tool result in the last 3 turns** (→ #1131 intake offload), **25 (26%) an attachment there** (→ document offload, its own item below), **27 (28%) neither** — long sessions (a3b884b1 46 turns, 4cb93e6e 45 turns) whose bulk is old history plus a 23–40k summary. @@ -429,6 +441,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Subtracts**: yes — a monkeypatch on a Strands internal, taken *because* the SDK offered no hook, on a class Strands is **actively changing**: this week alone an httpx2 adapter (#4183), an `mcp` floor relaxed to `<2.2` (#4151), SEP-2663 tasks (#4125), and changed `load_servers` defaults (#4177). - **Unlocks**: an App-frame header on a standards-blessed field rather than a private symbol, decoupled from the much larger `server/discover` migration (still gated on two unanswered questions: does AgentCore Gateway speak it, does Strands expose it). - **Status**: open — `ext-apps` v2.0.0's `schema.json` now documents **`io.modelcontextprotocol/serverInfo` in result `_meta`**, which is exactly what we intercept `initialize` to get. ⚠️ **Verify the servers we actually call emit it before deleting anything** — presence in `schema.json` is not evidence any of our Gateway targets or Lambda FastMCP servers populate it; keep the patch as a fallback until they do. ⚠️ **Same release changed host-facing error codes**: a handler-thrown `-32002` now reaches the View as `-32602`, invalid params on `ui/*` move `-32603` → `-32602`, and the `MCP error N:` message prefix is **gone** — grep the app-tool-error chain shipped in #1009/#1013 for code or message-text matching, since so much of it was built on message text. ✅ The MCP Apps **wire protocol is unchanged** in v2.0.0 (bidirectional interop test against 1.7.5), so nothing else in our host path needs touching. ⚠️ Related and new: `mcp` resolves at **1.28.1** in `uv.lock` while latest is **2.2.0** and Strands' floor now permits `<2.2` — the next refreshed resolve could cross a major under this patch. +- **Status 2026-10-02**: (b)/(c) **re-gated on the mcp-2.x migration instead of a date** (reviews/2026-09-25.md ▸ Carried Over). The `_meta serverInfo` path only matters once we speak 2026-07-28. Long run, MCP SEP-3371 (consistent SDK extension points, updated 2026-10-02) is the path that retires the substitution outright. Re-surface when `mcp<2` lifts. ### [2026-09-08] AgentCore Runtime workspaces — give the agent a filesystem; start by mounting the one we already have — ⛔ **S3 BRIDGE BLOCKED** (build attempted 2026-09-11, stopped before any code) - **Source**: conversation with Phil, 2026-09-08 — **not** from a research scan. Verified against the pinned `botocore` 1.43.68 service model, the AgentCore devguide, and https://aws.amazon.com/blogs/machine-learning/its-safe-to-close-your-laptop-now-hosting-coding-agents-on-amazon-bedrock-agentcore/. @@ -661,22 +674,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: closes the known "approval hook can't see through the tool-fold" hole (pairs with the Strands hook-ordering bump); "data sources used" provenance on `tool_result` cards (we already carry `serverName`/`icon` on `ui_resource`) — a top NN/g trust driver - **Status**: open — auto-resume once all approvals in a turn resolve (the multi-tool piece worth stealing from the AI SDK). -### [2026-07-03] Evaluate gateway-level Guardrails (AgentCore Policy) vs. in-agent #480 -- **Source**: research/2026-07-03.md ▸ Top 5 #5 — https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-guardrails.html; relates to queued issue #480. -- **Surface**: infrastructure + backend (PlatformStack Gateway construct + AgentCore Policy; `apis/shared` tool routing) -- **Effort × Impact**: M × M -- **Subtracts**: potential — one gateway-level policy vs. per-tool control complexity -- **Unlocks**: model-independent FERPA/injection enforcement across all Gateway MCP targets (the agent can't reason around it) -- **Status**: open — **fold into the #480 decision** rather than run a separate track: assess whether one gateway policy is preferable to or complements the in-agent `guardrail_id` approach. - -### [2026-06-19] Wire configurable Bedrock Guardrails (issue #480) -- **Source**: research/2026-06-19.md ▸ Top 5 #1 — internal issue #480 (June 15) + AWS Summit NYC Guardrails cluster (`InvokeGuardrailChecks` API + AgentCore policy Guardrails GA, June 16). Strands `BedrockModel` already supports `guardrail_id`/`version`/`stream_processing_mode`/`trace`. -- **Surface**: backend (`inference_api` `BedrockModel` construction) + infrastructure (optional `CDK_GUARDRAIL_ID` / `CDK_GUARDRAIL_VERSION` env vars threaded to inference-api runtime env) -- **Effort × Impact**: L-M × H -- **Subtracts**: addition only — config wiring of a capability Strands already exposes; zero-cost when unset; mirrors `CDK_ARTIFACTS_ENABLED`/`CDK_MCP_SANDBOX_ENABLED` optional-feature pattern -- **Unlocks**: deployers attach content-safety filtering + staff-alerting monitoring to all model invocations without modifying inference-api source (FERPA duty-of-care for higher-ed: proactive self-harm/crisis-language monitoring Claude's reactive layer doesn't surface) -- **Status**: open — strongest fit (filed issue + library-native path). **Decide in-agent vs. gateway-level in one pass** — the [2026-07-03] "gateway-level Guardrails (AgentCore Policy)" item folds into this #480 decision (one gateway policy blankets every MCP target, model-independent). Verify guardrail *resource* region availability + SSE streaming-mode compatibility. Reviewed reviews/2026-07-03.md ▸ Proposal #4. - ### [2026-06-19] Ship the interactive context-breakdown badge (Cursor + LibreChat convergence) - **Source**: research/2026-06-19.md ▸ Top 5 #5 — LibreChat v0.8.7-rc1 real-time context gauge + Cursor Context Usage Report (2026-06-05) + internal PR #433. **Reinforces** the [2026-06-05] "make the context-breakdown badge interactive" item with a second independent product datapoint. - **Surface**: frontend (context-breakdown badge component in `frontend/ai.client/src/app/session/`) @@ -685,13 +682,6 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Unlocks**: user-facing context-cost transparency + an actionable "what's eating context / how to trim it" follow-up - **Status**: open — presentation-layer only (no backend change). Consolidated the superseded [2026-06-05] Cursor-only entry into this item. Lower priority than the [2026-07-03] reliability/model cluster. Reviewed reviews/2026-07-03.md ▸ below-cap (defer 1 week). -### [2026-06-05] Bump `docling` past the 2.81.0 content-sniffing defect → close #405 (`.txt` uploads fail) -- **Source**: research/2026-06-05.md ▸ Top 5 #4 — docling 2.97.0 (June 3) + internal issue #405 -- **Surface**: backend (document-ingestion docling dep pin) -- **Effort × Impact**: L × M -- **Subtracts**: yes — library-native bump closes an open user-facing bug; no custom workaround needed -- **Status**: open — **#405 still open ~5 weeks; `requirements.lock` still pins `docling==2.81.0` (latest 2.109.0).** Cleanest subtraction; bump off 2.81.x, verify `.txt` upload, close #405. Reviewed reviews/2026-07-03.md ▸ Proposal #6. - ### [2026-06-05] De-risk #419 (admin-managed Gateway target registration) against the new AWS auth-code-flow + BYO-secrets references - **Source**: research/2026-06-05.md ▸ Top 5 #5 — AWS "secure OAuth auth-code flow with Gateway + MCP clients" + AgentCore Identity BYO Secrets Manager (both June 1) + internal issue #419 - **Surface**: infrastructure (Gateway target CRUD / `gateway_target_*`) + backend (`apis/shared/oauth/agentcore_identity.py` OAuth provider wiring + token-vault customParameters) + frontend (admin registration UI) @@ -752,6 +742,51 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Resolved +### [2026-09-18] Reverse the Astra decision — register at the full 1M window and make the price tier its own catalog field → RESOLVED — **DECLINED as scoped** +- **Decision**: declined and logged (`decisions.md` [2026-10-03]). +- **Reasoning**: `maxInputTokens: 272_000` is load-bearing pricing while `CuratedModel` holds one rate per bucket; lifting it opens AWS's second price card (2× input, 1.5× output) and under-charges every long turn. Re-open only alongside a per-tier rate field. +- **Reviewed in**: reviews/2026-09-25.md ▸ Retirement Candidates; reviews/2026-10-02.md ▸ Carried Over + +### [2026-07-03] Evaluate gateway-level Guardrails (AgentCore Policy) vs. in-agent #480 → RESOLVED — **DECLINED as a kaizen item** +- **Decision**: declined and logged (`decisions.md` [2026-10-03]), together with the [2026-06-19] entry it folded into. Issue #480 stays open as product backlog. +- **Reasoning**: in-agent vs. gateway-level is a product decision on #480, not a weekly improvement; eight carries without a surface. +- **Reviewed in**: reviews/2026-09-25.md and reviews/2026-10-02.md ▸ Carried Over + +### [2026-06-19] Wire configurable Bedrock Guardrails (issue #480) → RESOLVED — **DECLINED as a kaizen item** +- **Decision**: declined and logged (`decisions.md` [2026-10-03]). Issue #480 stays open as product backlog. +- **Reasoning**: eight carries without a surface anyone picked up. Guardrails returns as product work on #480, not as a queue entry. +- **Reviewed in**: reviews/2026-09-25.md and reviews/2026-10-02.md ▸ Carried Over + +### [2026-06-05] Bump `docling` past the 2.81.0 content-sniffing defect → close #405 (`.txt` uploads fail) → RESOLVED — **DECLINED as a kaizen item** +- **Decision**: declined and logged (`decisions.md` [2026-10-03]). Issue #405 stays open as product backlog. +- **Reasoning**: the queue is not where a backlog bug gets picked up; eight carries proved it. The bump itself is still welcome as a PR-sized slice (`requirements.lock` still pins `docling==2.81.0`), and the decision says so. +- **Reviewed in**: reviews/2026-09-25.md and reviews/2026-10-02.md ▸ Carried Over + +### [2026-09-25] Treat any non-`end_turn` stop reason as a failed side-channel call → RESOLVED — **SHIPPED** +- **Decision**: Ship (reviews/2026-09-25.md ▸ #2). +- **Reasoning**: #1368 (`7f1e8fd4`, 2026-09-27) accepts only `end_turn` in `compaction_summary.py` and `tool_summaries/summarizer.py`; any other stop reason returns `None` and the existing fallbacks run. +- **Reviewed in**: reviews/2026-10-02.md (scorecard). + +### [2026-09-25] Guard Stop against a turn that already finished → RESOLVED — **SHIPPED** +- **Decision**: Ship (reviews/2026-09-25.md ▸ #3). +- **Reasoning**: #1366 (`41dd4471`) gates `cancelChatRequest` on the parser's completion state and deletes the caller-less `isStreamCompleteFor()`; #1370 (`5956eb20`) applies the same rule to the page-hide path. Both 2026-09-27. +- **Reviewed in**: reviews/2026-10-02.md (scorecard). + +### [2026-09-25] Strands 1.57 + `bedrock-agentcore` 1.23.1 + boto — one paired bump → RESOLVED — **SHIPPED** +- **Decision**: Ship (reviews/2026-09-25.md ▸ #4). +- **Reasoning**: #1367 merged 2026-10-02 at strands **1.57.2** / bedrock-agentcore **1.24.0** / boto **1.43.103** (lock `mcp` 1.30.0, still `<2` by declared constraint). The two guard rails are tests in `tests/supply_chain/test_strands_agentcore_pairing.py`. Voice was ported to the graduated `strands.bidi` API behind an unchanged WebSocket contract and off `BidiAgent(session_manager=)`. Dev validation of a text turn and a voice turn is pending (reviews/2026-10-02.md ▸ #7). +- **Reviewed in**: reviews/2026-10-02.md (scorecard). + +### [2026-10-02] Land #1367 now at strands 1.57.1 / agentcore 1.24.0; take strands 1.57.2 as its own PR → RESOLVED — **SUPERSEDED** +- **Decision**: Superseded by events the same morning: #1367 merged at 1.57.2 with the voice change already in it. +- **Reasoning**: research/2026-10-02.md read #1367 as `CONFLICTING` at 1.57.1; it was rebased and merged at 09:39 MT. Of this entry's PR B, the `strands.bidi` import and the `session_manager` removal are done; the `usage_normalization.py` cache-write recovery deletion carries as reviews/2026-10-02.md ▸ Proposal #7. +- **Reviewed in**: reviews/2026-10-02.md. + +### [2026-09-21] Build the quality-veto harness — scoped, and smaller than both specs assume → RESOLVED — **SHIPPED (slice 1) and run** +- **Decision**: Build slice 1 (reviews/2026-09-18.md ▸ #3 deferred the harness to 2026-10-02; it shipped before the date). +- **Reasoning**: `56f9bc0b` (offline harness, slice 1), `291a55c8` (the veto ran: the cut passes, the compression fails), `3fdc05ae` (summarizer screening), `d28f90fc` (extract-then-compress clears the veto). The compaction waiver's trigger 1 fired and the stack passed; extract-then-compress then shipped default-on (#1355). Slices 2 and 3 (the live-path offload veto, the judged families) stay described in `scoping/2026-09-21-quality-veto-harness.md` and the document-offload waiver stands. +- **Reviewed in**: reviews/2026-10-02.md (Carried Over). + ### [2026-09-18] Merge #1147 — two fixes and the offload epic's measurement plan → RESOLVED — **SHIPPED** - **Decision**: Ship (reviews/2026-09-18.md ▸ #1). - **Reasoning**: #1147 merged 2026-09-18 with the PDF-aware `documentTokens` fix and `DOCUMENT_READ_ENABLED` coupling; the ReDoS bound (`71b0fdfb`) rode it and #1162 refined the budget. Its three `[2026-09-18]` measurement entries stay open as the authoritative follow-ups. diff --git a/docs/kaizen/reviews/2026-10-02.md b/docs/kaizen/reviews/2026-10-02.md new file mode 100644 index 000000000..b450cf2df --- /dev/null +++ b/docs/kaizen/reviews/2026-10-02.md @@ -0,0 +1,271 @@ +# Kaizen Review — Friday, October 2, 2026 + +> Prepared 10:15am MT. Review window: September 25 – October 2 (7 days). +> Source: research/2026-10-02.md (merged to `develop` as #1401 at 08:54 MT, so it was read from disk) + review-queue.md (76 open items before this PR's queue movement, 71 after) + the scorecard for reviews/2026-09-25.md, checked against git and disk. +> Open PR branches were read for `docs/kaizen/` changes. #1405 (smoke harness) and #1406 (parallel MCP pre-flight) touch none; #1405 carries `docs/testing/smoke-regression.md` and a `turn-path-ttft.md` edit, both cited below. +> ⚠️ The research doc went stale between its 08:33 commit and this review: its Top 5 #3 recommended landing #1367 at strands 1.57.1 and taking 1.57.2 separately. #1367 merged at **09:39 MT at 1.57.2**, with the voice change the research warned about already in it. The scorecard reflects the merged state, not the research text. + +## Scorecard — reviews/2026-09-25.md, checked against git and disk + +| # | Proposal | Outcome | Evidence | +|---|---|---|---| +| 1 | A/B the V2 Runtime in dev | ⏳ **Planned, not started** (2nd week as #1) | Spec merged 09-28 (#1376). `grep -i platformVersion scripts/build/deploy-runtime-image-if-changed.sh infrastructure/lib/constructs/inference-api/` finds nothing; `aws-cdk-lib` still 2.265.0 | +| 2 | Non-`end_turn` = failed side-channel call | ✅ **Shipped** | #1368 (`7f1e8fd4`), 09-27 | +| 3 | Guard Stop against a finished turn | ✅ **Shipped** | #1366 (`41dd4471`) + #1370 page-hide (`5956eb20`), 09-27. `isStreamCompleteFor()` was deleted rather than reused, per the commit message | +| 4 | Paired strands / agentcore / boto bump | ✅ **Shipped this morning** | #1367 merged 2026-10-02 09:39 MT at **1.57.2 / 1.24.0 / 1.43.103**. Pairing and `mcp<2` are now tests (`tests/supply_chain/test_strands_agentcore_pairing.py`). Voice moved to `strands.bidi` and off `BidiAgent(session_manager=)` (`61f12527`). **Dev validation of a text turn and a voice turn is still pending**, per the PR's own section | +| 5 | Fleet session-anatomy sweep, `missCause` derived offline | ⚠️ **Ran in part, off-repo, unrecorded** | No script in `backend/scripts/`; `grep -rn missCause` is empty (5th review). But a session note dated 2026-09-27 records a prod pass over 34,891 `C#` rows that answered sweep question 2 and **reversed the PR-5 decline** (see Retirement Candidates). Nothing in `docs/` carries it | +| 6 | Curate Opus 5.5 (a); Luna A/B (b) | ✅ (a) **Shipped**; (b) deferred to 10-23 | #1358 (`f3c90799`) 09-27; prod rows added 09-26; redirect verified on one retired-model agent 09-27 | +| 7 | MCP Apps header `toolName` precedence | ❌ **Not started** | `grep annotations apis/shared/mcp_apps/*.py` finds only `from __future__` | +| 8 | Pacific-midnight prefix rewrite | ⏸ Deferred on #5, which did not run | `system_prompt_builder.py:212-213` unchanged | +| 9 | Session-scoped MCP discovery + sidebar search | ⏸ Deferred to 10-09; **new context on both halves** | #1406 (open today) attacks the same pre-flight cost from a different angle (21.5 s serial on a KB agent); issue #1380 (09-28) is a user request for conversation search | + +**4 of the 6 Ship recommendations converted, 1 was planned, 1 ran off the record.** The four carried decline/accept recommendations (docling/#480, the RBAC audit, `bedrock-agentcore` #629, and the Astra reversal) were **not logged**: `decisions.md` has no entry after 2026-09-21. That is the second review running. + +## Week in Review + +A 100-PR week (121 commits, zero reverts, zero `develop`-push deploy failures, two releases) spent almost entirely on **the turn path to first token**. The map shipped (#1388), then six measured cuts landed behind it: the shared agent-build session (869 → 373 ms, #1395), the first-token clock (#1397/#1398), the overlapped long-term-memory prefetch (#1400), and the per-turn history count taken off the critical path (388 → 0 ms, first token 1560 → 1092 ms on dev, #1403). #1406 is the next one, and #1405 gives the path a regression pass. The bump that drifted for four weeks landed this morning, with the pairing turned into tests. + +The ecosystem handed over two price tags that both argue for measuring first. **AgentCore Runtime V2 bills memory at 1.79× V1**, so last week's #1 is no longer "flip it and see": it wins only if billed GB-hours fall below ~56% of V1's, which is plausible on the idle tail but is a measurement. And **Opus 5.5's always-on thinking may not tolerate the history rewrites our compaction and offloader do**: Anthropic's cookbook now says rewriting earlier history on preserved-thinking models can get the next request rejected. Opus 5.5 is in prod. Nobody has run a compaction turn on it. + +Internally the signal is **recording, not building**. The nightly has been red two nights because an ephemeral stack is stuck in `DELETE_FAILED`, and a helper that fixes exactly that (`scripts/common/recover-stack.sh`) exists but is not called by the nightly. Six ops entries blocked on "after the release" have been unblocked since 09-26 and none has its counts written down. The fleet cost pass ran and reversed a queued decision, and the reversal lives only in a session note. + +## Friction — the week's signal + +### Repeated patterns (≥2 occurrences) + +- **Finished work and finished measurements that never reach the repo** (4 occurrences this week). + - The Sonnet 5.5 curation (`d708b358`, 09-29) sits on an unpushed worktree branch with no PR. Verified today: `git merge-base --is-ancestor` says it is not on `develop`. + - #1367 was written, tested and mutation-checked, then sat five days with no review until `develop` moved under it. It merged this morning only after a rebase. + - The fleet `C#` pass of 2026-09-27 (34,891 rows) exists only as a session note. The queue still says PR-5 is "recommend DECLINE" on the number that pass overturned. + - The 1.25.0 post-deploy ops (orphan-row cleanup, byte-counter repair, the log-retention sweep's first prod tick) ran, per session notes, and the six queue entries that asked for their counts are untouched. + - *Hypothesis*: work that starts from a task chip or worktree ends in that session with no hand-off step that ends in a PR or a queue edit. The feedback memory already names the first half ("orphaned worktree diff"); the measurement half is new. + - *Candidate fix*: Proposal #6 (record and close the ops entries) and the Take. The durable fix is procedural: a measurement that changes a queued recommendation is not done until the queue entry says so. + +- **The nightly's ephemeral stack cannot tear down an AgentCore Runtime, and the pipeline has no recovery step** (2 failed nights, 10-01 and 10-02). + - 10-01 teardown: `InferenceApiAgentCoreRuntime… Request timed out while deleting AWS::BedrockAgentCore::Runtime (NotStabilized)` → `DELETE_FAILED`. + - 10-02 deploy: `nightly-develop-PlatformStack … is in DELETE_FAILED state and can not be updated`. + - `scripts/common/recover-stack.sh` already implements "find the `DELETE_FAILED` resources, retry with `--retain-resources`, wait" and the [2026-07-24] queue entry resolved the same class with it. `nightly-deploy-pipeline.yml` does not call it (verified: no `recover-stack` reference in either nightly workflow). The July fix was a one-off, not a step. + - *Candidate fix*: Proposal #3. + +- **"Finished" defined loosely at the edge of a model call** (carried from last week; both sites fixed, one new sibling). + - #1368 and #1366 closed the two verified sites. + - `7a8028a2` (today) skips the long-term-memory prefetch for the voice `BidiAgent`, hours after #1400 added it: a hook written for `Agent` ran on an agent type it had never seen. Same family as "one session served by more than one agent": per-agent-type assumptions in shared hooks. + +### One-offs worth watching + +- **#1372** (09-27): the KB reconciler over-counts a synced change the byte cap refused. Issue filed, no PR. It is the fourth byte-cap accounting defect in two weeks (#1347, #1361, #1365, #1373 are the other side of the same ledger). +- **#1367's reviewer flag**: a managed OpenAI-family row with `maxInputTokens` absent now resolves to 1.05M instead of `None`, so compaction would cut past Astra's 272K price tier. No guard was added because it is a policy call. Nobody has checked whether any dev or prod row lacks the field. +- **`CLAUDE.MD` is the tracked filename** (uppercase `.MD`), and #1405 edits it under that name. It works on a case-insensitive filesystem and on Linux CI because the name is consistent, but any tool that looks for `CLAUDE.md` by exact name misses it. Not a bug today; worth knowing before someone "fixes" the case. +- **9 enhancement issues from a product stakeholder in 7 days** (#1305–#1308, #1379–#1389), 0 issues closed, 41 open repo-wide. Issues are not the work-tracking surface here (the queue and PRs are), so these have no triage path. #1380 (search) overlaps a queued item; the rest do not. + +### Silence that matters + +- **`missCause`, 5th review.** Last week folded it into the sweep script so the cycle could end. The sweep ran in part, without the labelling, and without the script being committed. Proposal #5 ends the cycle one way or the other. +- **Four recommended decisions, zero logged, two reviews running.** `decisions.md` last changed 2026-09-21. Each carry costs a paragraph every week. The four entries are drafted verbatim under Carried Over; they need one commit. +- **Luna as a side-channel model** and issue **#1340** (admin-selectable summarizer model) are the same question from two directions. Neither has moved. Not due until 10-23. +- **V2 Gate 2 (B2, import-time state cloned by snapshot restore)** has had no probe. It is cheap and it precedes the A/B. + +## Proposals — ranked + +### 1. Re-base the V2 Runtime plan on the published price, land B1, and use CDK's typed `platformVersion` +- **Source**: research/2026-10-02.md ▸ Top 5 #1; review-queue.md ▸ [2026-10-02] + [2026-09-25] V2 A/B + [2026-09-04] W5 part (2). Last week's #1. +- **Surface area**: `docs/specs/agentcore-runtime-v2.md` §1/§4 (still say "No V2 price" at line 38 and "Unknown rate" at line 117, verified); `scripts/build/deploy-runtime-image-if-changed.sh` (`ALLOWED` list, no `platformVersion`); `infrastructure/package.json` (`aws-cdk-lib` 2.265.0 → 2.272.0, `constructs` alongside); `inference-agentcore-construct.ts:297`. +- **Change**: + - (a) Write the rates into §4 with the break-even stated **before** the A/B: memory ≤ 56% of V1 GB-hours, CPU ≤ 70% of V1 vCPU-hours; idle memory reclaimed after 120 s against our 900 s idle timeout. + - (b) Land plan step 1 (B1): `platformVersion` in `ALLOWED`, assert it after the update, fail fast if the runner's CLI lacks the field. Harmless on V1. + - (c) Bump CDK and set the typed property in place of the planned `addPropertyOverride`, keeping the spec's "always set it explicitly, V1 or V2" rule. +- **Subtracts**: yes. The untyped override step in the plan, and the [2026-09-04] W5 instance-SKU arithmetic entry (the committed baseline is now a published rate, $0.0132/GB-hour, and is the V2-native replacement for that arithmetic). +- **Unlocks**: a go/no-go threshold written down before the data arrives, which makes the dev A/B decisive instead of interpretive. +- **Effort**: Low +- **Impact**: High +- **Evidence**: V2 rates verified against the pricing page source (research, 2026-10-02). B1's code path verified on disk; its outcome (a `backend.yml` deploy reverting V2 to V1) is not yet reproduced. +- **Ship means**: one spec edit plus one deploy-script PR this week (a + b). The CDK bump (c) rides whichever PR first touches `infrastructure/package.json`. The A/B itself waits on B1 and on a B2 probe. +- **Decline means**: the W5 lane stays empty for a sixth month, and the one lever on 73% of the AICC bill stays untested. +- **Recommendation**: **Ship.** Second week as #1, and the price makes it smaller, not bigger: B1 is a script edit and the spec edit is arithmetic. ⚠️ The Cost Explorer sync (#1235) must show whether V2 bills under a distinct usage type, or the dev week can only be compared before/after. + +### 2. Verify compaction and offload on a preserved-thinking model (Opus 5.5) before more of them ship; push the stranded Sonnet 5.5 curation +- **Source**: research/2026-10-02.md ▸ Top 5 #2; review-queue.md ▸ [2026-10-02]; cookbook `bf24d45`; the Opus 5.5 prompting guide; local branch `claude/sonnet-5-5-bedrock-dev-e9a13e`. +- **Surface area**: `session/turn_based_session_manager.py` (checkpoint application, truncation anchor); `core/tool_result_offload.py`; the curated Opus 5.5 row; the unpushed Sonnet 5.5 row (`curated-models.ts`, +52 lines, with a spec change). +- **Change**: + - (a) In dev, on `us.anthropic.claude-opus-5-5` (the dev SCP allows `us.*`), drive one conversation past the compaction threshold with at least one offloaded tool result. Record whether the next call succeeds and its `cacheStatus` on the `C#` rows. If Bedrock rejects it, compaction on preserved-thinking models must keep earlier thinking blocks byte-identical or drop whole assistant turns, never rewrite inside one. + - (b) Grep for forced `tool_choice` / `structured_output` on the user-selected model. Opus 5.5 returns 400 on `any`/`tool`. + - (c) Check the Sonnet 5.5 branch against the 5.5 breaking changes (`between_tools` replaces `disabled`, recalibrated effort, account-bound thinking), then push and open the PR. + - (d) While in the catalog: check that no dev or prod OpenAI-family row lacks `maxInputTokens` (the #1367 reviewer flag). +- **Subtracts**: no. Addition, justified because this is a correctness check on a model already curated in prod, and the failure mode, a rejected request deep in a long conversation, is the costliest place to find it. +- **Effort**: Low +- **Impact**: High +- **Evidence**: the cookbook caveat is vendor-asserted for the Claude API; Bedrock behaviour is **unverified**, and (a) is the cheapest way to find out. The Sonnet 5.5 branch is verified on disk (one commit, not on `develop`). +- **Ship means**: one dev session for (a)+(b)+(d), and one PR for (c). +- **Decline means**: the first long Opus 5.5 conversation in prod finds out for us. +- **Recommendation**: **Ship.** ⚠️ Sonnet 5.5 is `global.*`-only on Bedrock and the dev SCP denies `global.*`, so the curated row cannot be exercised in dev at all; it goes to prod on the model card and the Price List API ($2.00/$10.00, cache read $0.20, verified 09-29), and the PR must say so. + +### 3. Make the nightly survive a Runtime that won't delete, with the helper that already exists +- **Source**: research/2026-10-02.md ▸ Top 5 #4; nightly runs 10-01 and 10-02 (verified from the failed-step logs today); `scripts/common/recover-stack.sh`; the resolved [2026-07-24] nightly `DELETE_FAILED` entry. +- **Surface area**: `.github/workflows/nightly-deploy-pipeline.yml` (the deploy step and the `teardown` job), `scripts/nightly/teardown.sh` (`:203-246`, which already surfaces the `DELETE_FAILED` reason and treats vector buckets as best-effort). +- **Change**: + - (a) **One-off, today, no review slot**: unstick `nightly-develop-PlatformStack` — delete the Runtime out of band, then `delete-stack`, or source `recover-stack.sh` and call `recover_delete_failed_stack` by hand. + - (b) Durable: before `delete-stack`, delete the Runtime via `bedrock-agentcore-control delete-agent-runtime` and poll until it is gone. On a `DELETE_FAILED` naming only the Runtime, call `recover_delete_failed_stack` (retry, then retain). Have the deploy step call the same function first, so one bad teardown costs one night, not every night after. +- **Subtracts**: no new code of consequence. It wires an existing helper into the one pipeline that needed it, and it removes the "fix by hand every time" loop the July entry left behind. +- **Effort**: Low +- **Impact**: Med. The nightly is the only end-to-end signal, and every review leans on its green streak. +- **Evidence**: verified: both failure logs, the helper's contents, and the absence of any `recover-stack` call in either nightly workflow. +- **Ship means**: (a) now; (b) as one workflow + script PR. +- **Decline means**: the nightly stays red until someone remembers the command, each time. +- **Recommendation**: **Ship.** ⚠️ Root cause of the Runtime delete timeout is unknown and nothing matches upstream. If it recurs after (b), file it with AWS using the request token in the 10-01 log. + +### 4. Headless runs never pause for a human: withhold the interactive tools, and classify the pause +- **Source**: research/2026-10-02.md ▸ Top 5 #5 (opencode #52740, pydantic-ai #9509); review-queue.md ▸ [2026-10-02]; relates to [2026-07-17] "Adopt Strands `Limits` on the headless lane" and LibreChat v0.8.8's bound scheduled-MCP consent. +- **Surface area**: `apis/shared/harness/models.py:20` (`RunStatus = completed | error | timeout | oauth_required`, verified), `harness/runner.py:266-272`, `harness/sse.py`; `apis/app_api/schedules/routes.py:74` (`_resolve_enabled_tools_snapshot`, verified: it snapshots the user's enabled tools at creation, so `ask_user_question` rides along whenever the flag is on); `tool_registry.py:117`. +- **Change**: preferred, strip `ask_user_question` and `request_user_login` from `enabled_tools` whenever the harness builds a run, at the snapshot and again at run time. Belt and braces, teach the accumulator `user_question_required`, `browser_login_required` and `tool_approval_required`, and classify a `done` that carries one as `needs_input`, never `completed`. +- **Subtracts**: yes. Two tool specs leave every scheduled run's `toolConfig`, so each run's cacheable prefix is smaller, and a run that paused stops being reported as finished. +- **Effort**: Low +- **Impact**: Med +- **Evidence**: the classification gap is verified on disk. Whether a scheduled run has actually paused in prod is unmeasured; the SSE channel would show a `user_question_required` after `message_stop` on a `runs` row. +- **Ship means**: one backend PR, two tests (a run whose model asks a question records `needs_input`; the snapshot never includes the two tools). +- **Decline means**: a scheduled run that asks a question records `completed` with a question nobody will answer. +- **Recommendation**: **Ship.** It is the cheapest item on the board with a subtraction in it. + +### 5. Commit the fleet `C#` sweep as a script, run the four questions it hasn't answered, and derive `missCause` inside it — or decline `missCause` for good +- **Source**: review-queue.md ▸ [2026-09-21] fleet sweep + [2026-09-04] `missCause` (5th review); reviews/2026-09-25.md ▸ #5 (Ship, not started); a session note dated 2026-09-27 (the partial run). +- **Surface area**: a read-only script under `backend/scripts/` over prod `C#` rows (content-free projection, date-bounded `GSI1SK BETWEEN`). No service code. +- **Change**: the 09-27 pass answered sweep question 2 (the PR-5 share) and nothing else, and its script was never committed. Commit it, extend it to questions 1, 3, 4 and 5 (write:read by release; the `miss_ttl_expired` gap distribution; `cacheStatus` mix by model with `maxInputTokens`; the `agentSwitched` rows), and label every miss with the hash that flipped relative to the previous call. That labelling **is** `missCause`, prototyped offline. Write the results into the queue entry. +- **Subtracts**: yes. It closes PR-5 on a recorded number, decides the Pacific-midnight item (count `system_prompt_changed` misses between 07:00 and 08:00 UTC), and either justifies building `missCause` into `prompt_cache.py` or retires it with a figure. +- **Effort**: Low (the hard part ran once already) +- **Impact**: High. One anatomy overturned a queued decision; one fleet pass overturned it back. +- **Evidence**: measured once, off-repo: 34,891 rows, 2026-09-06 → 27, load tests excluded. Only 18% of rows carried `prefixTokens`; the rest were imputed at the 15.8k median, which is the main caveat on the result. +- **Ship means**: the script in the repo, the numbers in the queue entry, in one PR. +- **Decline means**: decline `missCause` outright and log it; decide PR-5 on the off-repo number. +- **Recommendation**: **Ship, and this is the last carry.** If it is not picked this week, take the Decline branch on 10-09 rather than a sixth listing. + +### 6. Record and close the release-gated ops entries now that 1.25.0 is live +- **Source**: review-queue.md ▸ six `[2026-09-25]`/`[2026-09-26]` entries marked "blocked on release"; releases 1.25.0 (09-25) and 1.25.1 (09-26), which carry #1343, #1347, #1348, #1322, #1293/#1301, #1336 and #1332. +- **Surface area**: `docs/kaizen/review-queue.md` only, plus read-only prod checks. +- **Change**: per session notes (not the queue), three of the six already ran: the orphan-row cleanup (22 agents), the byte-counter repair (12 of 13 knowledge bases clean; the 13th waits on #1361 reaching `main`), and the log-retention sweep's first prod tick (`matched=3 updated=2 failed=0`). Record those counts and close the entries. Then run the three read-only checks still unrecorded: `global.*` native token counts on a few post-deploy Haiku 4.5 sessions; the reconciler's `tearingDown=` field and a ghost-row scan; the RAG bucket's drain in dev (due since 09-28) and the rule's presence in prod. The one write left for Phil is the log-retention one-off on the 8 legacy-named and 4 vended groups, still at null retention as of 09-27. +- **Subtracts**: yes. Six Open entries become at most two (the 13th KB settle, and the log-retention governance question for the cloud team). +- **Effort**: Low +- **Impact**: Med. Until the `global.*` check is recorded, the context meter, the cost page's `prefixTokens`, compaction's history calibration and the 1h-TTL arithmetic are all unverified for every production model. +- **Evidence**: session notes for the three that ran; the queue for the three that did not. None of it is in the repo yet, which is the point. +- **Ship means**: one read-only prod session and one queue PR. +- **Decline means**: the entries rot, and the next person re-derives whether the repair ran. +- **Recommendation**: **Ship.** It is 30 minutes and it is where this week's friction actually lives. + +### 7. Finish #1367: dev-validate a voice turn, then delete the now-redundant cache-write recovery +- **Source**: research/2026-10-02.md ▸ Top 5 #3 (the PR B half, overtaken); #1367's own "Dev validation after merge" section; `usage_normalization.py:33-42`. +- **Surface area**: `apis/shared/models/usage_normalization.py` (the `openai_cache_write_tokens` recovery, `:109-177`); the dev runtime (one text turn with a tool call and a cache read, one voice turn). +- **Change**: #1367 merged at 1.57.2, so both halves of the research's PR B are either done (voice on `strands.bidi`, off `session_manager=`) or now mechanical. The module's own docstring says the Responses cache-write recovery "is redundant on the current pin, and deleting it is a separate cleanup." Do the dev validation the PR asked for first, because a missed pairing shows up as a 30 s init timeout → 502 and not as a test failure; then delete the recovery, checking one cached GPT-5.6 Responses call prices identically before and after. +- **Subtracts**: yes. Dead code on the dominant OpenAI path, and the [2026-10-02] queue entry that asked for a split that no longer applies. +- **Effort**: Low +- **Impact**: Med. The validation is the real deliverable; the deletion is the subtraction. +- **Evidence**: verified on disk (docstring, imports, pins). The voice turn is the one unexercised path. +- **Ship means**: validation recorded on #1367, then one small PR. +- **Decline means**: the recovery stays as a harmless duplicate, and voice stays validated by unit tests only. +- **Recommendation**: **Ship the validation today; the deletion whenever convenient.** + +### 8. Tool rail: say "Preparing" until a tool is dispatched, and key message blocks by identity +- **Source**: research/2026-10-02.md ▸ Patterns (LibreChat #16619/#16455; assistant-ui #8684); both gaps verified by research on disk. +- **Surface area**: frontend. `tool-rail/tool-summary.ts:170` (`describeToolCall` treats any `pending` as running); `assistant-message.component.ts:161` (`@for … track $index`); the signals already exist (`agent_status` `tool_start` in `tool-insight.service.ts:207`, `inputComplete` at `assistant-message.component.ts:134`). +- **Change**: two labels instead of one ("Preparing X" while arguments stream, "Running X" after `tool_start`), and `track` on `toolUseId` for tool blocks, type plus ordinal otherwise. No SSE change. +- **Subtracts**: no. Addition, justified because on a large tool input (an MCP App, a document write) the argument-streaming window is most of the wait, and today it is labelled as execution. The identity key fixes a class of bug (an expanded rail or an App iframe attaching to the wrong block after a regroup) rather than one instance. +- **Unlocks**: honest status on the longest tool waits, and MCP App iframes that survive a mid-stream regroup. +- **Effort**: Low +- **Impact**: Med +- **Evidence**: verified on disk by research; no live reproduction of the regroup bug. +- **Ship means**: one SPA PR with a spec per change. +- **Decline means**: keep the single label and the index key. +- **Recommendation**: **Ship if a SPA slot opens, otherwise Defer 2 weeks (revisit 2026-10-16).** It is cosmetic next to #1–#7, and it is the kind of PR that bundles cleanly. + +### 9. Session-scoped MCP discovery and the sidebar search: re-scope both halves on this week's context +- **Source**: review-queue.md ▸ [2026-09-18]; reviews/2026-09-25.md ▸ #9 (deferred to 10-09, surfaced early because the context changed); #1406 (open); issue #1380 (09-28). +- **Surface area**: backend `external_mcp_client.py` (#1406 restructures exactly this loop); frontend `session-list`. +- **Change**: (a) The backend half should wait for #1406 to merge and be measured: it cuts the cold pre-flight from the sum of the servers to the slowest server (~9.5 s on the measured turn), which may be enough that session-scoped deferral no longer earns its complexity. Re-rank after a dev readout. (b) The search half now has a named user and a product-shaped request (#1380 asks for conversations, artifacts and agents). The client-side filter over loaded titles is still the right first slice; #1274's paging means it must say whether it spans unloaded pages. +- **Subtracts**: (a) possibly nothing once #1406 lands, which is the point of waiting; (b) no. +- **Unlocks**: a conversation search we have none of today, at zero token cost. +- **Effort**: (a) unknown until measured; (b) Low–Med +- **Impact**: Med +- **Evidence**: #1406 measured 21.5 s serial on dev; the parallel figure is asserted from local timing until the dev readout. #1380 is a request, not a measurement. +- **Ship means**: merge and measure #1406; ship the search slice as its own SPA PR. +- **Decline means**: no search; the pre-flight gets whatever #1406 gives it. +- **Recommendation**: **Defer (a) until #1406 has a dev readout; Ship (b) when a SPA slot opens.** Keep the 10-09 revisit for (a). + +## Carried Over From Prior Reviews + +- **Quality-veto harness slice 1** (deferred 2026-09-18 until **2026-10-02**, due today) — ✅ **Shipped, and it ran.** `56f9bc0b` (offline harness, slice 1), `291a55c8` ("the cut passes, the compression fails"), `d28f90fc` (extract-then-compress clears the veto), `3fdc05ae` (summarizer screening). The compaction waiver's trigger 1 fired and the stack passed. The document-offload veto (slice 2, live path) is still waived, correctly. *This review moves the [2026-09-21] harness entry to Resolved.* +- **Compaction v2 (#835) go/no-go** (revisit **2026-10-02**, due today) — the model-relative stack now has three weeks of prod data, extract-then-compress shipped on the v1 machinery and cleared the quality veto, and upstream's `ContextManager` is still converging. v2's remaining value was the frozen-segment engine. *Recommendation*: **Decline as scoped and log it.** Re-open only alongside adopting Strands' first-party context manager, never as a hand-built engine. The standing watch entry stays. +- **Platform-before-backend enforcement** (revisit **2026-10-02**, due today) — this window: Backend Deploy 23/23, Frontend 14/14, Platform Stack 3 success and 1 cancelled, zero failures. #1224 took the eviction half. *Recommendation*: **Decline the enforcement and log it.** The ordering convention stays in `CLAUDE.md`; re-open on the next `develop`-push failure of the "runtime may not exist yet" class. +- **`ClientSession` monkeypatch (b)/(c)** (re-gated last week on the mcp-2.x migration instead of a date) — new context: SEP-3371 (consistent SDK extension points, updated 10-02) is the long-run path that retires the substitution outright. *Recommendation*: keep gated; the queue entry's status now names SEP-3371. +- **Four decisions recommended last week and never logged.** Each needs one `decisions.md` entry. Drafts, for Phil to approve or edit: + - *[2026-10-02] Declined — docling #405 / Guardrails #480 as kaizen items.* Both stay as product backlog issues. Eight carries without a surface named; a kaizen item needs an entry point. Re-open only with a PR-sized slice (the docling bump alone qualifies if someone picks it up). + - *[2026-10-02] Declined — the standing "do derived execution contexts re-evaluate RBAC" audit.* Never had a surface. Projects (#1257/#1258) built a new derived context with its own named check (`apis/shared/projects/access.py`). Re-open as a named check on a named context, never as a standing audit. + - *[2026-10-02] Accepted — `bedrock-agentcore` #629 (TracerProvider not flushed before microVM freeze; last-turn telemetry under-reports).* Six cycles, upstream silent since August. Accept the under-report; re-open if the issue moves or a cost readout is visibly short by one turn per session. + - *[2026-10-02] Declined as scoped — reverse the Astra 272K decision.* `curated-models.ts:407-411` defends the cap as load-bearing pricing because `CuratedModel` has one rate per bucket. Re-open only alongside a per-tier rate field. +- **GPT-6 Luna side-channel A/B** (deferred 2026-09-25 until 2026-10-23) — not due. Note issue #1340 (admin-selectable summarizer model) is the same question; fold them when due. +- **`bedrock-agentcore` #646** (watch until 2026-10-09) — not due. +- **Named A2A participants** — keep deferred, no date. + +## Retirement Candidates + +- **PR-5, selective 1h static-prefix TTL (#1132, flag off)** — ⚠️ **the recommendation reversed and the queue doesn't know.** The queue says "recommend DECLINE" on a 65% break-even that `7f5f207f` failed at 23%. A session note dated 2026-09-27 says that rule used the wrong denominator: the 1h premium is paid only when the static prefix itself is re-written (`calculator.py:102` bills `min(cw, S − cr)`), so the test is avoided-TTL-misses over *remaining prefix writes*. Prod fleet, 34,891 rows, 09-06 → 27, load tests excluded: **0.78**, net **about +$22 over 21 days (~2% of spend)**, 476 of 1,350 sessions positive, and the anatomy session flips to +$2.23. Caveat: 82% of rows had no `prefixTokens` and were imputed at the median. *Recommendation*: **Decline on size, not on sign** (~$30/month is not worth a measured prod week), **or** flag it on in prod for one measured week. Either way, the queue entry and the thresholds spec §6 must carry the reversal; this review writes the number into the entry. +- **The [2026-10-02] "land #1367 at 1.57.1" queue entry** — superseded the morning it was written. Moved to Resolved in this PR; its residue is Proposal #7. +- **The [2026-09-04] W5 instance-SKU arithmetic (part 2)** — folds into Proposal #1 (the committed baseline is a published rate). Second listing; fold it now. +- **The `addPropertyOverride` step in the V2 plan** — retired by `aws-cdk-lib` 2.272.0's typed property (Proposal #1c). +- **The `cache_write_tokens` recovery in `usage_normalization.py`** — Proposal #7. +- **`scripts/sourcedir.tar.gz`** — gone (`ls` finds nothing). Stop listing it. + +## Risks Acknowledged But Not Acted On + +- **Preserved thinking vs history rewrites on Opus 5.5** (cookbook `bf24d45`) — *what breaks*: the first compaction or tool-result offload on a long Opus 5.5 conversation costs a full re-write at best and a rejected request at worst. Opus 5.5 is in prod. *Recommendation*: **Address now, Proposal #2(a).** +- **Opus 5.5 rejects forced `tool_choice`** — *what breaks*: any structured-output call on the user-selected model returns 400. Unverified whether one exists. *Recommendation*: **Address now, Proposal #2(b)**, it is a grep. +- **#1367's `maxInputTokens` fall-through to 1.05M for OpenAI rows missing the field** — *what breaks*: compaction cuts past Astra's 272K price tier on such a row. *Recommendation*: **Address in Proposal #2(d)**; it is a catalog read. +- **Strands 1.57.2 voice at runtime, not at import** — converted from a risk to a pending validation by #1367 merging with the fix. *Recommendation*: **Proposal #7, today.** `strands.experimental.bidi` removal in 1.60 is already handled (imports are `strands.bidi`, verified). +- **FastMCP can report success for a failed call** (#5304 `ResponseLimitingMiddleware` truncation, #5378 sync timeout) — *what breaks*: tool-rail failure counts and `agent_status ok=false` undercount on servers using either. *Recommendation*: **Accept, with a note.** The fix lives in the server repos (floor ≥ 4.0.9 unchanged); nothing to do here until a server adopts those features. +- **`bedrock-agentcore` #681: two discarded boto3 clients per `AgentCoreMemorySessionManager`** — the agent path is worked around (#1395); two app-api sites (`sessions/messages.py:389`, `shares/service.py:502`) are not, and neither is on the TTFT path. *Recommendation*: **Accept until the SDK fixes it.** +- **The nightly stays red until the stuck stack is cleared by hand** — *Recommendation*: **Address now, Proposal #3(a).** +- **Figma-style client allow-lists on third-party MCP servers** — *what breaks*: a catalogued external server can lock out custom clients. *Recommendation*: **Accept.** Risk to the external catalog, not an action. + +## What Shipped This Week + +*(100 PRs into `develop`; releases 1.25.0 on 09-25 and 1.25.1 on 09-26; zero reverts; zero `develop`-push deploy failures; nightly green 09-27 → 09-30, red 10-01 and 10-02 on the teardown.)* + +- **The turn path to first token, mapped and cut** (#1388, #1377, #1395, #1396, #1397, #1398, #1400, #1402, #1403, #1404): agent build 869 → 373 ms with a kill switch; the first-token clock on the EMF; the LTM prefetch overlapped with the message writes; the per-turn history count 388 → 0 ms (first token 1560 → 1092 ms on dev). Measured, not asserted, at every step. +- **The paired bump** (#1367): strands 1.57.2 + agentcore 1.24.0 + boto 1.43.103, pairing and `mcp<2` as tests, voice ported to the graduated Bidi API behind an unchanged WebSocket contract. +- **Last week's forum**: #1368 (non-`end_turn` = failed side-channel), #1366 + #1370 (Stop and page-hide on a finished turn), #1358 (Opus 5.5, GPT-6 Sol/Luna, GPT-5.5), #1376 (the V2 plan). +- **Compaction**: extract-then-compress default on (#1355), summary salvage at the token ceiling (#1353), the quality-veto harness slice 1 run against both defaults. +- **Long-term memory reaches the model** (1.25.0): relevance cut 0.7 → 0.4 on a labelled set, reconnect on dead pooled connections, score logging. +- **Managed-KB byte accounting** (#1347, #1348, #1361, #1365, #1373): deletes refund, imports reserve, synced changes re-ingest, failed stages roll back their gates. +- **Voice** (#1399): Nova 2 Sonic priced per modality; voice and microphone pickers in the composer. +- **CI** (#1392): the PR gate runs only the suites a PR's paths can reach; the backend job fans over logical CPUs. #1390/#1391 carry the sharding and fixture-scope follow-ups. +- **Observability and hygiene**: runtime log retention sweep live in prod (first tick clean), RAG bucket lifecycle, `UnmeteredModelCall` alarm, OTel context errors 5 → 0 per turn (#1357). + +## Take + +**The forum works; the ledger doesn't.** Four of six Ship items converted, and the one that didn't ran anyway, off the record. The system's current failure mode is not building things, it is **writing down what happened**: six ops entries unblocked for a week with no counts, four decisions recommended twice and logged zero times, a fleet measurement that reversed a queued decline and lives in a session note, and a finished curation on an unpushed worktree. Those are all one-commit fixes and they are in this review as Proposals #5, #6 and the Carried Over drafts. **The one change that would matter most this week is still #1**, because it is the only lever on 73% of the AICC bill and the price now gives it a threshold to clear. **The cheapest things on the board are the nightly unstick (one command) and the four `decisions.md` lines.** The retirement list is real this week (a superseded entry, a dead workaround, a recovery shim, and a reversed recommendation), which is a better sign than last week's thin one. + +--- + +## Review Protocol (for Phil) + +1. Read the scorecard and Friction (2 min). +2. Mark Proposals ✅ / ❌ / ⏸ (3–5 min). #3(a) and #7's validation are today's ops; #4 and #6 are under an hour each; #1 is a spec edit plus a script PR. +3. Retirement Candidates (1–2 min). PR-5 needs one keystroke: decline on size, or run the week. +4. Carried Over (2 min). **Four `decisions.md` entries are drafted above and need only approval**; two due-today deferrals (compaction v2, platform-before-backend) each recommend Decline. +5. Risks (1 min). +6. Pick 1–3 to ship. **Suggested: #1, #2, #3**, with #6 as the recording pass that costs nothing. + +Target: 10–15 minutes. + +## Post-review (for Phil — separate PRs) + +- ✅ Ship items → individual PRs over the week. +- ❌ Decline items → `docs/kaizen/decisions.md` with the reason (four drafts above). +- ⏸ Defer items → stay open in `review-queue.md` with their revisit date. + +### Queue movement in this PR + +Moved from Open to Resolved, based on outcomes recorded in reviews/2026-09-25.md and git: +- [2026-09-25] Treat any non-`end_turn` stop reason as a failed side-channel call → **SHIPPED** (#1368). +- [2026-09-25] Guard Stop against a turn that already finished → **SHIPPED** (#1366, #1370). +- [2026-09-25] Strands 1.57 + `bedrock-agentcore` 1.23.1 + boto paired bump → **SHIPPED** (#1367, at 1.57.2 / 1.24.0 / 1.43.103). +- [2026-10-02] Land #1367 at 1.57.1; take 1.57.2 as its own PR → **SUPERSEDED** (merged at 1.57.2 the same morning); residue is Proposal #7. +- [2026-09-21] Build the quality-veto harness → **SHIPPED** (slice 1) and run; slices 2–3 stay in the scoping doc. + +Status lines appended in Open (no entry moved): the Opus 5.5 entry (a shipped, b deferred to 10-23); the V2 A/B entry (plan merged, re-based by the [2026-10-02] entry); the `ClientSession` entry (gated on mcp 2.x, SEP-3371 named); the PR-5 entry and the fleet-sweep entry (the 09-27 reversal, recorded for the first time). diff --git a/docs/specs/agent-handoffs-and-consults.md b/docs/specs/agent-handoffs-and-consults.md index 7f3481ac5..ae00b0a24 100644 --- a/docs/specs/agent-handoffs-and-consults.md +++ b/docs/specs/agent-handoffs-and-consults.md @@ -1,9 +1,14 @@ # Agent handoffs and consults — carrying an `@`-mention's work across conversations -**Status:** Draft proposal. Not scheduled. **Phase 0 (task chips) ships first; its numbers decide whether Phase 1 +**Status:** Draft proposal. Not scheduled. **Phase 0 (handoffs) ships first; its numbers decide whether Phase 1 (consults) is built at all.** **Author:** Phil Merrell (drafted with Claude) -**Date:** 2026-09-24 +**Date:** 2026-09-24, revised 2026-09-27 +**Revision (2026-09-27):** A handoff now runs **in the background** while the user stays in the original thread, +after a **review step in place** (§4.2). It reports back through a handoff row with *running / needs your input / +done* states, the in-app inbox and browser notifications (§4.3). The feature is renamed from "task chips" to +**handoffs** (§4.0), because Shared Projects already uses "task" for a conversation. §4.6 and the §7 gate are +updated to match. **Targets branch:** `develop` **Related:** - `docs/specs/agent-marketplace.md` §D11 — the 2026-09-14 revision that made a mention *bind* (#1115) @@ -39,7 +44,7 @@ Two things are missing. They are separate, and this spec delivers them in order: We shipped a mid-thread mention once as a per-turn borrow (Phase 7, PR #738) and removed it (#1115). It failed in three ways, and both phases here are built to avoid all three: -| Failure | Mechanism | Phase 0 (chips) | Phase 1 (consult) | +| Failure | Mechanism | Phase 0 (handoffs) | Phase 1 (consult) | |---|---|---|---| | **History fork** | The mention changed the agent cache key, so a second `Agent` served the same session. Each instance held its own stale message list (#741, #751) | The work happens in an **ordinary new conversation** | The Agent runs in **its own hidden session** | | **Prefix re-write** | The Agent's bindings changed `toolConfig`, the first cachePoint, so system prompt and history were re-written behind it (~$0.12 per mention, measured) | The parent is never touched. A send-back is an append | The parent's `toolConfig`, system prompt and model never change. The report is an append | @@ -58,20 +63,25 @@ send-back is a row we can count. **§7 turns those counts into the go/no-go for | Phase | What the user gets | New plumbing | Status | |---|---|---|---| -| **0a — Handoff with context** | A mid-thread mention opens the Agent's conversation with a **context card** from the old thread, reviewed before anything is sent | `TASK#` rows, excerpt renderer, `handoff_context` on the request | Proposed | -| **0b — Send back** | "Send to *original conversation*" on a reply in a spawned conversation; lands as a context card on the parent's composer | Reuses 0a in the other direction | Proposed | -| **0c — Suggested tasks** | The model can offer a chip ("Build this rubric with Rubric Builder →") without stopping its turn | `suggest_task` tool, `task_suggested` SSE event | Proposed, opt-in | +| **0a — Handoff with context** | A mid-thread mention opens a **review sheet** in place. On Send, the Agent's conversation starts **in the background** with a **context card** from this thread; the user stays put | `HANDOFF#` rows, excerpt renderer, `handoff_context` on the request, a no-navigate send | Proposed | +| **0b — Handoff row + send back** | A row in the original thread tracks the handoff (*running / needs your input / done*) and offers **Bring back result** when it finishes; the new conversation offers "Send to *original conversation*" too | Status on `HANDOFF#`, reuses 0a in the other direction | Proposed | +| **0c — Suggested handoffs** | The model can offer a chip ("Hand off to Rubric Builder →") without stopping its turn; Start opens the same review sheet | `suggest_handoff` tool, `handoff_suggested` SSE event | Proposed, opt-in | | **Gate** | — | — | §7 | | **1 — Consult** | A mid-thread mention runs the Agent **in place** and its report comes back automatically | Hidden consult sessions, nested streaming, agent-resolution refactor | Only if the gate passes | -Out of both phases: **running a spawned task in the background** while the user stays in the original thread -(§4.6). +Background runs are **client-driven** in Phase 0: they live as long as the tab that started them (§4.6). +Handoffs that outlive the tab need the headless run entrypoint and are a follow-up. ## 3. What already exists - **A way to hand text to the composer.** `ComposerDraftService` puts text into the composer for a given session for the user to edit and send. Nothing is sent. Its first user is feedback retry-with-correction. Phase 0 is its second. +- **Concurrent streams, and a stream that keeps going when the user looks away.** The SPA's streaming layer is + keyed per session (`ChatStateService`, `StreamParserService`), and navigating away from a streaming conversation + does not abort it. When a stream finishes in a conversation the user isn't viewing, `setChatLoading` marks it + unread and the session list shows the blue dot. A background handoff reuses all of this. What's missing is a + send that doesn't navigate: `submitChatRequest` always calls `setViewedSession` and `navigateToSession`. - **Display text vs. model text.** `original_message` → `displayText` already separates what the user typed from the augmented prompt the model sees. That is how RAG chunks stay out of the rendered bubble. Context cards use the same split. @@ -101,41 +111,93 @@ Out of both phases: **running a spawned task in the background** while the user --- -## 4. Phase 0 — Task chips +## 4. Phase 0 — Handoffs + +### 4.0 Naming + +An earlier draft called Phase 0 "task chips", after Claude Code's `spawn_task` chip. That name is dropped from +copy, code and row keys, for two reasons: + +- It wasn't intuitive. In 0a and 0b the user never sees a chip, and in 0c the chip is only the entry point. +- **Shared Projects already uses "task" for a conversation in a project.** Users see a **Tasks** tab, "New task", + "Your tasks", "Shared with the project" and "Continue in my own task". The backend has `SHARED_TASK#` rows and + `/projects/{id}/tasks` routes. A handoff started from inside a project task would put two different "tasks" on + one screen. + +| Term | Means | In code | +|---|---|---| +| **Handoff** | Sending a request, plus reviewed context, to an Agent in a new conversation | `HANDOFF#`, `/sessions/{id}/handoffs`, `HANDOFFS_ENABLED`, `FEATURES.handoffs` | +| **Handoff row** | The line in the original thread that tracks a handoff (§4.3) | Replayed as `handoffs` on `GET /messages` | +| **Bring back result** / **Send to *{title}*** | Returning a handoff's result to the original thread, from either end | `kind: "send_back"` | +| **Suggested handoff** | A chip the model offers (§4.4) | `suggest_handoff`, `handoff_suggested` | +| **Consult** | Phase 1: the Agent runs in place | `CONSULT#` | +| **Task** | Reserved for Shared Projects. Not used by this feature | — | + +User-facing copy: "Hand off to *Rubric Builder*", "Handed off to *Rubric Builder*", "Bring back result", +"Send to *{title}*". Avoid "delegate", which already means delegated admin scopes in this codebase. ### 4.1 Decision summary | Question | Decision | |---|---| -| Unit of work | A **task**: a `TASK#` row on the *source* session. `kind` is `mention_handoff`, `suggested` or `send_back` | -| How context moves | A **context card**: a bounded, deterministic **excerpt** attached to the first message of the target conversation. **Visible and editable before sending** | +| Unit of work | A **handoff**: a `HANDOFF#` row on the *source* session. `kind` is `mention`, `suggested` or `send_back` | +| How context moves | A **context card**: a bounded, deterministic **excerpt** attached to the first message of the target conversation | +| Review | **Required, in place.** A review sheet opens in the source thread before anything is sent. The full prompt the Agent will receive is one click away | +| Where the user ends up | **Where they were.** The target conversation starts in the background and appears in the conversation list. The user opens it when they choose | +| How progress is shown | A **handoff row** in the source thread (*running*, *needs your input*, *done*, *stopped*), the existing unread dot, a toast, an entry in the **in-app inbox** (the bell), and a **browser notification** while the page is hidden. Never a model message | | Where the excerpt is rendered | **Server-side, once**, by a pure renderer in `apis.shared.handoff`. The SPA never builds it, because Phase 1 needs the same renderer in Python | -| Is anything sent automatically? | **No.** A handoff navigates and waits for the user's Enter. That costs one keystroke more than today, which is what makes the context reviewable | | How the model sees a card | Wrapped block **in the user message**; `displayText` = what the user typed; the card renders as a collapsible panel on the bubble | -| Model-suggested tasks | `suggest_task` tool, **non-blocking** (returns immediately; the turn continues). Constant spec. No Agent list in the schema | -| Linking | The target session records `spawnedFrom: {sessionId, taskId}`. The source's `TASK#` row records `targetSessionId` | -| Background execution | **Not in Phase 0** | -| Flags | Backend `TASK_CHIPS_ENABLED`, SPA `FEATURES.taskChips` (both default **off** in development); `suggest_task` separately gated by `TASK_SUGGESTIONS_ENABLED` | +| Background execution | **Client-driven.** The SPA runs the target's turn as a second stream. It lives as long as the tab (§4.6) | +| Model-suggested handoffs | `suggest_handoff` tool, **non-blocking** (returns immediately; the turn continues). Constant spec. No Agent list in the schema | +| Linking | The target session records `spawnedFrom: {sessionId, handoffId}`. The source's `HANDOFF#` row records `targetSessionId` and `status` | +| Flags | Backend `HANDOFFS_ENABLED`, SPA `FEATURES.handoffs` (both default **off** in development); `suggest_handoff` separately gated by `HANDOFF_SUGGESTIONS_ENABLED` | ### 4.2 Phase 0a — Handoff with context -**Flow.** The user types `@Rubric Builder draft a rubric for this` in a thread with history. +**Flow.** The user types `@Rubric Builder draft a rubric for this` in a thread with history and presses Enter. -1. The SPA calls app-api `POST /sessions/{id}/tasks` with `{kind: "mention_handoff", agentId, request}`, using - cookie auth (`get_current_user_from_session`). The server checks that the user owns the session, then: +1. **Nothing is sent yet.** The SPA calls app-api `POST /sessions/{id}/handoffs` with + `{kind: "mention", agentId, request}`, using cookie auth (`get_current_user_from_session`). The server checks + that the user owns the session, then: - renders the excerpt from persisted messages (§4.5) - - writes `TASK#` with `status: "open"` - - returns `{taskId, excerpt, excerptHash, request}` + - writes `HANDOFF#` with `status: "open"` + - returns `{handoffId, excerpt, excerptHash, excerptTokens, turnsCovered, contextBlock, request}`, where + `contextBlock` is the excerpt already wrapped as the Agent will receive it It does **not** create the target session. Nothing hides empty sessions from the list, so a row created here - would leave a blank "New Conversation" behind every abandoned handoff. -2. The SPA opens a new conversation with `?assistantId={agentId}`, using the staged-session path it already uses - for new chats, and hands `request` to that composer via `ComposerDraftService`. The **context card** sits above - the composer, like an attachment chip. It shows the source conversation's title, can be expanded to read in - full, can be edited, and can be removed. -3. On Enter, the invocation carries `handoff_context: {taskId, text}`, where `text` is the card as sent (possibly - edited), or it is absent if the card was removed. The server: - - checks that the task belongs to the user and is still `open` + would leave a blank "New Conversation" behind every cancelled handoff. +2. **A review sheet opens above the composer, in the source thread.** It is a labelled region, not a modal: the + thread stays readable above it, and Esc cancels. + + ``` + ┌ Hand off to [icon] Rubric Builder ──────────────────────────────────┐ + │ draft a rubric for this (edit) │ + │ ▸ Context from this conversation · turns 7–12 · ~3k tokens │ + │ Preview full prompt · Edit · Remove │ + │ Attached: syllabus-notes.docx │ + │ Cancel [ Send to Rubric Builder ] │ + └──────────────────────────────────────────────────────────────────────┘ + ``` + + - **Request**: what the user typed, minus the mention, editable. + - **Context card**: collapsed by default, showing the source range and size. **Edit** turns it into a text + area; **Remove** drops it. + - **Preview full prompt**: a read-only view of exactly what the Agent will receive: `contextBlock` (or the + edited card, wrapped the same way) followed by the request. The server performs the same join in PR-0.2; a + shared fixture pins the two so the preview can't drift from what is sent. + - **Attachments and invoked skills** from the mention message are listed; they follow the request, as they do + today. +3. **On Send**, the SPA generates the target session id and starts its turn **without navigating** (§3's no-navigate + send). The invocation carries `handoff_context: {handoffId, text}`, where `text` is the card as sent, or is + absent if the card was removed. In the source thread: + - the review sheet closes and the composer clears + - a **handoff row** appears (§4.3) + - a quiet toast confirms it: "Handed off to Rubric Builder · Open". It auto-dismisses and is announced politely + (`aria-live="polite"`), not assertively + + The new conversation appears at the top of the conversation list with the existing streaming dot. +4. **The server**, on the target's first turn: + - checks that the handoff belongs to the user and is still `open` - applies the excerpt cap to `text` again (the SPA can't raise it) - prepends the wrapped block to the model message: @@ -149,55 +211,142 @@ Out of both phases: **running a spawned task in the background** while the user ``` `original_message` = the user's message, so the bubble shows only what they typed. - - writes `spawnedFrom: {sessionId, taskId}` onto the new session's metadata as it is created by this first + - writes `spawnedFrom: {sessionId, handoffId}` onto the new session's metadata as it is created by this first turn - - marks the task `accepted` with `targetSessionId`, and records `contextEdited` (the sent text's hash ≠ - `excerptHash`) and `contextRemoved`. -4. The target is an ordinary bound launch from this point on. Binding, tools, interrupts, MCP Apps, artifacts, + - marks the handoff `running` with `targetSessionId`, and records `contextEdited` (the sent text's hash ≠ + `excerptHash`) and `contextRemoved` +5. The target is an ordinary bound launch from this point on. Binding, tools, interrupts, MCP Apps, artifacts, compaction and cost rows all behave as they do for any Agent conversation. +**The user's typed message never enters the source thread's history.** The source's model is not called, and its +prefix is untouched. The handoff row is the record of what happened (§4.3). + +**Why review in place.** The source thread is where the user can judge what should leave it: the conversation the +excerpt came from is on screen. Reviewing it after navigating away would mean judging it out of context, and it +would take the user out of the thread they meant to stay in. + **Why a card and not composer text.** An excerpt can be 4k tokens. Put in the textarea, it would bury the user's own sentence and could be sent half-edited by accident. A card keeps the request editable as a sentence and the context editable as a document, and lets the SPA show "Context from *Course planning*" on the sent bubble without parsing the prompt. -**Attachments and invoked skills** from the mention message follow the request to the target, as they do today. -Attachments in the *source* thread are named in the excerpt, not carried over (§4.5). +**Attachments** in the *source* thread are named in the excerpt, not carried over (§4.5). -**Cancelled handoffs.** If the user navigates away without sending, no session exists yet. The task stays `open`, -and one older than 24h reads as `abandoned` (derived when read, so no sweep job is needed). For a `mention_handoff`, -a removed context card still accepts the task, because the user did send; `contextRemoved` records the choice. +**Cancelled handoffs.** Cancel calls `POST /sessions/{id}/handoffs/{handoffId}/dismiss`, and no session is created. +If the user navigates away with the sheet open, the handoff stays `open`, and one older than 24h reads as +`abandoned` (derived when read, so no sweep job is needed). A removed context card still sends the handoff; +`contextRemoved` records the choice. -### 4.3 Phase 0b — Send back +**Inside a Shared Project.** A mention in a project task hands off to an ordinary conversation with the Agent. It +does not inherit `projectId` and doesn't appear in the project's Tasks list; the handoff row in the project task +links to it. -A conversation with `spawnedFrom` shows **"Send to *{source title}*"** in the action row of each assistant message. +### 4.3 Phase 0b — The handoff row and send back -1. `POST /sessions/{targetId}/tasks` with `{kind: "send_back", messageIndex}`. The server renders a **return - excerpt**: that assistant message's text, bounded at `SEND_BACK_MAX_TOKENS = 2000`, cut at a paragraph boundary - with a `[truncated]` marker. It writes a `TASK#` row on the **target**, and returns `{taskId, sourceSessionId, - excerpt, excerptHash}`. -2. The SPA navigates to the source conversation with the card on its composer (`from_conversation` = the spawned - conversation's title) and an empty request, and focuses the composer. -3. On send, the same `handoff_context` path appends the block to the source's next user message. +**The handoff row** sits in the source thread after the message that was last when the handoff was sent +(`sourceMessageIndex`). It is **UI only**: it comes from the `HANDOFF#` row, is replayed on +`GET /sessions/{id}/messages` as `handoffs` (like `toolSummaries`), and never enters the model's history. + +| State | Row reads | Set by | +|---|---|---| +| `running` | "*Rubric Builder* is working on *draft a rubric for this* · Open" | Acceptance on the first turn | +| `needs_input` | "*Rubric Builder* needs your input · Open" (amber) | The turn ends in an interrupt: OAuth consent, a tool approval, `ask_user_question`, a browser sign-in | +| `done` | "*Rubric Builder* finished · Open · **Bring back result**" | The turn completes | +| `stopped` | "*Rubric Builder* stopped · Open" | A `stream_error`, a Stop, or a dropped stream (§4.6) | + +While the target is streaming in this tab, the row follows the SPA's live per-session state. The persisted status +is written by inference-api **after** the turn's `done` block, on the same side of the stream as the `TSUM#` write, +so none of it is on the TTFT path. It follows the target's turns until the first `done` (a `needs_input` handoff +goes back to `running` when the user answers), then stays `done`: from there the conversation is the user's own. + +**Notifications.** The user hears about a handoff's state in three layers. Each one covers a case the one before +it can't. + +| Layer | Reaches the user when | What it shows | +|---|---|---| +| **The row, the unread dot and a toast** | They're in the app, on any page | Toast: "*Rubric Builder* finished · Open", or amber "*Rubric Builder* needs your input · Open", which stays until dismissed | +| **The in-app inbox (the bell)** | They missed the toast, reloaded, or come back later | An inbox entry per state change that needs them | +| **A browser notification** | The tab is open but hidden: another tab, another app | The same one line, from the operating system | + +A handoff that is waiting on the user and says nothing is the worst outcome of running in the background, which is +why *needs your input* uses all three. + +**The in-app inbox.** Shared Projects built a general-purpose inbox (`apis/shared/notifications/`, +`PK=INBOX#{email}`, `SK=NOTIF#{id}` on the projects table, 90-day TTL, `GET /notifications`, the bell in the +sidenav). It isn't tied to projects; handoffs become its second producer. + +- **New kinds:** `handoff_needs_input`, `handoff_done` and `handoff_stopped`. The payload is `{handoffId, + sourceSessionId, targetSessionId, agentId, agentName, sourceTitle}`. The request text is left out. +- **Written by inference-api**, beside the status write after the `done` block (§4.3's table), so no notification + is on the TTFT path. A tab that closes mid-turn still gets its `handoff_stopped`, because `_persist_interruption` + runs server-side. That entry is what the user sees on their next visit, since no toast was ever shown. +- **Read state follows the work, not the bell.** Opening the target conversation marks its `needs_input` and + `stopped` entries read. Opening it or using Bring back result marks `done` read. If the state change lands while + the user is viewing the source or the target, the SPA marks the entry read at once, so the bell counts only what + the user missed. +- ⚠️ Two changes this needs: + - **An IAM grant.** inference-api can `UpdateItem` on the projects table but not `PutItem`, and its policy comment + says it never creates rows there. Add a separate `PutItem` statement limited to inbox rows with a + `dynamodb:LeadingKeys` condition (`INBOX#*`), rather than widening the existing one. + - **The bell's visibility.** The sidenav shows the bell only when `features.projects` is on, because projects were + its only producer. Show it when `features.projects || features.handoffs`. + +**Browser notifications** use the **Notification API** (local): no service worker and no push server. + +- **Only when the page is hidden** (`document.visibilityState === "hidden"`) and a handoff turns *needs your input*, + *done* or *stopped* in this tab. When the page is visible, the toast is enough. +- **Permission is asked at a moment that makes sense.** The first time a handoff needs input, its row and toast offer + "Get a browser notification next time". There's also a setting. Never on page load: browsers downrank sites that + ask before the user has a reason. +- **Generic text:** "*Rubric Builder* needs your input". No request or conversation text, because these can show on + a lock screen. +- **One per handoff:** tagged with the `handoffId`, so a later state replaces the earlier one instead of stacking. + Clicking focuses the tab and opens the target. +- Without permission, or on a browser that doesn't support it (iOS Safari outside an installed web app), nothing + is lost: the inbox has the same entry. + +**Web Push** (notifications with the tab closed) is not in Phase 0. A Phase 0 handoff stops when its tab closes, so +Web Push would only ever announce a stopped run. It becomes worth building with durable handoffs (§4.6), and +scheduled runs have the same "your result is ready and you're not here" problem, so it should be built once for +both. + +**Interrupt prompts stay with the target.** The notification tells the user where to go; the prompt itself is +answered there. A consent, approval, question or sign-in prompt from the target's stream is held for that session +and shown when the user opens it. It must never render over the source thread. ⚠️ Verify first: the +concurrent-streaming refactor gated artifacts, compaction and MCP App state to the viewed session, but left the +OAuth, tool-approval, quota and error services session-tagged or global. Confirm each one scopes to its own +session before 0a ships. + +**Bring back result.** On a `done` row, **Bring back result** calls `POST /sessions/{targetId}/handoffs` with +`{kind: "send_back", messageIndex}` for the target's last assistant message. The server renders a **return +excerpt**: that message's text, bounded at `SEND_BACK_MAX_TOKENS = 2000`, cut at a paragraph boundary with a +`[truncated]` marker, and returns `{handoffId, sourceSessionId, excerpt, excerptHash}`. The card lands on the +source composer (`from_conversation` = the target's title) with an empty request. The user is already there, so +nothing navigates. + +**Send to *{title}*** does the same from the other end: a conversation with `spawnedFrom` shows it in the action row +of each assistant message, and it navigates to the source with the card on its composer. + +On send, the same `handoff_context` path appends the block to the source's next user message. **Cache.** The block goes into a *new* user message on the source. That is an append past the history cachePoint: the source's `toolConfigHash` and `systemPromptHash` don't move. Nothing in Phase 0 writes into existing history, the system prompt or the tool list. -**Trust.** A send-back is model output that may have been shaped by web pages or tool results in the spawned +**Trust.** A returned result is model output that may have been shaped by web pages or tool results in the target conversation. The framing sentence in the wrapper labels it. The user reads it and decides to send it, which is the strongest protection available. It is still not a guarantee, and the spec does not claim it is. -### 4.4 Phase 0c — Suggested tasks (`suggest_task`) +### 4.4 Phase 0c — Suggested handoffs (`suggest_handoff`) -The model can offer a task without being asked: "This rubric deserves its own conversation with Rubric Builder." -It works like Claude Code's `spawn_task` chip. +The model can offer a handoff without being asked: "This rubric deserves its own conversation with Rubric +Builder." It works like Claude Code's `spawn_task` chip. **Tool spec (constant):** ```json { - "name": "suggest_task", + "name": "suggest_handoff", "description": "Offer the user a separate conversation for a piece of work that would sidetrack this one …", "inputSchema": { "type": "object", @@ -212,41 +361,40 @@ It works like Claude Code's `spawn_task` chip. } ``` -- **Non-blocking.** The handler writes `TASK#` (`kind: "suggested"`), queues a `task_suggested` event, and +- **Non-blocking.** The handler writes `HANDOFF#` (`kind: "suggested"`), queues a `handoff_suggested` event, and returns `"Suggestion shown to the user."` The turn continues. It never pauses through `ToolContext.interrupt`; this is not `ask_user_question`. - **No Agent list in the schema.** An `enum` of the user's Agents would change whenever pins change, and each change would re-write `toolConfig`, the first cachePoint. `agent` is free text, matched on the server against the - user's own and pinned Agents (exact name, then case-insensitive). If nothing matches, the chip shows an Agent - picker. The model can see Agent names only if the user mentions them. That is fine: a suggestion without an Agent - is still a useful chip. -- **The prompt is the brief.** Unlike 0a, the model writes the brief, which is the best brief this spec can get. - The chip's **Start** button fetches the task (`GET /sessions/{id}/tasks/{taskId}`) and opens the new - conversation exactly as 0a does, with `prompt` as the card text and an empty request. The user still sees and - can edit it before sending, and the first send accepts the task. -- **Rate limits.** At most one `suggest_task` per turn and three per session. Further calls return `"Suggestion - limit reached; mention it in your reply instead."` rather than failing. + user's own and pinned Agents (exact name, then case-insensitive). If nothing matches, the review sheet shows an + Agent picker. The model can see Agent names only if the user mentions them. That is fine: a suggestion without an + Agent is still a useful chip. +- **The prompt is the request.** The chip's **Start** button fetches the handoff + (`GET /sessions/{id}/handoffs/{handoffId}`) and opens the **same review sheet** as 0a, with the model's `prompt` + as the editable request and the excerpt as the context card. Send runs it in the background exactly as 0a does. +- **Rate limits.** At most one `suggest_handoff` per turn and three per session. Further calls return + `"Suggestion limit reached; mention it in your reply instead."` rather than failing. - **Cost, per the token cost tenet.** The spec is in `toolConfig` for every session that has it. It is read from cache on every call and written once for everyone when it ships. Size it by **marginal CountTokens** (tool list with and without it), not by eyeballing the JSON. Expected: 200–400 tokens. - **Steering.** `ask_user_question` needed a **system-prompt clause** before the model used it well. Expect the same here, and budget for it: a base-prompt change is a one-time cache re-write for every user. Ship the tool without the clause first, measure how often it's used, and add the clause only if it's under-used. -- **Gating.** `TASK_SUGGESTIONS_ENABLED` defaults to off. The catalog entry is `enabledByDefault: false` until the - feedback cohort has lived with it, because a chip-happy model is worse than no chips. +- **Gating.** `HANDOFF_SUGGESTIONS_ENABLED` defaults to off. The catalog entry is `enabledByDefault: false` until + the feedback cohort has lived with it, because a chip-happy model is worse than no chips. **SSE event** (added to the `CLAUDE.md` table in the same PR): | Event | Payload | When | |---|---|---| -| `task_suggested` | `{type, sessionId, taskId, title, tldr, agentId?, agentName?}` | Drained after the tool's `tool_result`, like `steering_applied` | +| `handoff_suggested` | `{type, sessionId, handoffId, title, tldr, agentId?, agentName?}` | Drained after the tool's `tool_result`, like `steering_applied` | -The prompt is **not** on the event. It is fetched when the user presses Start (from the `TASK#` row), so a long +The prompt is **not** on the event. It is fetched when the user presses Start (from the `HANDOFF#` row), so a long brief isn't streamed for chips the user dismisses. **Chip UI.** The chip renders inline after the assistant message that produced it, with **Start** and -**Dismiss**. Chips replay on reload from `GET /sessions/{id}/messages`, which returns them as `tasks` (like -`toolSummaries`). Dismiss is `POST /sessions/{id}/tasks/{taskId}/dismiss`. +**Dismiss**. Once started, it becomes that handoff's row (§4.3). Chips replay on reload with the rest of +`handoffs`. Dismiss is `POST /sessions/{id}/handoffs/{handoffId}/dismiss`. ### 4.5 The excerpt renderer (shared with Phase 1) @@ -276,36 +424,65 @@ A **model-written** brief for 0a (a Nova Micro side-channel like `session_title` uncached 50k-token thread) is a deliberate follow-up, not v1. Build the deterministic version first. Replace it only if `contextEdited` shows users rewriting it. -### 4.6 What Phase 0 leaves out - -- **Background execution.** "Start this and let me keep working here" is appealing, and most of the parts exist: - the headless run entrypoint (scheduled runs F1), the unattended-run unread dot, and pause prompts persisted as - breadcrumbs. But it spends quota with nobody watching, and it inherits F1's per-owner token mint. A follow-up, - after Phase 0's numbers. +### 4.6 Background runs, and what Phase 0 leaves out + +**A background handoff is a second stream in the same tab.** That makes it cheap: no headless run entrypoint, no +per-owner token mint, no server-side SSE reader. The user's session cookie and the ordinary chat path do the work. +It also sets the limits: + +- **It lives as long as the tab.** Closing the tab, or losing the network mid-turn, drops the SSE stream. + Starlette cancels the response and the turn is persisted as **interrupted** (`connection_lost`); it does **not** + finish on its own. The row reads *stopped*, and the target shows the existing interrupted-turn marker with + Continue. While any handoff is running, the SPA's existing "is anything streaming" check (`ChatStateService`) + also guards a page unload, so the browser asks before the tab closes. +- **The quota is the user's, and they started it.** It is one reviewed turn, not an unattended loop, so the + original objection to background work (quota spent with nobody watching) doesn't apply. `quota_warning` and + `quota_exceeded` behave as they do for any turn. +- **No cap on concurrent handoffs in v1.** Each target has its own session and lease, so two handoffs can't + collide. Measure how many run at once before adding a cap. +- **TTFT is unchanged.** The source's next turn doesn't wait for anything, and the target's first-token path adds + only a string prepend. + +**Left out of Phase 0:** +- **Handoffs that outlive the tab.** The durable version runs the target through the headless run entrypoint + (scheduled runs F1), with the server-side `unread` flag and persisted pause breadcrumbs. It inherits F1's + per-owner token mint. Build it only if §4.7's `HandoffStopped` shows dropped streams are common. +- **Web Push.** Notifications with the tab closed need a service worker, VAPID keys, stored subscriptions and a + backend sender. They pay off only with durable handoffs, and should be built once for handoffs and scheduled runs + (§4.3). - **Automatic results.** That is Phase 1. -- **Chips that cross users.** A task is always started by the user who owns the source conversation. +- **Handoffs that cross users.** A handoff is always started by the user who owns the source conversation. +- **Telling the source's model.** The source's model doesn't know a handoff happened until a result is brought + back (§12 Q2). ### 4.7 Phase 0 measurement EMF, under the existing PromptCache namespace so the cost dashboard can show it: -- `TaskCreated` by `kind` -- `TaskAccepted` / `TaskDismissed` / `TaskAbandoned` by `kind` +- `HandoffCreated` by `kind` +- `HandoffSent` / `HandoffCancelled` / `HandoffAbandoned` by `kind` +- `HandoffPromptPreviewed`: the review sheet's full-prompt preview was opened - `HandoffContextEdited` / `HandoffContextRemoved` -- `SendBack` +- `HandoffCompleted` / `HandoffNeedsInput` / `HandoffStopped` +- `HandoffOpenedWhileRunning`: the user opened the target before it finished +- `HandoffNotified` by channel (`inbox`, `browser`), and `HandoffNotificationOpened` by channel +- `HandoffBrowserPermission` by outcome (`granted`, `denied`, `dismissed`) +- `SendBack` by origin (`row` for Bring back result, `target` for Send to *{title}*), with `openedTargetFirst` -The same facts are on the `TASK#` rows, so the per-user view can be queried without the metrics. +The same facts are on the `HANDOFF#` rows, so the per-user view can be queried without the metrics. --- ## 5. Goals and non-goals, by phase **Phase 0 goals** -- A mid-thread mention starts the Agent's conversation **with context the user has reviewed**. -- A spawned conversation's results can go back to the original thread in **one action**. -- The model can offer a task without derailing its turn. +- A mid-thread mention starts the Agent's conversation **with context the user has reviewed**, without taking the + user out of the thread they were in. +- The user can see whether a handoff is running, needs them, or is done, from the original thread. +- A handoff's result can go back to the original thread in **one action**. +- The model can suggest a handoff without derailing its turn. - No prefix is ever re-written; no session is ever served by two configurations. -- Every handoff, edit and send-back is counted. +- Every handoff, review action, completion and send-back is counted. **Phase 1 goals** (only if the gate passes) - A mid-thread mention can run the Agent **in place**, with a **bounded report** that the conversation's own agent @@ -313,7 +490,7 @@ The same facts are on the `TASK#` rows, so the per-user view can be queried with - Repeated consults of the same Agent in the same thread continue that Agent's own session. **Non-goals (both phases)** -- Nesting: a consulted Agent or spawned task cannot itself consult. Depth is 1. +- Nesting: a consulted Agent or a handoff's target cannot itself consult. Depth is 1. - Changing launch semantics: a mention on an empty thread still launches and binds. - An A2A server. If this ever runs across a Runtime boundary, the `streaming=True` rule in `CLAUDE.md` applies. @@ -321,8 +498,8 @@ The same facts are on the `TASK#` rows, so the per-user view can be queried with | Flag | Where | Default in development | Kill switch once shipped | |---|---|---|---| -| `TASK_CHIPS_ENABLED` / `FEATURES.taskChips` | backend / SPA | off | yes; off = today's immediate handoff | -| `TASK_SUGGESTIONS_ENABLED` | backend | off | yes; off = `suggest_task` never registered | +| `HANDOFFS_ENABLED` / `FEATURES.handoffs` | backend / SPA | off | yes; off = today's immediate handoff | +| `HANDOFF_SUGGESTIONS_ENABLED` | backend | off | yes; off = `suggest_handoff` never registered | | `AGENT_CONSULT_ENABLED` / `FEATURES.agentConsult` | backend / SPA | off | yes; off = Phase 0 behaviour | With a backend flag off, a request field it would have consumed (`handoff_context`, `agent_consult`) is @@ -330,13 +507,20 @@ With a backend flag off, a request field it would have consumed (`handoff_contex ## 7. The gate: does Phase 1 get built? -After **six weeks** of Phase 0 in prod at default-on, read the `TASK#` rows and EMF counts: +A background handoff with **Bring back result** already delivers most of what a consult does. Phase 1 adds only +the last step: the result comes back and is answered from **without** the user pressing anything. So the gate asks +whether users already do that step by hand, every time, without looking. + +After **six weeks** of Phase 0 in prod at default-on, read the `HANDOFF#` rows and EMF counts: | Signal | Reading | Implication | |---|---|---| -| `mention_handoff` volume | Small share of Agent conversations | **Stop.** Mid-thread use of Agents is rare; chips are enough | -| `SendBack` / accepted handoffs | **High** (proposed bar: ≥ 25%) | People want the result *in the original thread*. **Build Phase 1** | -| `SendBack` / accepted handoffs | Low | People are happy to keep working in the new conversation. **Stop** | +| `mention` volume | Small share of Agent conversations | **Stop.** Mid-thread use of Agents is rare; handoffs are enough | +| `SendBack` / completed handoffs | **High** (proposed bar: ≥ 25%), mostly **without** `openedTargetFirst` | Users bring the result back unread, so the manual step is overhead. **Build Phase 1** | +| `SendBack` / completed handoffs | High, mostly **after** opening the target | Users work in the target before bringing anything back. Background handoffs fit; a consult would cut that work short. **Stop** | +| `SendBack` / completed handoffs | Low | People are happy to keep working in the new conversation. **Stop** | +| `HandoffOpenedWhileRunning` | High | Users don't want to wait in the background. Consider opening the target on Send as a per-user preference, not Phase 1 | +| `HandoffNeedsInput` / `HandoffStopped` | High | Background runs stall on the user or on the tab. Fix that (interrupt UX, or the durable F1 path in §4.6) before anything else | | `HandoffContextEdited` | High | Improve the brief (model-written, §4.5) **before** Phase 1, which uses the same renderer | | `HandoffContextRemoved` | High | The excerpt is unwanted or distrusted. Investigate before sending excerpts *automatically* in Phase 1 | @@ -347,7 +531,7 @@ already reached. ## 8. Phase 1 — Consults -Everything below is built only if §7 says so. It reuses Phase 0's excerpt renderer, `TASK#`-style linking and +Everything below is built only if §7 says so. It reuses Phase 0's excerpt renderer, `HANDOFF#`-style linking and wrapper conventions. ### 8.1 Decision summary @@ -371,7 +555,7 @@ wrapper conventions. With both features on, a mid-thread mention offers two actions in the `@` menu's confirm step: - **Ask here**: a consult (this phase). -- **Open with context**: a Phase 0a handoff. +- **Hand off**: a Phase 0a handoff, reviewed and run in the background. The gate's data picks the default. `routeMention` gains the consult outcome; the empty-thread and same-Agent branches stay as they are: @@ -381,7 +565,7 @@ if (threadHasMessages) { if (choice === 'consult' && FEATURES.agentConsult) { return { sessionId, assistantId: boundAssistantId, handedOff: false, consultAgentId: mentionedAgentId }; } - return { sessionId: null, assistantId: mentionedAgentId, handedOff: true }; // Phase 0a path + return { sessionId: null, assistantId: mentionedAgentId, handedOff: true }; // Phase 0a: review sheet, background send } ``` @@ -558,15 +742,33 @@ v2 would add a `consult` `PendingInterrupt` kind that points at the consult sess **Phase 0** - **Excerpt renderer.** Deterministic output for the same input, whole-turn truncation, compaction-summary substitution, attachments named not inlined, and a `since_index` that only moves forward. -- **Task lifecycle.** Owner scoping on every `/tasks` route (cookie auth; another user's task id → 404), - `open → accepted | dismissed | abandoned`, and no double-accept. +- **Handoff lifecycle.** Owner scoping on every `/handoffs` route (cookie auth; another user's handoff id → 404), + `open → running → needs_input ⇄ running → done | stopped`, `open → dismissed | abandoned`, and no double-send. - **`handoff_context`.** The server applies the cap again; `displayText` is the user's text only; `contextEdited` / `contextRemoved` are recorded correctly; a flag-off server ignores the field. +- **Prompt preview parity.** A shared fixture: the SPA's preview of `contextBlock` + request equals the model message + the server builds, byte for byte. +- **Status writes.** Written after the `done` block, never before the first token; `needs_input` for each interrupt + kind; `stopped` on `connection_lost`; frozen after the first `done`. +- **Inbox writes.** One entry per state change that needs the user, written after `done`; `handoff_stopped` on + `connection_lost`; no request text in the payload. The IAM statement allows `PutItem` only on `INBOX#` keys + (CDK assertion test). - **Send-back prefix stability.** The source's `toolConfigHash` and `systemPromptHash` are byte-identical before and after a send-back turn. -- **`suggest_task`.** Non-blocking return; per-turn and per-session limits; unmatched `agent` still produces a +- **`suggest_handoff`.** Non-blocking return; per-turn and per-session limits; unmatched `agent` still produces a chip; the spec is byte-identical across users with different pins. -- **SPA.** Extend the mirrored `routeMention` tests; the card renders, edits and removes; replay from `tasks`. +- **SPA.** + - Extend the mirrored `routeMention` tests: a mid-thread mention opens the review sheet and sends nothing. + - The no-navigate send leaves `viewedSessionId` and the route on the source. + - The sheet's card renders, previews, edits and removes; Cancel dismisses. + - The handoff row renders each state live and on replay from `handoffs`. + - A target's interrupt prompt never renders over the source thread. + - The bell shows with `features.handoffs` alone, and renders the three handoff kinds. + - Entries are marked read by opening the target or bringing a result back, and at once when the change lands in + view. + - A browser notification fires only while the page is hidden, never before permission, and a later state + replaces the earlier one. + - Axe clean on the sheet and the row, in both themes. **Phase 1** - A consult turn leaves the parent's `toolConfig` and system prompt byte-identical (fingerprint hashes). @@ -575,29 +777,44 @@ v2 would add a `consult` `PendingInterrupt` kind that points at the consult sess that still runs. The next consult supersedes the pause (drive the real `_InterruptState`). - The excerpt watermark moves forward only. -**Both:** import boundaries. `apis.shared.handoff` holds the renderer and the `TASK#` store; `app_api` and +**Both:** import boundaries. `apis.shared.handoff` holds the renderer and the `HANDOFF#` store; `app_api` and `inference_api` import only from there. ## 10. Delivery plan **Phase 0** -- **PR-0.1 — Excerpt renderer + `TASK#` store** in `apis.shared.handoff`, and the app-api `/tasks` routes - (create, get, dismiss). Acceptance happens on the first send (PR-0.2), not as a route. Flags off. -- **PR-0.2 — `handoff_context` on the invocation path.** Wrapper, `displayText`, task acceptance, EMF. -- **PR-0.3 — SPA handoff.** `routeMention`'s handoff calls `/tasks`, navigates, and shows the context card; the - card on sent bubbles; the toast copy is updated. -- **PR-0.4 — Send back.** Action on assistant messages in spawned conversations, and a card on the source composer. -- **PR-0.5 — `suggest_task`.** Tool, `task_suggested` event, chip UI, replay, `CLAUDE.md` SSE table. Marginal - CountTokens measured and recorded in the PR. -- **Dev validation.** A mid-thread mention with an edited card; a send-back; a suggested chip that is started and - one that is dismissed. The source's `C#` rows show no prefix change on the send-back turn. Then flip defaults on - and start the §7 clock. +- **PR-0.1 — Excerpt renderer + `HANDOFF#` store** in `apis.shared.handoff`, and the app-api `/handoffs` routes + (create, get, dismiss). The create response includes `contextBlock` for the preview. Acceptance happens on the + first send (PR-0.2), not as a route. Flags off. +- **PR-0.2 — `handoff_context` on the invocation path.** Wrapper, `displayText`, acceptance, status writes after + `done`, EMF. +- **PR-0.3 — SPA handoff.** A mid-thread mention opens the review sheet (request, context card, full-prompt + preview); a no-navigate send; the handoff row with live and replayed states; completion and needs-input toasts; + the card on sent bubbles; the unload guard. Verify interrupt-service scoping (§4.3) here. +- **PR-0.3b — Notifications.** The three `handoff_*` inbox kinds, written by inference-api after `done`; the scoped + `PutItem` grant; the bell shown with `features.handoffs` and rendering the new kinds; read-state rules; browser + notifications through the Notification API, with the in-context permission ask. +- **PR-0.4 — Send back.** Bring back result on a `done` row, Send to *{title}* on assistant messages in spawned + conversations, and the card on the source composer. +- **PR-0.5 — `suggest_handoff`.** Tool, `handoff_suggested` event, chip UI that opens the review sheet, replay, + `CLAUDE.md` SSE table. Marginal CountTokens measured and recorded in the PR. +- **Dev validation.** + - A mid-thread mention with an edited card, previewed, sent, and finished while the user stays in the source. + - A handoff that needs OAuth consent: *needs your input* on the row and in the bell, a browser notification with + the tab hidden, and the prompt only in the target. + - A tab closed mid-run: *stopped*, a `handoff_stopped` entry in the bell on the next visit, and Continue works + in the target. + - Bring back result, and a send-back from the target. + - A suggested chip that is started, and one that is dismissed. + - The source's `C#` rows show no prefix change on the send-back turn. + + Then flip defaults on and start the §7 clock. **Phase 1** (after the gate) - **PR-1.1 — Extract `resolve_turn_agent`.** Pure refactor. - **PR-1.2 — Backend consult** (flag off): session prefix + list filter, orchestration, report, `CONSULT#` rows, SSE events, metering attributes, interrupt handling, `CLAUDE.md` SSE table. -- **PR-1.3 — SPA**: Ask here / Open with context choice, consult card, disclosure, replay. +- **PR-1.3 — SPA**: Ask here / Hand off choice, consult card, disclosure, replay. - **PR-1.4 — Transcript view + delete/export integration.** - **Dev validation:** a consult, a follow-up consult continuing the consult session, an OAuth-blocked consult, a Stop mid-consult; parent rows show `hit` with unchanged hashes; measured costs replace §8.10's estimates. @@ -606,7 +823,7 @@ v2 would add a `consult` `PendingInterrupt` kind that points at the consult sess - **A. Parent-initiated `consult_agent` tool** (automatic, not a chip). Needs a constant `toolConfig` spec with no Agent list; Strands re-runs an interrupted tool from the top; it costs an extra parent call. Phase 0c's - `suggest_task` gets most of its value without the nested execution, and a later `consult_agent` could reuse the + `suggest_handoff` gets most of its value without the nested execution, and a later `consult_agent` could reuse the Phase 1 consult function as its handler. - **B. Hand the other Agent the full parent history.** Best answer quality, but the most expensive option: about $0.21 per handoff or consult on a 50k-token thread on Sonnet 4.6. Rejected; the excerpt cap can be tuned upward. @@ -616,19 +833,35 @@ v2 would add a `consult` `PendingInterrupt` kind that points at the consult sess Phase 0 answers the question and reviews the context, for a fraction of the work. - **E. Run consults through the headless run entrypoint (F1)** as a separate Runtime invocation. Real isolation, but it brings in the per-owner token mint, a second container and server-side SSE reading. Revisit if consults, or - background tasks (§4.6), ever need to outlive the request. + background handoffs (§4.6), ever need to outlive the request or the tab. +- **F. Navigate to the target on Send** (this spec's first draft of 0a). Simple, but it takes the user out of the + thread they meant to stay in, and reviewing the context after leaving it means judging it out of context. + Replaced by the in-place review and background run. §7's `HandoffOpenedWhileRunning` would show if users want it + back as a preference. +- **G. Have the source's model announce the handoff.** It costs a model call on the source, adds latency before + anything shows, and has the model describe an action it didn't take. The handoff row says the same thing, + accurately, for nothing. ## 12. Open questions 1. **Excerpt size.** Is 4k tokens enough in practice? Measure `HandoffContextEdited` before tuning. -2. **Auto-send vs. review.** Phase 0a adds one Enter compared with today's immediate handoff. If - `HandoffContextEdited` and `HandoffContextRemoved` are both near zero, should the handoff send immediately with - the card attached? +2. **Should the source's model learn about a handoff?** Today it doesn't until a result is brought back, so "what + did Rubric Builder say?" gets a blank. One option: on the source's next user turn, append a one-line note + (`[Handed off "draft a rubric for this" to Rubric Builder]`). It is append-only, so the cached prefix is safe, but + it is a small per-handoff token cost and a note the user didn't write. 3. **Direct mode (Phase 1).** For Agents whose output *is* the answer, should a consult end the turn with the report as the assistant message and skip the parent call? 4. **Cohort.** An RBAC capability for a feedback cohort first, as scheduled runs did, or flags alone? -5. **Suggestion steering.** Does `suggest_task` need a system-prompt clause, and is its one-time fleet-wide cache +5. **Suggestion steering.** Does `suggest_handoff` need a system-prompt clause, and is its one-time fleet-wide cache write worth paying? 6. **Gate thresholds.** Agree §7's bars before Phase 0 ships. 7. **Model pinning (Phase 1).** An Agent that pins its own model runs the consult on it. Should the card say so, like the composer's "set by this agent" label? +8. **Review sheet default.** Review is required (decided 2026-09-27). Open is only whether the context card + should start expanded for a user's first few handoffs. `HandoffPromptPreviewed` will say whether people look. +9. **Should `handoff_done` go to the inbox at all?** It is the noisiest kind. If `HandoffNotificationOpened` shows + done entries are rarely opened from the bell, keep `done` to the toast and browser notification, and reserve the + inbox for states that need the user. + +**Decided (2026-09-27):** review before sending is required, and the handoff never sends immediately. This replaces +the earlier open question about auto-send. diff --git a/docs/specs/agentcore-runtime-v2.md b/docs/specs/agentcore-runtime-v2.md new file mode 100644 index 000000000..7b5fb9b9b --- /dev/null +++ b/docs/specs/agentcore-runtime-v2.md @@ -0,0 +1,185 @@ +# AgentCore Runtime V2 migration + +**Status:** plan, with pre-work findings. Tracks `docs/kaizen/review-queue.md ▸ [2026-09-25] A/B the V2 AgentCore Runtime in dev` (Proposal 1 in `reviews/2026-09-25.md`). +**Sources:** +- The AWS ML blog post [The new AgentCore Runtime: elastic, optimized, and consistently fast starts](https://aws.amazon.com/blogs/machine-learning/the-new-agentcore-runtime-elastic-optimized-and-consistently-fast-starts/) (2026-09-18). +- The [What's New post](https://aws.amazon.com/about-aws/whats-new/2026/09/new-agentcore-runtime-generally-available). +- The CloudFormation registry schema for `AWS::BedrockAgentCore::Runtime` (us-west-2), read on 2026-09-27. +- botocore's `bedrock-agentcore-control` service model on `develop`. +- A read-only `GetAgentRuntime` against the dev runtime. + +Labels used in this document: +- **Verified**: checked against AWS or our tree on 2026-09-27. +- **Asserted**: AWS says it, and we have not reproduced it. +- **Hypothesis**: a risk we inferred, not observed. + +--- + +## 1. What V2 changes (asserted by AWS) + +| | V1 (what we run today) | V2 | +|---|---|---| +| Opt-in | none | `platformVersion: "V2"` on Create/UpdateAgentRuntime | +| How an instance starts | pulls the image and boots the container per session | **loads the agent once, snapshots the running environment, and restores that snapshot for every new instance**. Excess transient state is stripped so snapshot size stays steady | +| Cold start (P75, echo agent) | rises with image size: 5.4 s at 200 MB, ~30 s at 2 GB | **~2 s flat** from 200 MB to 2 GB. Image size stops mattering | +| Memory | allocated memory is held, so usage tracks the **high watermark** | paged in on demand. **Freed or cold memory is reclaimed** | +| Billing | GB-hours at peak | **"a higher rate but on far fewer GB-hours"**, following the memory actually used. There is no standing charge for provisioned capacity | +| Architecture | ARM64 | ARM64 only. x86 is "coming soon" | +| Regions (What's New) | — | us-east-1, us-east-2, **us-west-2**, eu-west-1, ap-northeast-1 | + +The blog lists these as **coming soon**: +- Suspend and resume sessions with memory snapshotting. +- Serialize state before an active session terminates, "so sessions can resume indefinitely". +- Per-session scoped identity ("session context keys"). +- Larger RAM, vCPU and session storage. +- A committed-baseline discount: reserve a memory floor and burst above it on demand. + +**What AWS does not say:** +- No V2 price. The pricing page had no V2 row as of the 09-25 research. +- Nothing about the `/ping`/`/invocations` contract or `lifecycleConfiguration`. +- **When** in the container's startup the snapshot is taken. + +That last gap drives most of the risk in §3. + +## 2. Gate 1: will CloudFormation take it? Answered: yes (verified) + +- **The CFN schema has `PlatformVersion`.** It is a string of 1–128 non-whitespace characters with no enum. +- **It is not `createOnlyProperties`.** Only `AgentRuntimeName` is. Flipping the version is an **in-place update**: the runtime ID and ARN stay the same, so SSM `//inference-api/runtime-id` does not rotate. +- `aws-cdk-lib` 2.270.0 still has no typed prop, so the change is `runtime.addPropertyOverride('PlatformVersion', ...)`. +- **The dev runtime reports `platformVersion: 'V1'` today**, with `lifecycleConfiguration` `{idleRuntimeSessionTimeout: 900, maxLifetime: 28800}` (the defaults; our construct sets none). So the V1 literal is `"V1"`, which the rollback plan below depends on. +- **Our SDKs are too old to see the field:** + - botocore `develop` has `platformVersion` on `CreateAgentRuntimeRequest`, `UpdateAgentRuntimeRequest` and `GetAgentRuntimeResponse`. + - Our pin (`boto3==1.43.68`) does not, and neither does a local `aws-cli` 2.22.12. Latest botocore on PyPI is 1.43.103. + - The dev read above was done by pointing `AWS_DATA_PATH` at the upstream model, not by bumping anything. + +## 3. Blockers and risks found in our tree + +### B1. `backend.yml` would strip V2 on every image deploy (verified: the code path exists; the outcome is unverified) + +`scripts/build/deploy-runtime-image-if-changed.sh` rolls a new image with `update-agent-runtime`, which is a **full-replacement** API. It rebuilds the payload from `get-agent-runtime` through an `ALLOWED` allow-list, and **`platformVersion` is not on it**. `capacityProviderConfiguration` is not on it either. + +What happens next depends on the runner's CLI: +- **CLI knows the field.** `get` returns `V2`, the allow-list drops it, and `update` omits it. Omitted probably means "service default" (V1) on a full-replace API, but that is **unverified**. If so, every backend deploy reverts the runtime to V1 and the next `platform.yml` sets it back. The runtime flaps, and both arms of the A/B are contaminated without anything failing. +- **CLI too old.** `get` never parses the field, and the result is the same. + +This is the same class of problem as the image-tag-in-SSM convention in CLAUDE.md: a field CFN owns that the out-of-band deploy must not revert. + +**Fix, before the flag goes on anywhere:** +1. Add `platformVersion` to `ALLOWED`. +2. Assert after the update that `get-agent-runtime`'s `platformVersion` equals its pre-update value, and fail the job if it doesn't. +3. Fail fast if the runner's CLI doesn't know the field, for example with `aws bedrock-agentcore-control update-agent-runtime help | grep -q platformVersion`. Silently dropping it is the failure mode. + +`ubuntu-24.04` runner images ship a recent CLI, but "recent" is exactly what we'd be trusting without checking. + +### B2. Snapshot-restore clones process-start state (hypothesis; verify in dev) + +Under V2, anything computed at import or lifespan time is computed **once per snapshot** and then restored into every session's microVM, possibly hours later. Specific cases in our tree: + +1. **Idle-reaper origin: `apis/inference_api/runtime_health.py`.** The module-level `tracker = RuntimeActivityTracker()` stamps `_status_since = time.time()` at import. That is deliberate on V1: a microVM that boots and never gets a turn is reaped 900 s after boot. + - On a restored V2 instance the stamp is the **snapshot time**. Until the first request enters the middleware, `/ping` reports `Healthy` with a `time_of_last_update` that can be far more than 900 s old. That makes the new microVM immediately eligible for reaping. + - Whether that bites depends on whether the platform polls `/ping` (and acts on it) before routing the first invocation. That is unknown. + - Symptom to look for: 424s or doubled cold starts on first turns. + - Candidate fix: detect a wall-clock discontinuity (the gap between the last-seen poll and `now` is far larger than the ~2 s poll interval) and re-stamp. Or origin the idle clock at the first `/ping` rather than at import. Both keep V1 behaviour unchanged. +2. **Warm-up (`apis/inference_api/warmup.py`) could become free, or wasted.** + - Today it runs on a daemon thread so `/ping` answers immediately. It covers the 3.6 s residual first-turn cost from `load-test-assessment-2026-09.md` §1. + - If V2 snapshots after `/ping` goes healthy and **before** warm-up finishes, the work isn't in the snapshot, and each restore redoes the rest. + - If the snapshot waits for readiness, holding `/ping` until warm-up completes would bake the imports and botocore model parses into the snapshot. That is paid once per deploy instead of once per session, the reverse of the V1 trade-off. + - Measure before choosing. See §5. +3. **Cloned sockets and credentials.** Warm-up builds boto3 clients but makes no calls, so no pooled TCP connections should be in the snapshot today. + - Keep it that way: anything that opens a connection at init restores a dead socket into every session. That is the stale-connection class behind #1338 (the LTM retrieval fix). + - Add a comment in `warmup.py` so nobody "optimises" it into a real call. +4. **Cloned randomness and IDs.** Python's `random` module state and any ID generated at init are identical across every restored instance. + - `uuid4`, `secrets` and `os.urandom` read the kernel RNG, which is fine only if the platform reseeds on restore (unverified). + - Worth a grep of the inference-api import path for init-time `random` and ID generation. Today it only turns up in app-api modules, which don't run here. + - Check the **OTEL resource**: if ADOT (`opentelemetry-instrument` in `Dockerfile.inference-api`) derives `service.instance.id` at SDK init, every session reports the same instance. +5. **Monotonic clocks across a restore.** Any TTL cache keyed on `time.monotonic()` that is populated at init could be read as fresh after restore. That includes the 60 s catalog cache, `oauth_token_cache` and the agent cache. All of them are empty at init today, so this is only a rule to keep: don't pre-populate time-bounded caches at startup. + +### B3. The plan's latency metric can't see what V2 improves (verified) + +The review proposed comparing the turn-latency EMF (#1184). `turn_timing.py` states that its clock starts at handler entry, so it "excludes the app-api hop and any Runtime cold start". The ~1.5 s cold routing gap and V2's snapshot restore both happen before that clock starts. + +- `PreludeTotalMs` will only move if B2.2 changes where warm-up lands. +- The cold-start comparison has to come from the client side: `tests/load`, or the SPA-observed send→first-delta gap. +- Our image is ~183 MB compressed in ECR. The blog doesn't say whether its curve is by compressed or uncompressed size, so read our V1 baseline off our own measurement (cold 6.7 s vs warm 3.75 s prelude), not off the blog's chart. + +## 4. Cost model + +The billing basis inverts, so conclusions from V1 don't carry over: + +- **Idle sessions get cheap.** On V1 an idle microVM bills its peak for the full 900 s `idleRuntimeSessionTimeout`. On V2 that memory is reclaimed. + - The trade-off behind `runtime_health.py` shifts: it reaps idle VMs aggressively because idle time was expensive. + - With ~2 s restores, a *shorter* idle timeout is cheaper to live with. + - With reclaimed idle memory, a *longer* one costs little and saves restores. + - Leave the timeout at 900 s for the A/B and decide from data. +- **In-process caches now carry a latency cost.** The agent cache, catalog cache and MCP tool listings were free on V1 as long as they sat under the peak. + - On V2, a cache entry that goes cold is reclaimed and **paged back in on its next hit**. That is a TTFT cost on exactly the path the cache exists to speed up. + - The reclaim window isn't published. Watch agent-cache-hit turns for a latency regression that looks like a miss. +- **Unknown rate.** "Higher rate × fewer GB-hours" nets out unknown until AWS publishes the V2 price. The Cost Explorer sync (#1235) is the measurement. + - Check first whether V2 bills under a distinct usage type. If it doesn't, a dev week can only be compared as a before/after, not side by side. +- **W5 instance-based SKUs.** Hold that arithmetic until the committed-baseline pricing lands. That is the V2-native answer to the same question, and V1 crossover math would compare against the wrong basis. + +## 5. Plan + +1. **Deploy script PR (B1). Lands first, and is harmless on V1.** Add `platformVersion` to `ALLOWED`, the post-update equality assertion, and the CLI capability check. +2. **Infra PR (flag).** + - Add `CDK_AGENTCORE_RUNTIME_V2_ENABLED` → `config.inferenceApi.runtimeV2Enabled`. It is in-development, so only `"true"` enables it and `""` means off. + - **Always** set the property explicitly: `addPropertyOverride('PlatformVersion', enabled ? 'V2' : 'V1')`. If we omit it when the flag is off, whether removing the property reverts the runtime is up to CFN. An explicit `V1` makes rollback a deterministic in-place update. + - Add a synth test for both values, and forward the variable in `platform.yml`. + - This is infra-only, so there is no backend `feature_flags.py` or SPA flag. + - A branch `feature/agentcore-runtime-v2-flag` already exists in another worktree at the `develop` tip with no commits. Reuse it or delete it; don't fork a second one. +3. **Runtime-health hardening (B2.1).** Make the idle-clock origin restore-safe, with a unit test that simulates a wall-clock jump. Also harmless on V1. +4. **Dev A/B.** Turn on `CDK_AGENTCORE_RUNTIME_V2_ENABLED=true` in the `development` environment. + - Verify with `get-agent-runtime` after `platform.yml`, **and again after the next `backend.yml`**. The second check is B1's real test. + - Watch for 424s on first turns (B2.1). + - Measure the client-side cold first-token gap with `tests/load` (B3), `PreludeTotalMs` split by cold vs warm, agent-cache-hit turn latency (§4), and Runtime GB-hours from the Cost Explorer sync. +5. **Decide warm-up placement (B2.2).** Only after step 4 shows whether warm-up work lands in the snapshot. +6. **Prod.** Only after a clean dev week, and with the V2 rate known. + +## 5a. Phase 2: prewarm the session when the user engages (after the dev A/B) + +The blog's tip is to start the session as soon as the user engages, for example when they open a chat or begin typing, instead of waiting for submit. That hides the start time behind the time they spend typing. **V2 does not do this for us.** A microVM starts only when an invocation arrives with a runtime session ID. We pin that ID per conversation (`runtime_session_id_for` in `apis/shared/harness/runner.py`, a hash of the conversation's session ID). So today **every new conversation's first turn is a cold start**, and nothing happens before the user sends. + +**Why it waits for V2.** A prewarm for a chat the user never sends leaves a microVM idle for `idleRuntimeSessionTimeout` (900 s). +- On V1 that idle time bills at peak memory. Prewarming every composer focus would be a real cost. +- On V2, idle memory is reclaimed. Build this only once the A/B (step 4) shows what an idle V2 session actually costs. + +**Most of the pieces already exist (verified in our tree):** +- **The ID exists before the first send.** The SPA already mints the conversation ID client-side for file attachments before the first message (`stagedSessionId` in `session/session.page.ts`, `onFileAttached`). Prewarming would stage the same ID when the user shows intent. +- **The warm call lands on the right microVM.** The app-api proxy already reads `session_id` from the body and applies the affinity header (`apis/app_api/chat/proxy_routes.py`). A warm call that carries the staged ID restores exactly the microVM the first real turn will use. + +**What to build:** +1. **A no-op warm action on `/invocations`.** It cannot be a new inference-api route, because the runtime only proxies `/invocations` and `/ping` (see the Inference API boundary in CLAUDE.md). The action must: + - **not** take the single-flight lease, or the real first send collides with it (409, or the proxy's 424-while-lease-held path); + - create no session or metadata rows; + - charge no quota; + - make no model call; + - pass through `InvocationActivityMiddleware`, so the idle clock starts from the warm call. +2. **An app-api endpoint** (cookie auth, `get_current_user_from_session`) that forwards the warm action with the affinity header. + - Rate-limit it per user, and warm each staged ID at most once. + - Fire it on **intent**, meaning composer focus or the first keystroke. Never fire it on page load, since that would pay for every tab left open. +3. **SPA:** + - Stage the ID and fire the warm call on intent. + - Make it fire-and-forget: a failure or a slow warm call must never delay or block the send. + - A send that arrives mid-warm simply lands on the same microVM while it restores, which is no worse than today. +4. **Optional:** warm on opening an existing conversation whose last activity is older than the idle timeout. That microVM has been reaped, so its next turn is also cold. + +**What it hides.** On today's cold path it hides: +- the Runtime start: ~2 s on V2 per AWS, before our handler is entered; +- the in-container import warm-up (`warmup.py`, the 3.6 s residual in `load-test-assessment-2026-09.md`); +- part of the 6.7 s vs 3.75 s cold/warm prelude gap. + +A later step could also pre-build the agent for the currently selected model and tools, about 1.5 s on an agent-cache miss. Keep that separate: the selections can change before send, and a wasted build costs CPU on a microVM the user may never use. + +**TTFT.** This adds nothing to the send path. It moves work that is already there to before the send. Measure it client-side, from first keystroke to first delta on turn 1 (B3), not with `PreludeTotalMs`. + +**Flag.** In development, so it defaults OFF, with a backend flag (`SESSION_PREWARM_ENABLED`) and a matching SPA flag (`features.sessionPrewarm`), per the Feature Flags section of CLAUDE.md. The backend is the gate: while the flag is off, the endpoint 404s and the SPA never calls it. + +**Interaction with B2.1.** A prewarm puts a real request right after restore, which narrows the stale-idle-clock window. It does not replace the fix, because unwarmed paths (warming off, or a failed warm call) still restore cold. + +## 6. Open questions for AWS or the docs + +- When is the snapshot taken: after the first healthy `/ping`, or after some quiescence? Does restore reseed the RNG and step the monotonic clock? +- Does a full-replace `UpdateAgentRuntime` that omits `platformVersion` reset it to V1, or leave it unchanged? +- Does V2 bill under a distinct Cost Explorer usage type, and at what rate? +- What is the reclaim window for cold memory, and what does a page-in cost? +- What does an idle V2 session cost for 900 s after a single no-op invocation? This decides whether §5a pays for itself. diff --git a/docs/specs/assistant-kb-sync.md b/docs/specs/assistant-kb-sync.md index 80c9c0574..d84034393 100644 --- a/docs/specs/assistant-kb-sync.md +++ b/docs/specs/assistant-kb-sync.md @@ -192,6 +192,18 @@ boundary posture as today's ingestion Lambda). tail chunks — the one real gap in "reuse the pipeline as-is." Implemented as: worker stashes `previous_chunk_count` on the document record; the ingestion handler's completion step deletes the tail if present.) + **Managed knowledge bases** take the same S3 event to the managed + ingestion consumer instead, where the overwrite's event is + indistinguishable from a redelivery of the original upload's: the + document is already `complete` with its bytes settled. So the worker + also writes `stagedContentHash` before staging, and the consumer + re-ingests while it differs from the `ingestedContentHash` the last + re-ingest recorded — reserving the size growth against the byte cap + *before* submitting (a file that grew past the cap is refused and its + previous version keeps serving; the next sync retries), then settling + the difference. The same marker covers a changed page in §6.2. See + `kb_migration/ingestion_consumer.py` ("A changed source is not a + redelivery"). 6. Update `source_etag`, `content_hash`, `last_synced_at`, `last_result`. ### 6.2 Web crawl (`web_crawl`) diff --git a/docs/specs/conversation-rewind-and-fork.md b/docs/specs/conversation-rewind-and-fork.md new file mode 100644 index 000000000..370df7af6 --- /dev/null +++ b/docs/specs/conversation-rewind-and-fork.md @@ -0,0 +1,314 @@ +# Conversation rewind and fork + +**Status:** DRAFT spec, written 2026-10-03 against `develop` @ `ac4212a7`. Nothing built yet. +**Tracking issue:** #1421 +**Flags (in development, default OFF):** `SESSION_REWIND_ENABLED` / `CDK_SESSION_REWIND_ENABLED` / +`features.sessionRewind`, and `SESSION_FORK_ENABLED` / `CDK_SESSION_FORK_ENABLED` / `features.sessionFork`. +Two flags because the two halves ship separately (§8). + +## 1. What the user gets + +Two actions on every **user** message bubble, next to Copy, matching Claude Code's desktop app: + +- **Rewind to here.** "Removes this message and everything after it." The conversation is cut just + before the message, and the message's text goes back into the composer so it can be edited and + resent. The session keeps its id, URL, title and sidebar position. +- **Fork from here.** "Starts a new session from this point. This one stays as it is." A new session + is created holding everything *before* the message. The SPA navigates to it with the message's + text in the composer. The original is untouched. + +Both cut at the same point: just before a user message. A fork is a rewind into a new session. That +symmetry is deliberate. It gives both actions one set of rules about history, and the fork's first +turn is cache-identical to the rewind's (§3). + +Out of scope for v1: rewinding to a point the compaction checkpoint has already summarized (§5.3), +undoing a rewind after something new has been sent (§4.4), forking from an assistant message, +voice sessions, and steering bubbles (a mid-turn steer is a text block on a tool-result message, +not a cut point). + +## 2. The question this spec answers first: do rewinds and forks reuse the prompt cache? + +**Yes, within limits that are set by Bedrock's cache rules and by how history is restored.** + +Bedrock's prompt cache is not session-scoped. It is an exact-prefix match within one account, +region and model. A rewind or fork reads from cache whenever it sends the same prefix bytes the +original conversation sent, and the cache entry for that prefix is still alive. + +### 2.1 What is cached on each request + +Three cache points (`agents/main_agent/core/model_config.py:457`), four with a bound Memory Space: + +| Point | Covers | TTL | +|---|---|---| +| `toolConfig` tail | tool specs | 5 min, or 1 h with `AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h` | +| system tail | platform floor + instructions + date (day-stable, `utils/timezone.py`) | same as above | +| last user message (`strategy="auto"`) | the conversation history | 5 min, always | + +The message-level point only matches entries within Bedrock's ~20-content-block lookback of the +new breakpoint (see the fan-out note in `model_config.py`). + +### 2.2 Which history entries exist, and when they expire + +Each model call writes one history entry: the prefix ending at that call's last user-role message +(the user text or the latest tool-result message). The **first** call of turn *k+1* reads, via +lookback, the entry the **last** call of turn *k* wrote, which refreshes it. Nothing touches that +entry again, because later turns read longer prefixes. + +So: **a cut before user message *k* reads its history from cache if turn *k* — the turn being +replaced — started less than 5 minutes ago.** The new request is +`history[0 .. k-1] + new message`. The newest live entry ends at turn *k-1*'s last user-role +message, and the only blocks not covered are turn *k-1*'s final assistant reply plus the new +message. That is a few blocks, well inside the lookback. + +| Cut | History | Tools + system | +|---|---|---| +| Before the latest user message ("edit my last message"), within 5 min | **Hit.** Writes only the last reply + the new message. | Hit | +| Further back, turn being replaced started < 5 min ago | Hit | Hit | +| Further back, older than that | Re-written at 1.25× | Hit if inside its TTL | +| At or before the compaction checkpoint | Different prefix (raw turns vs. summary), always re-written | Hit if inside its TTL | + +Illustration at the default Haiku 4.5 Regional rates in CLAUDE.md, for 60k tokens of history: a +read costs about $0.007, a re-write about $0.083. The worst case is the same cost as resuming an +idle conversation today, so rewind and fork add no new class of spend. They make the cheap case +easy to reach. + +### 2.3 Warm vs. cold restore: rewind is cache-safe, fork is best-effort + +This is the part that decides the design. + +The cache entries were written from the **live, in-memory** message list. A rewind on a warm +container can cut that same list in place, so the bytes match exactly. A fork, or any restore on a +cold container, rebuilds history **from stored events**, and that rebuild is not byte-identical to +the live list today: + +1. **Long-term-memory context is sent but not stored.** `retrieve_customer_context` inserts a + `` block at the head of the user message + (`turn_based_session_manager.py:488`). That runs after the message was persisted, and + `tests/agents/main_agent/session/test_memory_retrieval_prefetch.py:118` asserts the block is + *not* in the stored event. Every cached entry built from a user message that received LTM context + therefore contains bytes no cold restore can reproduce. +2. **Restore runs sanitizers and pairing repair; live accumulation does not.** The docstring of + `_adopt_session_conversation` (`apis/inference_api/chat/service.py:208`) records this as the + reason re-restoring is rejected there: "the same conversation can serialize differently + depending on the path that produced it." + +Consequences: + +- **Rewind on a warm agent:** history hit, per §2.2. +- **Rewind on a cold container:** history miss if (1) or (2) applies. Any cold restore misses the + same way today, so this is not a regression. +- **Fork:** always a cold rebuild. Tools and system prompt hit. **History hits only when no kept + user message received LTM context and the sanitizers changed nothing.** With LTM retrieval on in + prod, expect a history re-write on most forks of conversations past their first few turns. + +Making fork history byte-stable means fixing (1) for every cold restore, not only forks. One option +is to persist the injected LTM text as a sidecar keyed by message index and re-insert it at restore. +That is a change to the turn path with a TTFT cost and is a decision in its own right (§9, Q1). v1 +ships forks without it and **measures** the hit rate (§6) before anyone pays to improve it. + +## 3. Invariants + +1. **A retained message's bytes never change.** Cutting removes a suffix. Nothing before the cut + point is re-serialized, re-truncated or re-ordered. +2. **Message indices are never reused.** Message ids are positional (`msg-{sessionId}-{index}`, + `apis/shared/sessions/messages.py:212`), and cost rows, feedback, `displayText`, tool summaries, + UI resources and artifact provenance (`producedByMessageIndex`) all key on that index. A rewind + that let a new message take a rewound message's index would attach the old message's feedback + and metadata to the new one. +3. **Spend is never un-spent.** Cost rows of rewound turns stay, and keep counting toward quota + and the session's lifetime `totalCost`. A fork does not inherit the parent's cost; its own calls + are its own. +4. **Nothing is added to the turn path.** Rewind state rides the session row the restore already + reads for compaction state; filtering is in-memory. Fork's copying runs in app-api, off the turn + path. (TTFT budget, CLAUDE.md.) +5. **Per-session state is re-read per turn, never cached on an agent instance.** A rewind made + while an agent is cached must reach that agent on the next turn (CLAUDE.md, #741, #751). + +## 4. Rewind + +### 4.1 Storage: tombstoned spans, not deletion + +AgentCore Memory events are append-only per session. A rewind does **not** delete events. It adds a +span to a new `rewind` attribute on the session's metadata row (static SK, the same item that holds +`compactionState`): + +``` +rewind: { + epoch: 3, # bumped on every rewind + spans: [[6, 10], [12, 14]], # half-open [start, end) in PHYSICAL event order + updatedAt: "…" +} +``` + +**Physical** index = position in `list_messages` order, the same order `GET /messages` and the +per-turn message-index base (`_start_history_count`, `stream_coordinator.py:3248`) already use. +**Logical** index = position after removing spanned events, which is what the agent sees. + +A new span always runs from the cut point to the current physical end. That keeps every logical +index *below the cut* unchanged across any number of rewinds, which is why compaction state, kept in +logical indices, stays valid (§5). + +Why not delete: deletion makes indices reusable (breaks invariant 2), gives up undo, and still has to +clean feedback/metadata rows by index. Why not fork-and-replace: the session id changes, which +breaks links, project task crumbs and anything else holding the id. It also pays for a full copy on +every edit, and inherits fork's cold-restore cache problem (§2.3) on the most common action. + +### 4.2 The endpoint + +`POST /sessions/{session_id}/rewind` in app-api (`get_current_user_from_session`), body +`{ messageId }`. The message must be a user text message (a valid cutoff, +`_find_valid_cutoff_indices`, `turn_based_session_manager.py:2037`) and visible. + +1. Take the session's single-flight lease, or answer 409 while a turn is streaming. +2. Append the span, merge overlaps, bump `epoch`, and apply the compaction rules (§5) — one + conditional `UpdateItem`. +3. Clear any `PendingInterrupt` breadcrumb and paused-turn snapshot whose turn falls inside the + span. The paused turn no longer exists. +4. Return `{ nextMessageIndex, prefillText, rewoundMessageIds }`. `nextMessageIndex` is the physical + event count; `prefillText` is the message's `displayText` (not raw content, which may carry the + `[Attached files: …]` marker). + +### 4.3 Restore and the warm agent + +- **Cold restore:** right after `super().initialize()` loads the events, before compaction is + applied, drop spanned events. The cut is always just before a user message, so the retained tail + ends on a complete assistant message and pairing repair has nothing new to fix. *Verify:* the + SDK's repair runs inside `super().initialize()` on the unfiltered list. It must not insert or + remove messages in a way that shifts physical positions. If it does, filter at the load call + instead. +- **Warm agent (the important path):** the per-turn re-read that adopts compaction state + (`_adopt_persisted_compaction_state`, `:1029`) also reads `rewind.epoch`. If it moved, cut the + live list **in place** (`messages[:] = messages[:k - live_offset]`). Slice assignment keeps the + `_adopt_session_conversation` alias, so every cached agent for the session sees the cut, and it + re-serializes nothing — invariant 1, and the reason a warm rewind is a cache hit. That function's + length guard ("a live instance trails what was restored") must treat a shorter list after an + epoch change as expected. +- *Verify:* whether Strands' own `message_id` counter (used on append) is read back for ordering + or de-duplication. With spans, newly appended events get message ids that collide with spanned + ones. That is harmless if the id is write-only. + +### 4.4 Undo + +Removing a span is safe only while nothing has been appended after it. Logical indices of later +messages would otherwise shift under the compaction state. v1 offers an **Undo** toast until the +next send; the endpoint refuses the undo (409) once the physical count has grown past the span. + +## 5. Compaction state under a cut + +Compaction indices (`checkpoint`, `truncation_anchor`, `pending_checkpoint`) are absolute indices +into the loaded list, which after §4.3 is the logical list. For a cut at logical index *k*: + +1. **`truncation_anchor = min(anchor, k)`.** Required, not cosmetic: truncation applies strictly + below the anchor (`_apply_compaction`, `:820`). An anchor left above *k* would truncate the + *new* messages that take indices *k…* on a later restore, which breaks invariant 1. Lowering it to + *k* changes nothing below *k*: those messages were already below the old anchor. +2. **Clear `pending_*`.** A parked cut computed over the rewound history is stale. Clearing costs + nothing; the policy recomputes it after the next turn. +3. **Cut at or before `checkpoint` — refused in v1.** The summary covers messages up to the + checkpoint, and the prior checkpoint is not stored, so there is nothing correct to roll back to. + Resetting to raw history could exceed the context window before the post-turn policy runs. The + SPA hides Rewind and Fork on summarized messages (the compaction badge already knows where the + checkpoint is) and the endpoint answers 409. Phase 2 would keep a bounded checkpoint history, + bearing in mind the 400 KB item limit. +4. `updatedAt`, `armed`, `policy`, `lastPrefixKey` are untouched. + +## 6. Observability: make the cache claim provable + +Without a marker, a rewound turn will be **misclassified as waste**. `classify_cache_status` +(`apis/shared/observability/prompt_cache.py:178`) compares against the session's previous call. After +a rewind past the history TTL, the call reads tools + system and re-writes history, so +write > 3 × read (`PARTIAL_MISS_WRITE_READ_RATIO`) inside the TTL of the *previous* call, which is +classified as `partial_miss` and priced into `wastedUsd`. That re-write is unavoidable. + +- Stamp `rewound: true` (and the `epoch`) on the first `C#` row after a rewind, and `forkedFrom: + ` on a fork's first row. Treat them like `agentSwitched`: counted, and exposed + as `rewindMissCount` / `rewindUsd` (and `fork…`), but subtracted from unexplained waste. +- These two fields are also how the §2.3 claims get checked on dev and prod: hit rate on the first + post-rewind call split by warm/cold, and on forks split by whether LTM context was present. + +## 7. Fork + +### 7.1 Storage: copy, don't point + +`POST /sessions/{session_id}/fork`, body `{ messageId }`. It creates a new session and **copies** +the visible events before the cut, rather than storing a pointer to the parent. + +- A pointer would add a second `list_events` to every restore of the child (TTFT), make + `GET /messages` and the message-index base two-session aware, and break when the parent is + deleted: session delete hard-deletes its events (`session_service.py:397`). +- Copying costs one `CreateEvent` per event, once, in app-api. Run them concurrently with each + event's **original `eventTimestamp`**, so list order and `count_created_before` are unchanged. + The child's physical indices are then contiguous `0..k-1`: no spans, and logical equals physical. + +What gets copied: + +| Item | Copy? | Note | +|---|---|---| +| Message events before the cut | Yes | Payload bytes verbatim | +| Strands SESSION / AGENT state events | **Yes** | Without them the SDK takes its new-session branch and loads no history (`initialize` docstring, `:664`). Rewrite the embedded session id | +| `compactionState` | Yes | Verbatim, then apply §5 at *k*. A fresh state would mean checkpoint 0 (full history instead of summary) and anchor 0 (untruncated tool output): a guaranteed history miss | +| Agent / project / Memory Space binding, model | Yes | A different config is a different prefix, and a different agent | +| `displayText`, tool summaries (`TSUM#`), UI resources for indices < *k* | Yes | Display parity on reload | +| Session files (S3 + metadata) | Yes, server-side `CopyObject` | `document_read` resolves files per session, and the parent's delete would remove shared objects. Counts toward the user's file quota | +| Cost rows, feedback, pending interrupts | No | Invariant 3; feedback belongs to the parent's messages; a paused turn cannot be forked | +| Title | `" (fork)"` | No Nova call | + +Plus two child-only fields: `forkedFrom: { sessionId, messageIndex }` (sidebar "Forked from …" +link, §6 marker), and `rootSessionId`. Document offload buckets on `session_id` +(`document_offload.py:115`). At the default 100% rollout that is a no-op, but under a partial rollout +a fork could change arms and age documents differently. Bucket on `rootSessionId` instead. + +### 7.2 Long-term memory side effect (open) + +Copied events are new events to AgentCore Memory, so the semantic, preference and summary +strategies will extract from them again. Summary records are session-scoped, so the child getting +its own is correct. Semantic and preference records are actor-scoped and consolidated, so a +re-extraction probably merges into existing records, but that is unverified. **Measure on dev before +the fork flag turns on** (§9, Q2): count the actor's records before and after forking a long +conversation, and price the extraction against the AgentCore pricing page. + +## 8. Delivery + +| PR | Scope | Flag | +|---|---|---| +| 1 | Backend rewind: `rewind` attribute, endpoint, cold filter, warm in-place cut, §5 rules, `GET /messages` filtering + `nextMessageIndex`, §6 markers | `SESSION_REWIND_ENABLED` | +| 2 | SPA rewind: hover action on user bubbles, composer prefill (route-keyed draft), Undo toast, hide on summarized messages. **Message ids:** the SPA derives the next id from `currentMessages.length` (`message-map.service.ts:213`, `stream-parser.service.ts:825`); switch both to the server's `nextMessageIndex` | `features.sessionRewind` | +| 3 | Backend fork: copy (§7.1), `forkedFrom` / `rootSessionId`, offload bucketing | `SESSION_FORK_ENABLED` | +| 4 | SPA fork: action, "Forking…" state, navigate, sidebar provenance | `features.sessionFork` | + +Rewind ships first: it is the more common action and the one with the strong cache story. + +### 8.1 Tests that carry the invariants + +- **Byte equality (the key test), rewind:** run a conversation through the live path, capturing the + exact `messages` sent on turn *k*'s first call. Rewind to *k*, restore warm and cold, and assert + `fingerprint_canonical_json(history[:k])` equals the captured prefix minus message *k*. The cold + case runs with LTM retrieval returning nothing; a second cold case with LTM context asserts the + known mismatch, so a later fix (§9 Q1) flips it deliberately. +- **Byte equality, fork:** the same, against the child's cold restore. +- **Anchor clamp:** anchor above *k*, rewind, append two turns, restore twice; the new turns are + never truncated and the two restores are identical. +- **Index uniqueness:** rewind, send, and assert the new message's id, cost row and feedback key + differ from every rewound message's. +- **Alias:** two cached agents for one session (`@`-mention), rewind, next turn on either sees the cut + (extends `test_second_cache_key_for_a_session_shares_the_conversation`). +- **Smoke matrix (`scripts/smoke_turns.py`):** rows for rewind-last within TTL (expect `hit`), + rewind-older after TTL (expect `rewound` and no `wastedUsd`), and fork (expect `forkedFrom`; record, + don't assert, the history outcome until Q1 is settled). Add rewind/fork to + `docs/testing/smoke-regression.md`. + +## 9. Open questions + +1. **Persist LTM context so cold restores are byte-stable?** This would make forks and cold rewinds + hit, and it also fixes ordinary cold restores inside the TTL. It needs a sidecar read on the + restore path, which has a TTFT cost and must be measured. Decide after §6 shows how often forks + actually miss for this reason. +2. **LTM re-extraction on fork copies** (§7.2): duplicate records? Cost per fork? +3. **CreateEvent throughput** for copying a long conversation (a few hundred events) inside an + interactive request: AgentCore TPS limits and wall time. If it is too slow, the fork becomes + async with a "Preparing fork…" state. +4. **Rewound artifacts and files.** v1 keeps them: they exist, and the user may want them. Revisit if + users expect rewind to remove them. +5. **Shared Projects sessions.** v1 is owner-only, like every session route. Whether a member may fork + a project thread they can see is a Projects decision (§9.6 of that spec governs degraded + harnesses, and a fork inherits the binding). diff --git a/docs/specs/turn-latency-preamble.md b/docs/specs/turn-latency-preamble.md index aac0de11e..56f578524 100644 --- a/docs/specs/turn-latency-preamble.md +++ b/docs/specs/turn-latency-preamble.md @@ -1,5 +1,7 @@ # Turn latency: inside the preamble +**Continued by:** `docs/specs/turn-path-ttft.md` (2026-09-29), which maps the whole path from send to first token, assesses PR #1377, and orders what is left — including this spec's unstarted PR-5. + **Status:** COMPLETE for the warm path. Five PRs shipped, merged, deployed and validated on dev (#1184, #1191, #1193, #1198, #1201): @@ -732,7 +734,7 @@ Noted there too: contextvars do not cross into a `ThreadPoolExecutor`, and the MCP load path crosses one, so a future caller marking from inside it would silently record nothing. -## PR-5 — split `agent_build.tools` (NOT STARTED — the one thing worth doing next) +## PR-5 — split `agent_build.tools` (SHIPPED #1396 — 62ms on dev with P2 in; see `turn-path-ttft.md` P1a) 2039ms, 62% of the cold build, and undifferentiated. Split it into the MCP pre-flight, gateway integration, and local tool assembly, exactly as PR-1 split @@ -746,7 +748,104 @@ synchronous SDK handshake, TLS setup, or tool-registry work nobody has looked at. Second target after that: `agent_build.session_mgr` at 830ms (AgentCore Memory -restore, never timed). +restore, never timed). Now PR-6. + +## PR-6 — one shared boto3 session for the agent build, built at warm-up (SHIPPED) + +The second target named above, `agent_build.session_mgr`, measured **616-738ms** +on dev first turns (2026-09-28) — more than half the 1.0-1.2s build. + +**What it does.** The AgentCore SDK's `AgentCoreMemorySessionManager.__init__` +builds a `MemoryClient` (a fresh `boto3.Session` plus two clients), then a second +fresh session plus two more clients that **replace** the first pair, then calls +`read_session`. For a new session that is two sequential `list_events` (the +second is a legacy-format fallback) and a `create_event`, each on a cold +connection pool. A fresh session re-parses every service model it touches, and +the parse is the cost. Two more fresh sessions sat next to it on the same first +turn: `_discover_strategy_ids` built its own `MemoryClient`, and Strands' +`BedrockModel` built its own `boto3.Session`. + +**Every first turn is a fresh process.** Each conversation's turns run in their +own Runtime process (`service.instance.id` differs per session in the runtime +logs). So a first turn always pays cold construction, and process-wide caching +helps it only if the shared work is done **before** the turn arrives. That is why +the session lives in `apis.shared.aws_clients` (`shared_boto_session`, a +`ClientReusingSession` whose `client()` returns one client per configuration) +and is built at container warm-up (`apis/inference_api/warmup.py`, on the +startup daemon thread): its `bedrock-agentcore`, `bedrock-agentcore-control` +and `bedrock-runtime` clients are constructed there, and `_discover_strategy_ids` +is called once so the strategy ids are cached before any turn. That discovery is +a control-plane read of static configuration, and it opens the one connection +warm-up otherwise avoids; `docs/specs/turn-path-ttft.md` §5 P2 accepts that +(botocore retries a connection error on this idempotent call) and names it as +the thing to watch on the Runtime V2 restore. + +**Who gets the session.** The factory hands it to the SDK session manager +(`boto_session=`) and rebinds the SDK module's `MemoryClient` so the discarded +pair is built from it too (the only seam the pinned bedrock-agentcore offers for +a one-line upstream bug; a test fails if an upgrade stops going through it); +`_discover_strategy_ids` builds its `MemoryClient` on it; and +`ModelConfig.to_bedrock_config` passes it to `BedrockModel` as `boto_session` +(and then no `region_name`, which Strands rejects alongside a session). + +**The A/B that decided it** (dev, 2026-09-30, `AGENT_BUILD_EXPERIMENT=ab` set on +the Runtime out of band, 15 first turns per arm interleaved in rotating order, +every turn a cold process, all 30 ok, arms verified server-side, sessions +soft-deleted afterwards). Median / p75 in ms: + +| Stage | control | shared session | +|---|---|---| +| `agent_build.session_mgr_clients` | 446 / 497 | 8 / 9 | +| `agent_build.session_mgr` (network) | 197 / 217 | 207 / 237 | +| `agent_build.strands_agent` | 55 / 72 | 5 / 5 | +| `agent_build.finalize` (restore) | 100 / 109 | 99 / 103 | +| `agent_build` (group) | 869 / 955 | 372 / 408 | +| prelude total | 1497 / 1655 | 1003 / 1049 | +| client: first token | 4211 / 4632 | 3564 / 3902 | + +Decision metric (`session_mgr_clients` + `session_mgr`): control ranged 601 to +813, the shared arm 171 to 289 — the distributions do not overlap, and no stage +regressed (`session_mgr`'s +10 is inside its own spread). The `strands_agent` +drop is `BedrockModel`'s fresh session going away. Warm-up did its part on the +Runtime: logs show every fresh container building the three shared clients in +15-53ms and reading the strategy ids at warm-up (median 193ms, one outlier at +3.2s, all ok). The rule written before the data — ship if the decision metric +beats control by more than the run-to-run spread with no regression — was met, +so the experiment flag and its arms were replaced by one kill switch: + +- **`AGENT_BUILD_SHARED_SESSION_ENABLED`** — default ON; only the literal + `false` disables (house style). Off, every SDK builds its own session exactly + as before and warm-up skips the shared step. No CDK entry (the Runtime is at + 48 of its 50 environment variables and unset means on); a deployment that + needs it off sets the variable on the Runtime out of band. `turn_prelude` + carries `sharedSession` (a property) and `processBuilds` (1 on a first turn). + +**Withdrawn before the A/B: the off-loop arm.** The PR as opened had a third +arm, `shared_clients_off_loop`, which ran `create_agent` under +`asyncio.to_thread` so the route could emit a first-turn title mid-build. It +came with hardening the frozen loop used to provide for free: process-wide +build serialization, locks on `ExternalMCPIntegration`'s maps and singleton, +and a consumer pin on every external MCP client handed to a build. It was +dropped, and the hardening with it, on the assessment in +`docs/specs/turn-path-ttft.md` §4: moving a synchronous build to a thread does +not make it faster (CPU-bound parts contend on the GIL, network-bound parts take +the same time), so the arm could not reduce time to first token and its A/B +would have read as noise; and the concurrency it would have bought — running +the build's independent IO at the same time — is reachable inside the +constructor itself (§5 P3a, two threads of one executor for `session_mgr` and +`tools`) without any of the hardening. A title that lands before `prepared` is +not worth shipping locks for. + +**Sub-stages that stay.** `agent_build.session_mgr_clients` (everything before +the SDK's `read_session`, i.e. client setup) precedes `agent_build.session_mgr` +(the session read/create network). `agent_build.strands_agent` (Strands' own +`Agent` construction, including MCP `load_tools`) precedes `agent_build.finalize` +(the session restore). + +**Next target after this:** the two `list_events` in `session_mgr` (~200ms) and +the restore's `ListEvents` in `finalize` (~100ms), which are now the whole of +the build's network — see `turn-path-ttft.md` §5 P3 for overlapping them with +`tools`. ## Declined / overtaken — `asyncio.to_thread` for the DynamoDB calls diff --git a/docs/specs/turn-path-ttft.md b/docs/specs/turn-path-ttft.md new file mode 100644 index 000000000..dcc064bda --- /dev/null +++ b/docs/specs/turn-path-ttft.md @@ -0,0 +1,954 @@ +# The turn path: request to first token + +**Status:** assessment and plan, written 2026-09-29 against `develop` @ `e4646277` +(PR #1378 merged) with PR #1377 (`feature/agent-build-latency`) open. +**Progress (2026-10-03):** P2 shipped (#1395), P1a (#1396) and P1b (#1397, #1398) shipped +and read on dev, P4a shipped (#1400) and read on dev; P3a parked on the P1a readout. F5 +(the per-turn history count) measured on a long conversation, then taken off the critical +path (#1403) and read on dev — see §5 P4. A KB agent's cold first turn spent 21.5s loading +two MCP servers in series; they now load concurrently (#1406, read on dev) — see §5 P3. +P3b (KB search ahead of the build) shipped (#1411) and read on dev. The agent-binding +"does this thread have messages?" check now costs one filtered `ListEvents` instead of a +full history read — see §5 P4. The managed KB search reads its record once (#1416, read on +dev): a managed follow-up's `rag_wait` is now the managed `Retrieve` itself — see §5 P3c. +**Supersedes nothing; it joins three specs that each cover one slice of this path:** +- `docs/specs/turn-latency-preamble.md` — the preamble (455ms → 22–37ms warm) and the + decomposition of `agent_build`. Its PR-5 (split `agent_build.tools`) shipped as P1a below (#1396) and was read on dev 2026-10-01. +- `docs/specs/agent-state-feedback.md` — PR-3 deferred the agent build into the stream and + narrates it (`preparing` / `prepared`). +- `docs/specs/agentcore-runtime-v2.md` — the Runtime V2 migration and the prewarm-on-intent + idea (§5a). +**Assesses:** PR #1377, *perf: A/B the first-turn agent build (shared Memory clients, +off-loop build), default off.* + +Labels: **Measured** = a number from dev or prod with a source. **Read** = read off the code +on this date. **Hypothesis** = inferred, not observed. The preamble spec records what +happens when those get confused (its "~53ms per GSI query" turned out to be boto3 client +construction on a throttled CPU), so the labels are load-bearing. + +--- + +## 1. Why this document exists + +A turn is the most complicated process in the app and nobody had one map of it. The handler +in `apis/inference_api/chat/routes.py` is ~2,200 lines in one function with two nested +generators closing over ~40 locals; the build it triggers spans `service.py`, +`base_agent.py`, `chat_agent.py`, `agent_factory.py`, `session_factory.py`, +`turn_based_session_manager.py`, `external_mcp_client.py` and two SDKs; the stream that +follows starts in `stream_coordinator.py` and hands off to Strands. Each spec above opened +one window on it. This document is the whole path, stage by stage, with what runs where, +what is network, what blocks the event loop, and which `turn_prelude` mark (if any) sees it. + +Three things fall out of drawing it: + +1. **The plan** (§5): where the time still is on a cold first turn, ordered so every change + is measured before it is trusted, and leaning toward Runtime V2. +2. **The PR #1377 assessment** (§4): it aims at the right stage, half the fix lands lazily, + and the arm that carries all the complexity buys something other than what it measures. +3. **A regression plan a cheaper model can execute** (§6): invariants that must never move, + named tests per change, and the dev recipes that settle timing claims. + +--- + +## 2. The map + +### 2A. Before the handler (no server-side clock sees any of this) + +| # | Stage | Where | What happens | IO | Cost (warm / cold) | +|---|---|---|---|---|---| +| A1 | Send | SPA | `fetch-event-source` `POST /api/chat/stream` with an `InvocationRequest` body (`session_id`, `message`, `model_id`, `enabled_tools`, `enabled_skills`, `file_upload_ids`, `rag_assistant_id`, …). The SPA mints the session id client-side. | — | — | +| A2 | Edge + BFF | CloudFront → ALB → app-api `chat_stream` (`apis/app_api/chat/proxy_routes.py`) | Cookie auth (`get_current_user_from_session`: BFF session row; user upsert throttled to 300s), CSRF. Reads `session_id` out of the body and sets the **runtime affinity header** `sha256(session_id)` so the turn lands on the conversation's own microVM. Forwards `Authorization: Bearer <id token>` and `OAuth2CallbackUrl`. `httpx` `stream=True`; headers are relayed as soon as inference-api flushes them. | 1 DynamoDB read | **Measured** ~478ms warm for A2+A3 together (agent-state-feedback PR-3) | +| A3 | Data plane | AgentCore Runtime | Routes to the microVM pinned by the affinity header. **No microVM → cold start**: V1 pulls the image and boots the container (**Measured** cold turn 6.7s vs warm 3.75s end to end on 2026-09-19; AWS's V1 curve is 5.4s at 200MB); V2 restores a snapshot (**Asserted** ~2s flat). `/ping` is polled ~every 2s. Only `POST /invocations` and `GET /ping` are proxied. | — | 0 / 1.5–5s | +| A4 | Process start (cold only) | inference-api `main.py` lifespan | Starts `warmup.py` on a daemon thread: imports `strands_tools.calculator` (sympy), the agent factory modules, and builds one client each for `bedrock-runtime`, `bedrock-agentcore`, `dynamodb`, `s3` **on boto3's default session**. `/ping` answers immediately. | none (client construction only) | **Measured** container: `warmup step=boto:bedrock-runtime ms=6` after imports | +| A5 | Middleware | inference-api | Outermost `InvocationActivityMiddleware` (marks the container busy for the whole streamed body), CORS, `AgentCoreContextMiddleware` (workload token, callback URL, session id → contextvars), `StreamSafeGZipMiddleware` (forwards SSE headers immediately; stock GZip withheld them until the first token). Then `get_current_user_trusted` verifies the bearer. | — | ~ms | + +**Two facts about A4 that matter later.** First, the warm-up parses service models into the +*default* boto3 session's loader. **Every fresh `boto3.Session()` re-parses** (**Measured** +on a laptop 2026-09-29: default-session second client 2ms; a client on a fresh session +150–170ms *every time*, for `bedrock-agentcore` and `bedrock-runtime` alike). The AgentCore +Memory SDK, `MemoryClient`, and Strands' `BedrockModel` all construct fresh sessions, so +today's warm-up does not reach them (§3 F2). Second, under V2 whatever the warm-up has +finished before the snapshot is taken is paid once per deploy, not once per session — and +whatever it has not is redone on every restore (`agentcore-runtime-v2.md` B2.2). + +### 2B. Inside `POST /invocations`, before the stream opens + +Every row here runs **before** the `StreamingResponse` returns, i.e. with no channel open +to the client. The `turn_prelude` log line and the `AgentCoreStack/TurnLatency` EMF record +carry one mark per stage; the mark column is the name to filter on. + +Legend for IO: **D** DynamoDB, **S** S3, **M** AgentCore Memory, **B** Bedrock, **C** +tool catalog (DynamoDB through a per-process 60s `config_cache`, or a per-tool 10s +freshness cache), **R** RBAC (per-process TTL cache), **sync** = a synchronous boto3 +call inside `async def`, which blocks the event loop for its round trip. + +| # | Mark | What runs | IO | Warm (**Measured** post-#1198) | +|---|---|---|---|---| +| B1 | `preamble.ownership` | `load_session_meta`: **one** `SessionLookupIndex` query answering both "is this someone else's session" (→ 404) and "the caller's META row" (threaded as `session_meta` into everything below). | D ×1 sync | 6–7ms | +| B2 | `preamble.skills` | `_apply_enabled_skills_filter(await _resolve_accessible_skill_ids(...))` **only when** `enabled_skills` is non-empty (opt-in, D6). | R + skill catalog | ~0 (6–21 with skills) | +| B3 | *(unmarked)* | Model retirement resolve (`resolve_effective_model`, 60s catalog cache). Then the two **non-turn early exits**: `app_tool_call` and `app_context_update` (MCP Apps) build/reuse the agent with `cache_write=False` and return JSON. | C | ~0 | +| B4 | `preamble.files` | `pop_pending_attachments` (snapshot; a write only if a marker exists) → `_select_recovered_attachments` → per-message file cap → `resolve_files` (**one S3 GET per upload id**) → dedupe → `_partition_attachments` (inline / tabular / pptx / oversized) → inline byte budget → marker names. | S ×N | 1ms with no attachments | +| B5 | `preamble.session_state` | `ensure_session_metadata_exists` (a conditional `put_item` on a new session; snapshot short-circuit otherwise) → `clear_paused_turn`, `clear_pending_interrupts`, `clear_truncated_turn`, `clear_interrupted_turn` (all snapshot short-circuits; a write only when a marker exists) → `set_pending_attachments` (write if uploads) → **title task spawned** on a first turn (`generate_conversation_title`, Nova Micro in `to_thread`, persists in the background). | D 0–3 | 3–4ms | +| B6 | `preamble.quota` | `check_quota(session_total_cost=<from snapshot>)`: tier (cached), user cost summary (cached), session notice from the snapshot. Early return with a conversational message if exceeded. | cached | 5ms | +| B7 | *(unmarked)* | Retired-model denial (conversational). Model access RBAC check. `_load_user_settings` (**one GetItem**, feeds default model + personal instructions). | R, D ×1 sync | ~6ms | +| B8 | `rag` | **Agent turns only** (`rag_assistant_id`, not resume). In order: `_session_has_messages` (mention turns only); `get_session_metadata` (**a second read of the META row**, outside the PR-2 snapshot); `get_assistant_with_access_check` (assistants table + shares); `resolve_invocation_agent` (version snapshot); `mark_share_as_interacted` (**write**, non-owner); `bump_last_used_at` (**conditional write every turn**, throttled to one real bump/day) + `resume_inactive_policies`; project gate (project row); `resolve_agent_invocation` (bound tools/skills/model/memory, RBAC); **knowledge-base search** (embedding + vector query; the one wait here the answer genuinely needs); prompt composition; memory hydration (`to_thread`); `get_session_metadata` (**third read**) + `store_session_metadata` (**full-row write every bound turn**). Every DynamoDB call in `assistants/service.py` constructs its own `boto3.resource("dynamodb")` (14 sites) rather than using `apis.shared.aws_clients` — the ~47ms-per-construction cost the preamble spec removed from `metadata.py` is still paid here. | D ×4–6 sync, KB | 0 on plain chat; unmeasured on agent turns | +| B9 | *(unmarked)* | Active custom prompt (`resolve_active_prompt_text`: a prompts-table read when the turn or session selects one); personal instructions composed for plain turns. | D 0–1 | ~0 | +| B10 | *(in `tools`)* | **Single-flight lease** `acquire_session_lease` (one conditional update; 409 if held) and, on a resume, `seed_steer_queue`. | D ×1 sync | ~6ms | +| B11 | `tools` | Non-resume: model fallback (`_resolve_fallback_model`: settings → catalog default → RBAC re-check), `_resolve_model_settings` (catalog cache; admin bounds/locks), attachment auto-enable (`_session_has_tabular`: a DynamoDB query unless memoized; RBAC per spreadsheet tool id), always-on union (catalog snapshot), **injected tool builders** (spreadsheet / artifact / word / workspace / excel / powerpoint / account / memory — closures, no IO), `_build_document_tools` (`_session_has_documents`: a query unless memoized), cache-key material. Resume: `get_paused_turn` + an eager `get_agent` from the snapshot. | C, R, D 0–2 | 98–102ms warm, 196–244 cold — **never decomposed** | +| B12 | `stream_setup` | Citations list, the two generators wired, `StreamingResponse` returned. Headers flush here. | — | ~1ms | + +**`agent_build` is not in this table** because with `AGENT_PREPARING_PHASE_ENABLED` +(default on) it runs *inside* the stream (C1 below). With the flag off it runs eagerly +between B11 and B12 and the `preparing`/`prepared` frames are never sent. + +### 2C. Inside the stream, before the first token + +| # | Mark / clock | What runs | IO | Cost | +|---|---|---|---|---| +| C1 | `agent_build.*` | `preparing` frame → `get_agent` → `prepared` frame (`durationMs`) → `turn_prelude` emitted → lease heartbeat task started. **The build is synchronous and runs on the event loop** (`create_agent` is called without `await`), so for its whole duration nothing else in the process runs: the title task cannot deliver, `/ping` cannot answer, and a server-side timer around it cannot fire (agent-state-feedback PR-3 learned this on dev). Sub-stages in §2D. | see 2D | **Measured** 1–49ms warm (cache hit); **~3300ms cold** | +| C2 | *(unmarked)* | `stream_with_quota_warning`: `quota_warning` / `quota_session_notice` / `agent_notice` / `citation` frames; attachment guidance; **`_build_tabular_inventory`** (when spreadsheet tools are enabled: KB file listing + session file listing — DynamoDB/S3 reads); MCP App context drain; recovery / interruption / slash-command notes; `agent.stream_async(...)`. | D/S 0–2 | ~0 unless spreadsheet tools | +| C3 | *(unmarked)* | `ChatAgent.stream_async` → `PromptBuilder.build_prompt` → **`StreamCoordinator.stream_response` head of turn**: env vars, fingerprint/cancel reset, lease stamped, `apply_pending_compaction` (**re-reads compaction state from DynamoDB every turn**, #751), `apply_document_offload`, `reset_stale_interrupt_state`, **`_get_initial_message_count` → `session_manager.list_messages` → `ListEvents` over the whole session history**, just for `len()` (§3 F5), display-text arm, MCP Apps broker subscribe. | D ×1, **M ×1 (full history)** | unmeasured; grows with history | +| C4 | coordinator `stream_start_time` | `agent.stream_async(prompt)` through `process_agent_stream` and the 100ms `_merge_agent_status` poll (which also polls the title). Strands `MessageAddedEvent` → SDK `append_message` (`CreateEvent`, `to_thread`) **and `retrieve_customer_context`**: up to two `RetrieveMemoryRecords` (preferences + facts namespaces) in a `ThreadPoolExecutor`, awaited before the model call, one attempt each with a bounded timeout (`memory_retrieval_timeout_seconds`). `BeforeInvocationEvent` / `BeforeModelCallEvent` hooks (status, attribution capture, fingerprint, ledger, census). `count_tokens` is local (`native_projection=False`). `ConverseStream` opens. | M ×1–3, B | unmeasured server-side except the model call | +| C5 | `first_token_time` | First `contentBlockDelta` with text. The persisted `time_to_first_token` field measures **C4 → C5 only** — it is a *model* TTFT and is blind to everything above it (preamble spec PR-1b). | — | — | +| C6 | — | app-api relays chunks (keepalive comments during silence); the SPA renders. | — | — | + +### 2D. Inside the build (`get_agent` → `create_agent`, a cache miss) + +| Sub-stage | What runs | IO | Cold (**Measured** 2026-09-19) | +|---|---|---|---| +| *(pre-mark)* | `get_freshness_hash(enabled_tools)`: per-tool `updated_at`, 10s TTL, **one `GetItem` per enabled tool on a cold cache**; skills freshness; cache key; hit → return. | C ×N | in `tools` (B11) | +| `prompt` | `ModelConfig.from_params`, `RetryConfig.from_env`, `SystemPromptBuilder`. | — | 160 | +| `registry` | `create_default_registry` (imports; sympy behind `calculator`), `ToolFilter`, `_register_external_mcp_tools`: a `ThreadPoolExecutor` + fresh event loop + `asyncio.run`, **one `get_tool` GetItem per enabled tool** (uncached; `repository.get_tool` is a direct `get_item`). | C ×N | 50 | +| `session_mgr` | `SessionFactory.create_session_manager`: `load_memory_config`; **`_discover_strategy_ids`** (`MemoryClient(region)` → 2 clients on a fresh session, then a control-plane `get_memory_strategies` call; `lru_cache` per process, so **once per fresh process = once per conversation**); retrieval config; `TurnBasedSessionManager(...)` → SDK ctor builds **`MemoryClient()` (fresh session + 2 clients) and then a second fresh session + 2 clients that replace the first pair**; Strands `RepositorySessionManager.__init__` → `read_session` (**1 `ListEvents`, 2 on a new session** via the legacy fallback) → `create_session` (**1 `CreateEvent`** on a new session). | 4 parses, M ×2–3 | 616–830 | +| `tools` | `filter_tools_extended`; gateway: `_expand_gateway_tool_ids` (executor + `asyncio.run`, `get_tool` per gateway id) and `get_gateway_client_if_enabled` (client object; started later by Strands); external MCP: capture contextvars, executor + `asyncio.run` → `load_external_tools`: `get_tool` per server, `create_external_mcp_client`, **`await client.load_tools()` per server, serially** = `MCPClient.start()` (a background thread with its own loop, HTTP `initialize`) + `tools/list`; OAuth pre-flight recovery; `extra_tools` appended. | C ×N, MCP ×servers | **2039** (62% of the build) with **one** server on dev — undecomposed | +| `hooks` | Ten hooks constructed (no IO). | — | 0 | +| `plugins` | Skills runtime (skill records if any), `build_tool_result_offloader`. | 0–1 | 13 | +| `finalize` (#1377 splits it into `strands_agent` + `finalize`) | `AgentFactory.create_agent`: `CountTokensBedrockModel` → Strands `BedrockModel` → **fresh `boto3.Session()` + `bedrock-runtime` client (another parse)**; `Agent(...)`: tool registry `process_tools` (MCP clients register as consumers; `load_tools` is a no-op because the pre-flight primed the cache); `AgentInitializedEvent` → `TurnBasedSessionManager.initialize` → SDK `read_agent` (**1–2 `ListEvents`**, skipped on a new session) → `list_messages` (**1 `ListEvents`, `MAX_FETCH_ALL`**) → converter → document-byte strip → sanitize → compaction (`_load_compaction_state`: **1 DynamoDB GSI read**; possibly `_retrieve_session_summaries`) → pairing repair. New session: `create_agent` `CreateEvent`. Then `_adopt_session_conversation`, snapshot stamping, cache insert. | 1 parse, M ×2–3, D ×1 | 194 | + +Sum of a measured cold build: 3286ms of a 4131ms prelude. Everything in 2D runs +**serially on one thread**, and the two big network stages — the MCP pre-flight and the +Memory session read/restore — do not depend on each other. + +--- + +## 3. What the map exposes + +**F1. The largest cold number is still a black box (Read).** `agent_build.tools` is +2039ms with one external MCP server, and the preamble spec's PR-5 (split it) is not +started. PR #1377 adds `session_mgr_clients` and `strands_agent` marks but leaves `tools` +whole. Nothing below should be built on a guess about what is inside it: it may be the MCP +`initialize` handshake against a cold Lambda, the two thread-pool hops with fresh event +loops, TLS setup, or the per-tool catalog reads. PR-5 comes first. + +**F2. The build parses the same service models four to seven times, and the warm-up +parses them into a session nobody on this path uses (Measured on a laptop, mechanism +Read).** `_discover_strategy_ids` builds a `MemoryClient` (2 parses); the SDK constructor +builds one (2 parses) and throws it away for a second pair (2 parses); `BedrockModel` +builds its own session (1 parse). Each parse is ~150ms on a laptop and client +construction on the container measured ~30× a laptop in the preamble spec. `warmup.py` +builds `bedrock-agentcore` and `bedrock-runtime` clients on the **default** session, which +none of those constructors use. PR #1377's `shared_clients` arm fixes the SDK's pair — +lazily, on the first build, so a first turn still pays two parses plus the strategy-id +call — and does not touch `_discover_strategy_ids` or `BedrockModel`. + +**F3. The build is serial where it could overlap (Read).** `session_mgr` (network: +read/create session; ~0.6–0.8s) and `tools` (network: MCP pre-flight; ~2s) are +independent, and the restore in `finalize` (network: `read_agent` + `list_messages`) is +independent of both. On an agent turn the knowledge-base search (B8) is independent of the +build too. Today each waits for the previous one. PR #1377's off-loop arm moves the whole +serial build to a worker thread; it does not overlap any of these with each other. + +**F4. Agent turns carry writes and duplicate reads on the critical path (Read).** B8 +reads the META row twice more (PR-2's snapshot covers only the preamble), writes it once +in full, does a conditional write for `lastUsedAt` on every turn, and marks shares on +every non-owner turn — all awaited before the stream opens, and each constructing its own +DynamoDB resource. None of them changes what the model is told this turn. + +**F5. Every turn fetches the whole session history a second time to count it (Read; cost Measured 2026-10-02, §5 P4).** +`StreamCoordinator._get_initial_message_count` prefers `session_manager.list_messages` +over the maintained `message_count`, "because metadata retrieval uses global indices across +all agents' messages" (voice + text). For a `TurnBasedSessionManager` that is one +`ListEvents` with `MAX_FETCH_ALL` on the head of **every** turn, cache hit or not, and its +cost grows with the conversation. It sits in the gap between `prepared` and the first +`thinking` that the agent-state-feedback epic observed and could not explain. + +**F6. The clock stops at the agent (Read).** `turn_prelude` ends at C1. C2–C4 (inventory, +head-of-turn compaction re-read, the history count, LTM retrieval, the hooks) are between +`prepared` and `thinking` and no server-side number covers them; the persisted +`time_to_first_token` starts after them. A first token that moved because of C3 would be +invisible to every dashboard we have. + +**F7. Per-tool catalog reads are paid once per build per tool, twice (Read).** +`get_freshness_hash` and `_register_external_mcp_tools` each do a `get_item` per enabled +tool (the first on a 10s TTL, the second uncached); `load_external_tools` and +`_expand_gateway_tool_ids` read again per server. The whole catalog is already in the 60s +`config_cache`. Small per read with a cached client, but it is N×4 round trips per cold +build and every one is `sync` inside `async`. + +**F8. Three sync→async bridges, each a thread and a fresh event loop (Read).** +`_register_external_mcp_tools`, `_expand_gateway_tool_ids` and the external MCP load each +spin a `ThreadPoolExecutor` to `asyncio.run` a coroutine, because the constructor is +synchronous and the loop is busy. Correct, and the reason the build cannot be `await`ed +piecemeal today. Any restructuring of the build should collapse these into one deliberate +boundary rather than three incidental ones. + +**F9. What V2 changes about all of the above (Read from the V2 spec; Hypothesis on +timing).** Under V2 the cold start (A3) drops to ~2s flat and *every* first turn is a +restore. Process-start work (A4) is paid once per snapshot. So: anything static that is +computed at start (strategy ids, parsed service models, imported modules) becomes free per +session if it finishes before the snapshot; anything that opens a connection at start is a +dead socket in every restore (§B2.3 of the V2 spec); anything time-bounded pre-populated at +start (TTL caches on a monotonic clock) is a stale read after restore. The idle-clock +origin in `runtime_health.py` is the known hazard. The prewarm-on-intent idea (§5a) is the +one lever that hides the build itself, and it is V2-only for cost reasons. + +**F10. The route handler is where the ambiguity lives (Read).** Most of the "how does a +turn work" questions above were answered by reading a 2,200-line function whose phases are +separated by comments and whose two generators close over its locals. The path is *not* +badly designed — every stage has a reason and most have a spec — but it is not legible. +§7 starts on that. + +--- + +## 4. PR #1377, assessed + +**What it does well.** It measured on dev before proposing, found the real mechanism (the +SDK's double client construction), noticed that every conversation is its own Runtime +process (so laptop warm-process numbers do not transfer), wrote the decision rule before +the data, and shipped everything **off by default** behind one flag with an interleaved, +session-stratified A/B and a script that joins client and server timings. That is exactly +the discipline the preamble spec asked for. The instrumentation (`session_mgr_clients`, +`strands_agent`, `buildArm`, `processBuilds`, the experiment script) has no cost on the +control arm and should merge regardless of the arms' fate. + +**Where it lands short, by component.** + +*Shared clients (`memory_shared_clients_enabled`).* Right target (F2), incomplete fix: + +- The shared session is created **lazily by the first build**, so the first turn of every + conversation — the only turn the A/B is about — still pays two parses plus the + strategy-id round trip. The session, its two clients, and `_discover_strategy_ids` belong + in `warmup.py`, on the same daemon thread, so a first turn finds them built. On V2 that + also puts them in the snapshot. (Build the *clients*; make no calls except the one + strategy-id read, which is static config — see P2 for the V2 caveat.) +- It leaves `_discover_strategy_ids`'s own `MemoryClient` and `BedrockModel`'s fresh + session alone. Both take a `boto3_session` / `boto_session` argument today (verified in + the pinned SDKs), so the same shared session closes them with no monkeypatching. +- The mechanism — rebinding `MemoryClient` inside the SDK module, gated by a contextvar + set for the duration of one constructor — is the only seam the SDK offers for the + throwaway client, and the PR pins it with a test that fails if an upgrade stops going + through it. Acceptable, but it is a workaround for a one-line upstream bug + (`MemoryClient(region_name=region_name, boto3_session=boto_session)`). File that + upstream; the rebinding should have an end date. + +*Off-loop build (`agent_build_off_loop_enabled`).* This arm carries all of the PR's risk +(the integration lock, the singleton lock, consumer pins, process-wide build +serialization) and its stated payoff is the title landing before `prepared`. Two things +to be clear about: + +- **Moving a synchronous build to a thread does not make it faster.** CPU-bound parts + contend on the GIL; network-bound parts take the same time. By itself the arm cannot + reduce TTFT, and the A/B's TTFT comparison will most likely read as noise. +- **What it actually buys is concurrency on the loop during the build**: the title + (measured), `/ping` (never measured — today a 3s build blocks the health poll; nothing + has failed because of it, and V2's platform may or may not be as tolerant), and — the + valuable one — the ability to start an agent turn's knowledge-base search (and anything + else independent) *before* the build and have it finish *during* it. The A/B as designed + measures the title only, and on plain chat turns, so the arm's real value is not on the + scorecard. + +**Recommendation** (applied to #1377 on 2026-09-30; kept here as the record of why)**.** + +1. Merge the instrumentation and the `shared_clients` arm now, amended so the shared + session, its clients and the strategy ids are built at warm-up (P2). Keep the arm behind + the flag until the dev A/B reads; the decision rule in the PR stands. +2. Keep the off-loop arm in the A/B, but decide it on the right question. Either extend the + experiment to agent turns with a knowledge base and start the KB search ahead of the + build (P3b, small), or accept the PR's own rule and expect to remove the arm and its + MCP hardening. Do not ship the hardening for a title. +3. Do not let #1377 be the vehicle for F1/F3. Split `tools` first (P1); the overlap that + actually shortens a cold build (P3a) is a different, smaller change inside the + constructor, and it does not need any of the off-loop machinery. + +--- + +## 5. The plan + +Ordered so every change has a measured before and after. Each item names its V2 posture +because the migration is coming and nothing here should have to be undone for it. + +### P0 — Land the measurement in #1377, run the A/B, decide `shared_clients` + +Unchanged from the PR: `AGENT_BUILD_EXPERIMENT=ab` on the dev Runtime out of band, +`experiment_agent_build_arms.py --per-arm 15 --cleanup`, the decision rule in the spec. +Amend the arm per §4 before the run, or the first-turn number will understate it. + +### P1 — Close the two measurement gaps (F1, F6) + +**Status (2026-10-01): P1a shipped (#1396) and read on dev.** Sub-stages are +`agent_build.tools.{filter,gateway,mcp,extra}`. `tools.catalog` and +`tools.mcp_preflight` are not separate stages: both happen per server inside the one +executor hop, and a mark cannot be taken there, so `tools.mcp` times the hop and the +`mcpServers` log property carries each server's `{id, outcome, catalogMs, preflightMs, +totalMs}` (`outcome` ∈ `cached`, `loaded`, `recovered`, `dropped`, `skipped`, `error`). +`tools.mcp` minus the servers' `totalMs` is the hop's own thread and event-loop startup. +`groups` now sums every dotted prefix, so `agent_build.tools` and `AgentBuildToolsMs` +survive the split. The catalog lookups that classify external MCP ids happen earlier, in +`agent_build.registry`. + +**Dev readout (2026-10-01, one cold first turn, `sharedSession: true`, `processBuilds: 1`).** +`agent_build.tools` was **62ms**, not the 2039ms §3 F1 was written against: `filter`, +`gateway` and `extra` 0, `mcp` 62 — the one server (`hello_world`) took 4ms of catalog +read and 54ms of pre-flight before being dropped on its standing 403, and the executor +hop's own startup was 3ms. Acceptance met: the sub-stages sum to the group exactly and +one owns all of it. The 2039ms most likely predates P2's shared session (a cold build +parsed service models several times over), but it was never decomposed, so that is a +Hypothesis. **Caveat:** dev's only MCP server fails fast, so a cold `initialize` + +`tools/list` against a *working* server is still unmeasured; 62ms is a floor for that +case, not a typical value. The rest of that turn, for scale: prelude 1337ms = preamble +464 (`skills` 179, `quota` 145, `files` 77) + `rag` 67 + route `tools` 220 + +`agent_build` 577 (`session_mgr` 219, `finalize` 102, `prompt` 87, `registry` 64, `tools` +62, `plugins` 31). + +*P1a. Split `agent_build.tools`* (the preamble spec's PR-5) into `tools.filter`, +`tools.catalog` (the per-tool reads), `tools.gateway`, `tools.mcp_preflight` and +`tools.extra`, with `groups.agent_build` still summing. Marks are taken on the calling +thread around each executor hop, never inside it (contextvars do not cross the pool). Add a +per-server `ms` list as a log property of the MCP stage so one slow server is +distinguishable from many. + +**Status (2026-10-01): P1b shipped (#1397) and read on dev** (readout below). The clock is the same +`TurnPrelude`, passed to `ChatAgent.stream_async(turn_clock=...)` and on to the +coordinator. Stages after `turn_prelude`: `head_of_turn.handoff` (the route's generator +wiring, the `prepared` frame, quota warnings, the prompt build), `head_of_turn.compaction` +(compaction re-read, document offload, the per-turn resets), `head_of_turn.history_count` +(the `ListEvents` of F5), `head_of_turn.rest`, then `pre_model` and `model`. The boundary +between those two is `AgentStatusHook.first_model_call_at`, a `perf_counter()` stamp at +the turn's first `BeforeModelCallEvent` (stamped whether or not narration is on), closed +on the clock with `mark_at`. `ContextAttributionHook` is registered before the status hook +and so counts as `pre_model`; census, ledger and fingerprint are after it and count as +`model`. Without a stamp the gap is reported whole, as `pre_model_and_model`. "First +token" is the first model output of any kind — text, reasoning, or a tool call's first +block — since a tool-first turn's first visible output is the tool call. + +Two departures from the text below. The numbers ride a **second line**, +`turn_first_token` (joined to `turn_prelude` by `sessionId`, carrying `firstTokenMs`, +`preludeTotalMs` and only the post-prelude stages), rather than the first line held +until a token: a held line is lost on every turn that never produces a token (an error, +a Stop, an interrupt), which are the turns most worth reading. And the line and its EMF +record (`FirstTokenMs`, `HeadOfTurnMs`, `PreModelMs`, `ModelMs`, …) are written on the +coordinator's next pass, **after** the first token has been yielded — a log write in +front of the token would be the latency being measured. + +**Dev readout (2026-10-01, Haiku 4.5, one new conversation: cold first turn A, warm second +turn B).** Client columns are a passive `fetch` wrapper in the in-app browser (send → +first `content_block` frame). The server's first-token time is the sum of the stages: +#1397 read `firstTokenMs` at emit time, one event late, and overstated it by 131ms and +182ms here — fixed in the follow-up to read it at the `model` mark. + +| | A (cold) | B (warm) | +|---|---|---| +| prelude (`turn_prelude.totalMs`) | 1263 | 229 | +| `head_of_turn` (handoff / compaction / history_count) | 157 (4 / 67 / 86) | 83 (4 / 5 / 74) | +| `pre_model` | 664 | 450 | +| `model` | 942 | 925 | +| **server first token** | **3026** | **1687** | +| client send → first content frame | 4264 | 2119 | +| hop (client − server: app-api, Runtime routing, cold start) | 1238 | 432 | + +What `pre_model` is, from the runtime log on turn B — three network calls **in series**: +the user message's `CreateEvent` (~110ms), a second `CreateEvent` the Memory SDK logs as +`Created agent: default … with event` after every message (~107ms), then the two LTM +`RetrieveMemoryRecords` (~230ms warm, ~454ms cold, the namespaces finishing ~117ms apart +on the cold turn). LTM retrieval needs only the user's text, not either write, so the +writes and the retrieval need not be serial. `pre_model` is the largest stage we own on a +warm turn (450 of 1687ms), and it is paid on **every** turn, cold or warm. + +`history_count` is 74–86ms on a two-message conversation (one `ListEvents` page). It grows +with history: ~2.25ms per stored event plus a step per 100-event page, ~390ms at 100+ +events (measured 2026-10-02, §5 P4). `model` (~930ms) is Bedrock's own time to first token on a ~13k-token +prefix and is not ours to move here. + +*P1b. Extend the clock to the first token.* Add `head_of_turn` (C3: compaction re-read, +history count, offload) and `pre_model` (C4: LTM retrieval + hooks, up to the model call) +marks, recorded by the coordinator on the same `TurnPrelude` (thread it through +`stream_async` like `turn_started_at` already is), and stamp `firstTokenMs` from handler +entry on the emitted line. `PreludeTotalMs` stays what it is; the new number is +`FirstTokenMs`. This is what makes P4/P5 provable. + +*Tests.* `test_turn_timing.py`: new marks render, group sums hold, metric names derive. +Cost: perf_counter calls only. V2: none. + +### P2 — Warm the right things, once (F2) + +**Status (2026-09-30): built in PR #1377 as narrowed** — the off-loop arm was withdrawn, the shared session lives in `apis/shared/aws_clients.py`, warm-up builds its three clients and primes the strategy ids unconditionally, and the SDK session manager, `_discover_strategy_ids` and `BedrockModel` all take it on the `shared_clients` arm. The strategy-id cache is keyed by arm so control's first turn is unchanged. What remains of P2 is P0: the dev A/B decides whether the arm ships default-on. + +- One process-wide `boto3.Session` in `apis/shared/aws_clients.py` (it already owns the + client cache); warm-up builds `bedrock-agentcore`, `bedrock-agentcore-control` and + `bedrock-runtime` clients on **it**. +- `SessionFactory` hands that session to the SDK (`boto_session=`), to + `_discover_strategy_ids` (`MemoryClient(boto3_session=)`), and `ModelConfig.to_bedrock_config` + passes it to `BedrockModel` (`boto_session=`; drop `region_name` when doing so — Strands + rejects both). The `_SharedSessionMemoryClient` rebinding from #1377 covers the SDK's + throwaway client until upstream takes the one-line fix. +- Warm-up calls `_discover_strategy_ids` once. It is a control-plane read of static + configuration; on V2 that puts the ids in the snapshot. **Caveat:** it opens a + connection. Either accept that the pool may hold one idle socket at snapshot time + (botocore retries connection errors on this idempotent call), or take the ids from the + environment instead — the memory construct could export them, but `CfnMemory` exposes no + strategy-id attributes today, so that is a CDK custom resource, not a one-liner. Start + with the warm-up call and watch the first restore's logs. +- Expected: `session_mgr` drops by the four parses it no longer does; `finalize` by one. + Laptop arithmetic says ~0.6–0.9s of a fresh process's first build; the container ratio is + unknown until measured. Warm turns: nothing (the agent is cached). + +*Tests.* Warm-up builds each client once on the shared session (patch `boto3.Session`); +the factory passes the shared session (assert on the SDK ctor kwargs); `BedrockModel` +receives `boto_session` and no `region_name`; `reset_cached_clients` also resets the +session (the moto trap). V2: **positive** — all of it is snapshot-friendly, except the one +connection, which is the thing to watch. + +### P3 — Overlap the build's independent IO (F3) — the real cold-turn lever + +**Re-sized 2026-10-01 by the P1a readout: do not build P3a as written.** It was sized as +`max(2039, 830)` instead of `2039 + 830`. Measured on dev with P2 shipped, `tools` is 62ms +and `session_mgr` 219ms, so overlapping them saves at most ~60ms, and less once the second +thread's startup is paid. It returns only if a dev turn against a *working* MCP server +shows `tools.mcp` in the hundreds of milliseconds, or a deployment carries several +servers. P3b is unaffected, and is sized by `rag` on a KB agent's first turn. The next +decision should be made on P1b's `FirstTokenMs` breakdown, not on the build. + +**That condition is now met (dev, 2026-10-02).** The first turn of a KB agent carrying two +external MCP servers (both Lambda function URLs, OIDC-forwarded) spent `turn_prelude.totalMs` +24276ms, of which `agent_build.tools.mcp` was **21543ms**: `mcpServers` = +`student_myboisestate` pre-flight 9462ms, then `class_search` 12058ms, `outcome: loaded` +for both. Nearly all of each is the first `initialize` POST (9.2s and 11.8s to its 200); +the requests after it took ~100–130ms, which reads as a Lambda cold start on an idle dev +environment (Hypothesis — prod traffic may keep them warm). `session_mgr` was 57ms, so +P3a *as written* (session manager ‖ tools) still buys nothing; the lever is its second +half — **load the per-server pre-flights concurrently**, merged in catalog order. On this +turn that is `max(9474, 12066)` instead of the sum, ~9.5s. Lambda warmth (provisioned +concurrency or a keep-warm on the servers we own) addresses the remaining ~12s and is a +separate, cost-bearing decision. `rag` on the same turn was 1576ms (P3b's size). + +**Shipped (#1406) and read on dev (2026-10-02, runtime v521, image `817330dcd50b7bbd`).** +`ExternalMCPIntegration.load_external_tools` now builds each server in its own coroutine +(`_load_server`) and, with more than one, gathers them with each pre-flight on a worker +thread (`_preflight(off_loop=True)` → `asyncio.to_thread(asyncio.run, client.load_tools())`). +A plain `gather` would not have overlapped anything: Strands' `MCPClient.load_tools()` is +a coroutine whose body calls the synchronous `start()`, which blocks on the `initialize` +handshake. `gather` returns in input order, so clients — and the tool order in +`toolConfig` — are in catalog order; a lone server keeps the inline call. Kill switch +`MCP_PARALLEL_PREFLIGHT_ENABLED` (runtime-only, default on). Same agent, new conversation +each time: + +| | `student_myboisestate` | `class_search` | sum | **`tools.mcp`** | `turn_prelude.totalMs` | +|---|---|---|---|---|---| +| before (serial), cold, 15:27 | 9474 | 12066 | 21540 | 21543 | 24276 | +| after, cold (~50 min idle), 16:18 | 2313 | 1894 | 4207 | **2316** | 4909 | +| after, warm, 16:21 | 384 | 464 | 848 | **473** | 2986 | + +`tools.mcp` now tracks the slowest server, not the sum: ~1.9s saved on the cold sample and +~375ms warm. The servers were nowhere near as cold the second time (2.3s and 1.9s against +9.5s and 12s), so the 21.5s → 2.3s drop is mostly that; the part this change owns is the +sum-to-max gap, which on the first readout's numbers would have been ~9.5s. Both builds +produced the same `toolConfigHash` (`b12c22b73956…`) and `systemPromptHash`, so tool order +did not move. The cold turn called `search_classes` three times and answered from it. +With MCP loading reduced to the slowest server, `rag` (1404–1485ms) is now the largest +stage on this agent's first turn — P3b's case. + +*P3a. Inside the constructor, no async plumbing.* In `BaseAgent.__init__`, `session_mgr` +and `tools` do not depend on each other (hooks, which take the session manager, are built +after both). Run them on two threads of one `ThreadPoolExecutor` and join — the same +mechanism the MCP load already uses, applied one level up. Tool order stays deterministic +(one thread builds one list). Marks: take `agent_build.session_mgr_ms` and +`agent_build.tools_ms` as properties from each thread's own clock and one wall-time stage +`agent_build.parallel` on the calling thread, so the sum-of-stages invariant the +unaccounted-time widget depends on still holds. Expected on the measured cold build: +`max(2039, 830)` instead of `2039 + 830`, roughly **−0.8s**. Then, if P1a shows the MCP +pre-flight is the bulk of `tools`, the per-server loads run on the pool too, merged in +catalog order (the preamble spec's warning: the *merge* must be deterministic because tool +order is the prompt-cache prefix). + +The restore (`initialize`, inside `Agent(...)`) can join the overlap later by prefetching +its three `ListEvents` on a third thread and handing the SDK the results for the one +`initialize` that follows; that is a second step, after P3a is measured. + +*P3b. On agent turns, start the knowledge-base search before the build.* The build closure +does not read `context_chunks`; only the stream does, when it composes `final_message`. +Submit the search to a thread before the deferred build starts and await its future in +`stream_with_quota_warning`. This is the overlap #1377's off-loop arm would enable "for +free"; without that arm it needs the search to be on a thread, which it can be. Keep +citations and augmentation byte-identical (order of chunks, cap) — they land in the +persisted user message, which is the cacheable prefix on every later turn. + +**Shipped (#1411) and read on dev (2026-10-02, runtime v524, image `7ec07e0050354291`).** +The search is not on a thread of its own: it schedules loop-bound work (the managed +backend's shadow read, the observation, the activity touch), so a private event loop would +cut those off. It runs as an `asyncio` task (`_search_and_augment`) started where it used +to be awaited, and `_guarded_stream` awaits it after the build and before `turn_prelude` +is emitted (new stage `rag_wait`). The classic backend's `QueryVectors` used to run +synchronously on the loop, which a cold build holds; it now goes out with the embedding +on one worker thread, on a module-cached client (the kb-sync and rag-ingestion images do +not ship `aws_clients`). Kill switch `KB_SEARCH_AHEAD_ENABLED`. Same agent and prompts as +the #1406 readout, new conversations: + +| | `rag` | `rag_wait` | `turn_prelude.totalMs` | +|---|---|---|---| +| cold, before → after | 1485 → **993** | — → 128 | 4909 → **4662** | +| warm, before → after | 1404 → **955** | — → 116 | 2986 → **2544** | + +About 330–360ms net. The stored augmented user message is byte-identical before and after +for both prompts (SHA-256 match), and both turns streamed the same 5 citations. The +residual ~120ms `rag_wait` is post-search work that runs back on the loop and so waits for +the build: the document-status filter (`_filter_chunks_by_document_status`, a DynamoDB +lookup per source document) and `resolve_context_cap`'s record read. Moving both into the +worker is a small follow-up. + +**Managed engine (dev, 2026-10-02 evening, runtime v525).** The readout above is the +classic engine. On a managed knowledge base (an agent with 5 documents, +`retrievalEngine: managed`) the search itself is far slower — `Retrieve` took ~3.6s on a +cold first turn and ~2.3s on a warm one — and #1411 hid all of it behind the build +(`rag_wait` 128 / 119ms; build 3297 / 2045ms), so the overlap is worth ~2–3.5s on a managed +agent's first turn against ~0.35s on classic. The stored augmented message was identical +across both conversations and equal to a local recompute from a direct managed search +(managed cap 8,000, 6,876 chars). **A follow-up turn exposes it:** with the agent cached the +build is ~36ms, and `rag_wait` was 786ms of a 1,254ms `turn_prelude` — the largest stage on +a managed agent's follow-up turns. Why `Retrieve` takes 1–3.6s is being investigated +separately. + +**Follow-up built (2026-10-02): the post-search reads move off the loop.** The facade now +runs the backend search and `_filter_chunks_by_document_status` together on one worker +thread with a private event loop (`_search_then_filter`; `legacy_ms` still times the +search alone, for the dual-read comparison). Everything that schedules onto the caller's +loop — the dual-read observational read, the observation, the activity touch — stays on +it. The cap is read by `asyncio.to_thread(resolve_context_cap, …)` started alongside the +search. A side effect pinned by `test_kb_dual_read.py`: with legacy awaited on a worker, +the observational managed read now *starts* before a failing legacy search returns, as it +always did against a real (thread-hopping) search; it is still cancelled and never +finishes. Expected: `rag_wait` ≈ 0 on a cold first turn (from ~120ms); on a cached-agent +follow-up turn the filter's ~60ms overlaps nothing and remains. + +*P3c. The managed KB search, decomposed (dev, 2026-10-02, runtime v525).* On a managed +agent's **follow-up** turn the build is ~36ms, so the search is no longer hidden: +`rag_wait` 786ms of a 1254ms `turn_prelude`. The Runtime's botocore spans (log group +`aws/spans`, exact in-region durations) split the ~896ms search as: + +| step | runs on | ms | +|---|---|---| +| `load_record` — KB_Record `GetItem`, fresh `boto3.resource` | loop | 51 | +| `ManagedKbBackend._locate` — **the same row again**, fresh resource | thread | 56 (+11 construct) | +| `Retrieve` | thread | **660** | +| status filter — first `GetItem` on a fresh resource, then 3 at ~3ms | loop | 56 (+9) | +| `resolve_context_cap` — **the same row a third time**, fresh resource | loop | 42 (+8) | + +`Retrieve` is the managed engine's own cost (in-region 627/660/946ms; from a laptop p50 +~500ms over a ~106ms round-trip floor; the `MANAGED` reranker is ~50–130ms of it, and is +kept — it is what separates the scores the 8,000-character cap relies on; +`numberOfResults`, idle gaps and repeated queries change nothing). The other ~230ms is +ours: one row read three times, every read on a new resource, each paying ~45–55ms for its +first request against ~4ms on a reused one. Classic turns show the same three reads +(~55/54/46ms). **The 2.3–3.6s first-turn figures above were not `Retrieve`:** the spans put +it at 627ms (cold) and 946ms (warm), ending in under a second; the coroutine then waited for +a lone MCP server's pre-flight, which runs inline on the loop (1.5–2.8s), and the log line +measured that wait. + +**Fix (built on top of the off-loop follow-up).** The facade reads the KB_Record once and +hands it across the seam (`search(..., record=)`, so the managed backend takes `awsKbId` +from it; `_search_then_filter` passes it through on its worker) and returns the cap with +the results (`search_assistant_knowledgebase_with_cap`), so the separate cap-read worker +is gone. The record read and the status lookups go through the process-cached +`aws_clients` table, on one shared connection pool. The augmented message is unchanged — +the cap comes from the same record. Expected on a follow-up turn, where the off-loop move +overlaps nothing: ~215ms off `rag_wait` (one record read instead of three, and warm +connections for the read and the filter; classic loses ~150ms). On a cold first turn the +off-loop move already took the tail to ≈0. Next lever on follow-ups: start the search right +after the access check (~170ms; today it waits on `bump_last_used_at`, model/role +resolution and the session-metadata writes that hold the loop). + +**Shipped (#1416) and read on dev (2026-10-03, runtime v528, image `cfc37a32688c`).** Same +managed agent, two new conversations of three turns each (the first-turn prompt, then two +follow-ups with the agent cached). The spans show the mechanism exactly as built, on every +follow-up: **one** KB_Record `GetItem` per search at 3.7–4.7ms, `Retrieve` starting ~6ms +after it (no `_locate` read), the status filter at 3.3–4.4ms per document, and no cap read. +Our DynamoDB time around `Retrieve` went from ~180ms to ~20ms. Raw `rag_wait` did not fall, +because `Retrieve` ran slower in this sample than in the single #1415 sample; normalised by +`Retrieve`, the follow-up `rag_wait` lost ~100ms against #1415 and ~135ms against v525: + +| follow-up turn | `Retrieve` | `rag_wait` | `rag_wait` − `Retrieve` | +|---|---|---|---| +| v525 (before #1415), n=1 | 660 | 786 | +126 | +| v526 (#1415), n=1 | 445 | 535 | +90 | +| v528 (#1416), n=4 | 539 / 585 / 539 / 645 | 626 / 568 / 532 / 633 | −7 / −17 / −7 / −12 | + +The estimate of ~215ms was against v525 and counted savings #1415 had already taken, plus +reads that overlapped the tools and build stages anyway. A managed follow-up's `rag_wait` +is now `Retrieve` itself, minus the little of it that overlaps the cached build: what is +left is the managed engine's own cost, which varies by ±100ms call to call (445–660ms +across these samples). First turns: `rag_wait` 4 / 4ms (≈0 since #1415). The stored +augmented user messages are byte-identical to the pre-change conversations: first turn +`adf20dfe881a…` (6,876 chars) and the holistic-rubric follow-up `042ecda2411c…` (6,728) +match the v525 and v526 conversations exactly, and the third turn matches across both new +conversations; every turn streamed 5 citations. The kb-migration dispatcher ran on the new +image with no import errors; the other KB Lambdas had not been invoked at readout time. + +*Tests.* P3a: a fake session factory and a fake tool loader that each sleep; assert the +build's wall time is the max not the sum, that `tools` order is unchanged across runs +(`test_prompt_cache_determinism.py` already pins order), and that a failure in either thread +surfaces as the build's exception with the other thread joined (no orphaned MCP client: +`consumer_pin` semantics from #1377 apply here too). P3b: the KB search is started before +`preparing` is emitted and its result reaches `final_message` unchanged; a failed search +still yields the unaugmented message (today's fail-open). Dev: fetch-tee harness, first +turn with an MCP tool enabled and a KB agent; compare `agent_build` group and +`FirstTokenMs` before/after. V2: neutral — threads, no sockets at start. + +### P4 — Take the bookkeeping off the critical path (F4, F5, F7) + +*P4a. Overlap the long-term-memory lookup with the message writes (shipped #1400, read +on dev 2026-10-01).* + +**Dev readout (2026-10-01, Haiku 4.5, one new conversation: cold first turn A, warm second +turn B; same recipe as the P1b readout, with #1398 reading `firstTokenMs` at the `model` +mark).** Before is the P1b readout corrected by #1398 (its stage sums). + +| | A (cold) before → after | B (warm) before → after | +|---|---|---| +| `pre_model` | 664 → **352** | 450 → **270** | +| prefetch `waitedMs` / `lookupMs` | — / 338 | — / 268 | +| `head_of_turn` (handoff / compaction / history_count) | 157 → 118 (5 / 56 / 57) | 83 → 93 (6 / 4 / 83) | +| `model` | 942 → 810 | 925 → 867 | +| **server first token** | **3026 → 2589** | **1687 → 1451** | +| client send → first content frame | 4264 → 4543 | 2119 → 1898 | + +`pre_model` fell by about the lookup it stopped waiting for. On the warm turn the lookup +finished inside the two writes (`waitedMs=0`), so what is left of `pre_model` is the +SDK's two `CreateEvent`s — the user message's (~110–170ms) and `sync_agent`'s (~100ms, +see *Not done here*). On the cold turn the lookup outlasted the writes by 141ms. The cold +client number rose on one sample while the server number fell 437ms; the difference is +the hop and the prelude (1263 → 1307), which this change does not touch. `firstTokenMs` +now equals the stage sums to within 2ms. Byte check, in AgentCore Memory `ListEvents`: +the persisted user messages carry one block and no `<user_context>`, while the cold turn +inserted three records into the live message — the persisted/live split is unchanged. + +The P1b readout found `pre_model` — the largest stage we own on a warm turn, 450 of +1687ms, and paid on every turn — to be three network calls in series, all on +`MessageAddedEvent` for the user's message: the SDK's persist callback (`append_message`'s +`CreateEvent`, ~110ms, then `sync_agent`'s `CreateEvent`, ~107ms) and then +`retrieve_customer_context` (two `RetrieveMemoryRecords`, ~230ms warm, ~454ms cold). +The lookup needs only the user's text, not either write. + +*The constraint that shapes it.* `retrieve_customer_context` prepends a `<user_context>` +block to the **live** user message, after the persist callback has written that message. +So the persisted message — what restore rebuilds history from on a rebuilt agent — has no +context block, and the live one does. That split is load-bearing for the prompt-cache +contract (restored history must be byte-stable), so running the two callbacks +concurrently is wrong: the insert could land while `append_message` is serializing the +same dict on its worker thread, and a context block would reach Memory on some turns and +not others. + +*What it does instead.* `TurnBasedSessionManager.register_hooks` registers one sync +`MessageAddedEvent` callback **before** delegating to the SDK. Strands runs a hook's +callbacks one at a time in registration order (`invoke_callbacks_async`), so it runs +first: it checks the same eligibility the lookup does (last message is a user message +whose first block is text, retrieval configured, session not cancelled), snapshots the +query string, submits `_fetch_customer_context(query)` to a small process-wide pool and +returns. The SDK's persist callback then runs exactly as before. When the SDK calls +`retrieve_customer_context` — still after the persist — it takes the prefetched future +(only if it was started for this same message object; otherwise it fetches inline, as +before), waits for it, and inserts the block. Only the network round trip moves; the +insert happens at the same point in the sequence, so the persisted bytes and the live +bytes are what they were. Expected: `pre_model` falls by about `min(writes, lookup)` — +~200ms warm, more cold. + +*Kill switch:* `MEMORY_RETRIEVAL_PREFETCH_ENABLED` (default on; only `"false"` +disables), runtime-only like `AGENT_BUILD_SHARED_SESSION_ENABLED`. Off, nothing is +prefetched and the lookup runs inline in `retrieve_customer_context`, as before. + +*Tests.* Through a real Strands `HookRegistry` with the SDK's own async-mode wiring: the +lookup starts before `append_message` returns, the message `append_message` serialized +carries no context block while the live message does, and the result is byte-identical to +the switch-off path. Plus: prefetch is skipped for assistant and tool-result messages; a +prefetch for a different message is never applied; a failed or slow prefetch leaves the +turn without context rather than failing it. *Measure:* `pre_model` on a warm and a cold +dev turn against the P1b readout, and the `memory retrieval prefetch` line's `waitedMs` +(0 means the lookup finished inside the writes). + +*Not done here:* `sync_agent` after every message is the SDK writing agent state that its +`AfterInvocationEvent` callback writes again at the end of the turn. Skipping the +per-message write would save ~107ms more, but it changes what the SDK persists if a turn +dies mid-way; it needs its own look. + +- **`_get_initial_message_count`:** use the maintained `message_count` and drop the + `ListEvents`. The global-index concern it cites is for mixed voice+text sessions; verify + with `get_messages_from_cloud` whether the index still needs to be global (voice writes + under a different agent id) and, if it does, count once at `initialize` and keep the + count maintained rather than refetching the history every turn. + + **Measured (2026-10-02, dev, image `815293d3a627d57e`).** One disposable conversation + (`exp-histcount-…`, Haiku 4.5, no tools) driven headless to 60 short plain turns, three + seconds apart, every turn warm after the first; `head_of_turn.history_count` from each + turn's `turn_first_token` line, against the events stored before that turn. A plain + turn stores two events (user + assistant), ~0.5 KB each, so 50 turns is the first + two-page read. + + | Events before the turn | Turns | `history_count` median (min–max) | share of `firstTokenMs` | share of what we own (`firstTokenMs` − `model`) | + |---|---|---|---|---| + | 0 (cold first turn) | 1 | 59 | 3% | 4% | + | 1–19 | 8 | 92 (72–112) | 8% | 17% | + | 20–39 | 10 | 120 (104–171) | 10% | 23% | + | 40–59 | 10 | 167 (151–204) | 14% | 30% | + | 60–79 | 10 | 219 (187–246) | 17% | 35% | + | 80–99 | 10 | 266 (232–323) | 19% | 40% | + | 100–122 (two pages) | 11 | **388** (363–450) | **26%** | **49%** | + + On one page it fits ~62ms + 2.25ms per event; the second page adds a round trip + (~80ms above the per-event trend). It draws level with `pre_model` (~210–300ms here, + flat) at ~60–79 events and passes it after that, which makes it the largest stage we own + on any longer warm turn — and the only one that grows. **These are the cheap events.** The read pulls every payload + (`includePayloads=True`) and converts them all for a `len()`, so bytes count too: a + read-only replay of the same call from a laptop against 40 existing dev conversations + took 150ms at 25 events / 30 KB, 300ms at 83 events / 206 KB, and 419ms at 50 events / + 2.6 MB (a document conversation). Tool turns store more events per turn than chat. + + **Recommendation: build it next.** The fix is to stop reading the history to count it — no change + to anything the model sees, so the cacheable prefix is untouched. What the count feeds + is the message index for per-message metadata (`message_index=initial_message_count` + and the `+ 2*i + 1` arithmetic in the coordinator), so the open question is still the + one above: whether that index must be global across voice and text agents. Verify it + against `get_messages_from_cloud` before choosing between the maintained count and a + count taken once at `initialize`; either removes the read from every warm turn. + *Accept if:* `head_of_turn.history_count` ≈ 0 on a 100-event conversation, and the + per-message metadata (cost, latency, citations) still lands on the right message after + reload in a text-only and a mixed voice+text session. + + **Shipped (#1403) and read on dev (2026-10-02, runtime v519, image `86e94164a788488e`).** + The count is the same global read, started on a worker at the head of the turn + (`HistoryCount`, `agents/main_agent/streaming/history_count.py`) and counting only + messages whose `created_at` precedes that instant; the displayText hook, the artifact + anchor, the end-of-turn metadata and the interruption persistence each await it where + they use it. The maintained per-instance `message_count` was rejected for the reason + above it — the voice, `@`-mention and synthetic-message writers make it stale. Kill + switch `HISTORY_COUNT_PREFETCH_ENABLED` (runtime-only, default on). Same recipe as the + measurement: 15 more turns on the same conversation, 10 on a fresh one. + + | | `history_count` | `firstTokenMs` | what we own (`firstTokenMs` − `model`) | `pre_model` | + |---|---|---|---|---| + | before, 100–122 events (n=11) | 388 | 1560 | 791 | 225 | + | after, 122–150 events (n=14) | **0** | **1092** | **395** | 236 | + | after, fresh conversation (n=9) | 0 | 1043 | 403 | 229 | + + A 150-event conversation now costs what a fresh one does. The read itself still runs + (`lookupMs` 409–635 on the long conversation, 62–109 on the fresh one — it now overlaps + the writes and the model call), and `waitedMs` was 0 on all 25 turns. *Correctness:* + every `count=` equalled the messages stored before the turn (120, 122, … 148; 0, 2, … + 18), although each read ran long past that turn's user-message write — the cutoff is + doing the work. A read-only replay of the `GET /messages` join on both conversations put + every cost row on an assistant message, none on a user message, none past the end of the + history. *Augmented path:* two turns on a KB agent stored `displayText` at append time on + user messages 0 and 2 with `waitedMs=0`, and the reload shows it on exactly those + messages. Still unexercised: a mixed voice+text session. +- **The agent-binding history check (found 2026-10-02, fixed in this change).** On an + agent conversation's first turn and on every `@`-mention, `_session_has_messages` + asks whether the thread already has messages. It called `get_messages(limit=1)`, which + reads the whole history plus metadata, interrupts, UI resources and tool summaries and + then keeps one message: ~540ms of `rag` on a new conversation on dev, growing with + history. Constructing the `AgentCoreMemorySessionManager` behind it also wrote a + `SESSION` event as a side effect. Now `session_has_messages` + (`apis/shared/sessions/messages.py`) is one `ListEvents` — `maxResults=1`, no payloads, + `filter.eventMetadata` `stateType NOT_EXISTS` (the SDK tags its SESSION/AGENT records + with that reserved key, and messages carry none). Read-only on dev: 91ms median, 108ms + p90; the same answer as the old semantics on 42 sampled conversations, and False on all + 14 conversations that hold only a `SESSION` record. Fails open on a failed read, as the + old path effectively did. + + **Shipped (#1413) and read on dev (2026-10-02, runtime v525).** A new conversation's + first agent turn spent `rag` **571ms**, against 955–993ms after #1411 and 1,404–1,485ms + before either. `Created session` is now logged after the stream opened (during the + build), not before the assistant load. The binding rules held on real data: a second turn + on the same agent continued; a different agent in the same thread was refused ("Attempted + to change assistant mid-session", 400); an agent onto a plain thread with history was + refused ("Attempted to attach assistant to session with existing messages", 400) — the + check's *True* path. +- **B8 writes:** `mark_share_as_interacted`, `bump_last_used_at` + `resume_inactive_policies`, + and the binding persistence `store_session_metadata` become fire-and-forget tasks (strong + references held, like `_pending_title_writes`) or move to the coordinator's post-stream + metadata write. Validation stays where it is. The binding write in particular must still + land before the *next* turn's validation reads it; post-stream is early enough. +- **B8 reads:** thread `session_meta` (PR-2's snapshot) into the assistant block's two + `get_session_metadata` calls; convert `assistants/service.py`'s 14 resource constructions + to `get_dynamodb_table`. +- **Catalog reads:** `get_tool` served from the `config_cache` snapshot inside its TTL, + falling through to `get_item` on a miss. Removes N×4 round trips per cold build. + +*Tests.* Each deferred write still happens exactly once per turn and after `done` at the +latest (patch the repository, assert call order against the frames); a turn that errors +before the stream still binds; the snapshot-threaded reads behave as PR-2's did +(`test_a_snapshot_with_no_row_is_an_answer_not_a_cache_miss` pattern). V2: neutral. + +### P5 — Runtime V2 track (F9) + +In the order the V2 spec gives, with two additions from this map: + +1. Deploy-script fix (B1: `platformVersion` in `ALLOWED`, post-update assert, CLI check). +2. Explicit `PlatformVersion` flag in CDK, always set. +3. Idle-clock re-stamp in `runtime_health.py` (B2.1) — plus **an audit of every + `time.monotonic()`-keyed cache reachable at start** (tool freshness, `config_cache`, + `oauth_token_cache`, RBAC): none may be pre-populated by warm-up. P2 deliberately warms + clients and strategy ids, which are not time-bounded. +4. Dev A/B with `FirstTokenMs` (P1b) and the client-side send→first-delta gap, split cold + vs warm; `get-agent-runtime` after `platform.yml` **and** after the next `backend.yml`. +5. Decide warm-up placement from the restore logs: if warm-up work lands in the snapshot, + hold `/ping` until it completes so every restore inherits it. +6. **Prewarm on intent** (V2 spec §5a): a no-op `/invocations` action that takes no lease, + writes no rows, charges no quota and makes no model call, fired from the SPA on composer + focus with the staged session id. This is the only change that hides A3 + A4 + the first + build from the user, and it is worth more than everything in P2–P4 on a cold turn. A + later step pre-builds the agent for the current selection; keep it separate (selections + change before send). + +### P6 — Keep the room clean + +- `count_tokens` stays local; the LTM retrieval stays bounded and one-attempt; the 100ms + status poll stays. None of them are on this list because each was already measured and + chosen. +- Every hook on `BeforeModelCallEvent` keeps capturing raw facts only (CLAUDE.md). +- Nothing in P2–P5 touches the prompt, `toolConfig` or restored history. If a change here + ever needs to, it is a different spec. + +--- + +## 6. Regression plan — for hand-off + +Written so a less expensive model can run it without this document's context. Three +parts: invariants that must never move, the checklist per plan item, and the dev recipes +that settle timing claims. Run everything from the worktree with the main checkout's +interpreter: + +```bash +cd <worktree>/backend +PYTHONPATH=$PWD/src AWS_EC2_METADATA_DISABLED=true \ + <main checkout>/backend/.venv/bin/python -m pytest tests/ -q +``` + +`PYTHONPATH` is not optional: without it the interpreter imports `develop`'s code from the +main checkout and a green run proves nothing about the branch. Verify once with +`python -c "import apis.shared; print(apis.shared.__file__)"`. The suite's socket guard +fails any test that reaches the network; a new test that "passes" while hitting AWS is a +test whose assertion never ran. + +### 6A. Invariants (assert before and after every change on this path) + +| # | Invariant | How to check | +|---|---|---| +| I1 | **SSE frame order.** Head: `quota_warning?` → `quota_session_notice?` → `agent_notice?` → `citation*` → `message_start`. Deferred build: `agent_status{preparing}` first frame, `agent_status{prepared, durationMs}` before `message_start`, always paired. Tail: `message_stop` → interrupt events → `metadata` → `compaction?` → `done`. `session_title` may appear anywhere, at most once, never `"New Conversation"`. | `tests/routes/test_inference.py`, `tests/apis/inference_api/test_preparing_phase.py`, `TestRouteContract` (string pins in `routes.py`) | +| I2 | **Prompt-cache byte stability.** Same session, same configuration, consecutive turns: `toolConfigHash` and `systemPromptHash` identical; `historyHash` extends. Tool order in `toolConfig` is deterministic across builds. | `tests/agents/main_agent/test_prompt_cache_determinism.py`; on dev, `GET /admin/costs/sessions/{id}/calls` — consecutive rows with `cacheStatus=hit` and equal `toolConfigHash` | +| I3 | **One cache key per configuration, shared by all three callers.** The main turn, the resume (`is_resume=True`, `cache_write=False`, `memory_binding=snapshot.memory_binding`) and the MCP App dispatch (`cache_write=False`) compute the same key for the same session state. | `tests/apis/inference_api/test_get_agent_call_sites.py` (AST-pinned), `test_chat_service.py::test_resume_replay_from_snapshot_hits_same_cache_slot` | +| I4 | **Lease discipline.** Acquired exactly once per turn before the stream; released on every exit: stream end, cancellation, pre-stream `HTTPException`, pre-stream error; heartbeat cancelled first. | `tests/apis/inference_api/test_turn_lease_release.py`, `test_carried_steering.py` | +| I5 | **No session state on an agent instance.** `turn_lease`, `cancelled`, display-text arm, fingerprints, interrupt state are reset at the head of every turn; conversation list is aliased across instances; compaction state is re-read per turn. | `test_chat_service.py::test_second_cache_key_for_a_session_shares_the_conversation`, `tests/agents/main_agent/session/test_compaction_deferred_apply.py`, `test_turn_based_session_manager.py` | +| I6 | **Early exits stream, they do not 4xx.** Quota exceeded, retired model without successor, blocked agent binding, archived project: a conversational `message_start … done` with the metadata event, persisted via `persist_synthetic_messages`. Only ownership (404), model access (403), resume validation (400), duplicate turn (409), missing assistant (404/403) raise. | `tests/routes/test_inference.py`, `test_model_retirement_routes.py`, `test_project_harness_invocation.py` | +| I7 | **Attachment semantics.** CSV/XLSX never inline; PPTX never inline; oversized dropped with a note; per-message cap and byte budget applied before S3 reads; marker names follow attachment order; recovered attachments dropped when the turn carries its own. | `test_attachment_turn_guard.py` (37), `test_attachment_recovery.py`, `test_presentation_attachment_carveout.py`, `test_attachment_tool_autoenable.py` | +| I8 | **`turn_prelude` shape.** One line per agent turn with `stages`, `groups` (`preamble`, `agent_build`), `totalMs`, `isResume`, `deferredBuild`, `hasAssistant`; EMF metric names derived from stage names; disabled by `TURN_LATENCY_METRICS_ENABLED=false`. | `test_turn_timing.py` | +| I9 | **No network in tests.** | the socket guard in `tests/conftest.py` (fails at teardown) | +| I10 | **Warm-up never opens a connection** (until P2 deliberately adds the one strategy-id read). | `test_warmup.py` (`boto3.client` patched, assert no call methods invoked) | + +### 6B. Per-change checklist + +For each plan item, the cheaper model should: (1) run the named tests before touching +anything and record the count; (2) make the change; (3) run the same tests plus the new +ones; (4) run the full suite once; (5) run the dev recipe and paste the numbers into the +PR with the *before* numbers from the same recipe on the same day. + +| Item | Unit tests to add | Existing tests to run | Dev recipe | Accept if | +|---|---|---|---|---| +| P1a split `tools` | marks appear in order; `groups.agent_build` sums; per-server ms list is a property | `test_turn_timing.py`, `test_preparing_phase.py` | R1 on a first turn with one MCP tool enabled | sub-stages sum to `tools` ±5ms; a named owner of ≥60% of it | +| P1b first-token clock | `head_of_turn`, `pre_model`, `firstTokenMs` on the line; `FirstTokenMs` metric | `test_turn_timing.py`, `tests/agents/main_agent/streaming/` | R1 + R2 on a warm and a cold turn | `firstTokenMs` ≈ client send→first-delta minus the A2/A3 hop | +| P2 shared session at warm-up | see §5 P2 | `test_warmup.py`, `test_session_factory*.py`, `tests/agents/main_agent/core/` | R1 on three first turns per arm | `session_mgr` + `finalize` down by ≥ the parses removed (≥300ms cold); warm unchanged | +| P3a parallel constructor | see §5 P3a | `test_prompt_cache_determinism.py`, `test_base_agent_external_registration.py`, `tests/agents/main_agent/integrations/` | R1, R3 | `groups.agent_build` cold ≈ max not sum; I2 holds across 5 turns | +| P3b KB search ahead of build | see §5 P3b | `tests/routes/test_inference.py` (assistant cases), `test_project_harness_invocation.py` | R1 on a KB agent's first turn | `rag` mark ≈ 0 and `FirstTokenMs` down by ≈ the old `rag` | +| P4 bookkeeping off the path | see §5 P4 | `tests/routes/test_assistants.py`, `tests/routes/test_sessions.py`, `test_agent_binding_scoped_tools_e2e.py` | R1 + R4 | `head_of_turn` and `rag` down; every write still observed in DynamoDB after the turn | +| P5 V2 | per the V2 spec | `test_runtime_health.py`, infra `jest` | R2 cold vs warm, `get-agent-runtime` twice | no 424s on first turns; `FirstTokenMs` cold down | + +### 6C. Dev recipes + +**R1 — server-side stages.** Log group from SSM `/<prefix>/inference-api/runtime-id` +(the group is `/aws/bedrock-agentcore/runtimes/<runtime_id>-DEFAULT`; the prefix-named +group returns zero rows, not an error). Then: + +```bash +aws logs filter-log-events --log-group-name <group> --filter-pattern turn_prelude \ + --profile dev-ai --start-time <epoch ms> --query 'events[].message' --output text +``` + +Lines are wrapped in OTEL JSON with escaped quotes. Confirm the image tag in +`/<prefix>/inference-api/image-tag` changed before trusting a number as "after". A window +that spans a deploy mixes two populations: read min or individual lines, never the mean. + +**R2 — client-side first token.** In the in-app browser signed in to dev, wrap +`window.fetch` to `tee()` the `/chat/stream` body and timestamp frames matching +`agent_status|session_title|content_block_delta|done` from the moment of the request (the +recipe in the agent-state-feedback and session-title memories). The gap between request +and the first `content_block_delta` is the number the user feels; subtract `PreludeTotalMs` +(or `FirstTokenMs` after P1b) from the same turn to get the A2/A3 hop. + +**R3 — cache stability.** `GET /admin/costs/sessions/{id}/calls` for the session; on +turns 2..n, `cacheStatus` should be `hit` with equal `toolConfigHash` / `systemPromptHash`. +`partial_miss` or a changed hash after a build change means tool order or prompt bytes +moved. + +**R4 — the writes still land.** After an agent turn: the session's META row carries +`preferences.assistantId`; the assistant's `lastUsedAt` is today (first turn of the day); +the share row is marked interacted (non-owner). One `get-item` each. + +**R5 — the A/B.** `backend/scripts/experiment_agent_build_arms.py` (from #1377) with +`AGENT_BUILD_EXPERIMENT=ab` set on the dev Runtime out of band. Pass env vars explicitly; +zsh does not word-split `$var`. + +### 6D. The end-to-end matrix + +Moved to [`docs/testing/smoke-regression.md`](../testing/smoke-regression.md), which +now owns the per-release matrix: §2.2 is the scripted half +(`backend/scripts/smoke_turns.py` runs the rows that need no human and asserts I1–I3 on +the wire), §3 is the in-app-browser half for the rows that need a rendered surface. Run it +once per release on dev, as before; the invariants above and the recipes in §6C are what +its failures point back at. + +--- + +## 7. Clarity refactors + +The route's phases are now named functions with typed results, in the order the map +lists them, so the handler reads as the sequence §2B describes. Behaviour-preserving by +construction: each extraction moves a contiguous block, keeps every module-level seam the +tests patch (`_load_user_settings`, `_resolve_fallback_model`, `get_agent`, +`ensure_session_metadata_exists`, …) and every string the `TestRouteContract` and +call-site AST tests pin. See the PR that introduced this document for the list. + +Next, in this order, each its own PR: + +1. The assistant block (B8) into `resolve_agent_turn(...)` returning either a refusal to + stream or the resolved instructions/overrides/citations, with its writes moved per P4. +2. The build's three sync→async bridges (F8) into one `run_blocking_async(coro)` helper + with the contextvar capture in one place. +3. `stream_with_quota_warning` and `_guarded_stream` into a `TurnStream` object that owns + the head frames, the build, the lease heartbeat and the title poll, so the handler ends + at "return the stream". + +Rules that keep this legible afterwards: a phase function takes what it reads and returns +what it produces (no closure over handler locals); a phase that can end the turn returns +the `StreamingResponse` or raises, it does not set a flag; every new stage gets a mark; and +a comment that explains *why* stays with the code it explains, not in this document. + +--- + +## 8. Open questions + +- What is inside `agent_build.tools` (P1a answers it). +- ~~What C3 costs on a long conversation~~ — measured 2026-10-02 (§5 P4): ~2.25ms per + event plus a page step, ~390ms at 100+ events; off the critical path since #1403. The + global read was kept (voice and `@`-mention writers), so the voice+text question no + longer blocks anything. +- The container's ratio for service-model parsing (laptop ~150ms each). +- Whether AgentCore Runtime tolerates `/ping` being blocked for the length of a cold build + on V2 as it evidently does on V1. +- Whether a `CfnMemory` custom resource is worth it to make strategy ids environment + configuration instead of a start-up read. diff --git a/docs/testing/smoke-regression.md b/docs/testing/smoke-regression.md new file mode 100644 index 000000000..ca7d2ad68 --- /dev/null +++ b/docs/testing/smoke-regression.md @@ -0,0 +1,265 @@ +# Smoke and regression testing — the turn path and everything it emits + +**Audience:** any agent or developer about to merge a change, and the release +driver before a tag. Written so it can be followed without the rest of this +repo's history: each layer says what it proves, what it needs, how to run it, +and how to read a failure. + +**Scope:** the chat turn path end to end — `POST /chat/stream` on app-api, +the AgentCore Runtime `/invocations` behind it, every SSE event type in +CLAUDE.md's table, the interrupt/resume round trips, persistence and restore, +and the SPA surfaces that render them. It does not cover admin CRUD, the +marketplace, knowledge-base ingestion or voice; those have their own specs. + +The three layers are cumulative. Run as many as the change warrants (§4). + +| Layer | What it proves | Cost | Where | +|---|---|---|---| +| **L0 Hermetic** | the branch's code compiles and its unit/contract tests pass | free, ~8 min backend | `pytest`, `ng test`, `jest` | +| **L1 Turn matrix (API/SDK)** | the deployed or local stack streams every response code path with the right frames, resumes, 409s, restores | real model calls, cents | `backend/scripts/smoke_turns.py` | +| **L2 Browser** | the SPA renders what L1 proved the wire carries: consent, question picker, Stop, steer, attachment cards, reload | real model calls, minutes of agent time | in-app browser against dev | + +--- + +## 1. L0 — hermetic suites + +Backend, from the branch's checkout. `PYTHONPATH` is not optional in a +worktree: the shared venv's editable install points at the main checkout, so a +green run without it validates `develop`, not the branch. + +```bash +cd <checkout>/backend +PYTHONPATH=$PWD/src AWS_EC2_METADATA_DISABLED=true \ + <main checkout>/backend/.venv/bin/python -m pytest tests/ -q +``` + +Confirm once per worktree that the import resolves to the branch: + +```bash +PYTHONPATH=$PWD/src <main checkout>/backend/.venv/bin/python -c "import apis.shared; print(apis.shared.__file__)" +``` + +The suite's socket guard fails any test that reaches the network, and that +teardown assert is load-bearing: the fail-open production code swallows the +raised error, so a test that "passes" while hitting AWS is a test whose +assertion never ran. Do not fix a stall with dummy AWS credentials; that turns +the silent fail-open into real failures in unrelated files. + +Frontend and infrastructure: + +```bash +cd frontend/ai.client && npx ng test --watch=false +``` + +```bash +cd infrastructure && npx tsc --noEmit && npx jest +``` + +The unit tests that pin the turn-path invariants, worth running by name when a +change touches the stream: `tests/routes/test_inference.py`, +`tests/apis/inference_api/test_preparing_phase.py`, +`tests/agents/main_agent/test_prompt_cache_determinism.py`, +`tests/agents/main_agent/streaming/`. + +--- + +## 2. L1 — the turn matrix + +`backend/scripts/smoke_turns.py` drives real turns through the same +`POST /chat/stream` the SPA uses and asserts the wire contract. One run, one +table, exit 1 on any FAIL. + +### 2.1 Targets + +| Target | Command | Covers | Needs | +|---|---|---|---| +| **Local stack** | `uv run python scripts/smoke_turns.py --base-url http://127.0.0.1:8000` | everything incl. 409 / Stop / steer | app-api on :8000 with `SKIP_AUTH=true` in `backend/src/.env`, inference-api on :8001. `SKIP_AUTH_ROLES=system_admin` to read cache rows. Local code, **dev data**. | +| **Deployed, through app-api** | `SMOKE_USERNAME=… SMOKE_PASSWORD=… uv run python scripts/smoke_turns.py --base-url https://<host>/api --auth cognito` | everything | a Cognito user with a permanent password on a Hosted UI pool (the nightly pipeline's E2E users qualify; managed login v2 cannot be scripted) | +| **Local inference-api with a dev bearer** | `AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant --user-id <sub> --prefix dev-boisestateai-v2 --invocations-url http://127.0.0.1:8001` | the Runtime rows, against the **branch's** inference-api code (dev data, dev model calls) | inference-api running locally from the branch checkout; same grant as below. The fastest way to prove a turn-path change before it is deployed | +| **Deployed, straight at the Runtime** | `AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant --user-id <sub> --prefix dev-boisestateai-v2` | frame contract, resume, restore, cache rows | an **active headless grant** for `--user-id` (enable headless runs / "Run now" once while signed in; lapses 30 days after that login) and an AWS profile for the account. Skips the app-api-only rows; reads and deletes in-process through the shared services. | + +Run from `backend/` so `apis.shared` imports. The script refuses a target that +looks like production unless `--allow-prod` is passed; the prod account also +has a read-only guard, and nothing here is worth running there. + +Every run spends real model dollars as the user it runs as and writes real +session rows. Prompts are a few words, the model is the user's default unless +`--model-id` says otherwise, and the sessions it created are deleted at the +end unless `--keep`. `--delete-sessions a,b` cleans up after a kept or +crashed run without sending a turn. + +### 2.2 The rows + +Status meanings: **PASS/FAIL** assert a contract. **SKIP** means the row +cannot run here, usually a missing fixture flag, and is never a failure. +**OBSERVED** rows report what happened and never fail, because the behaviour +is a race by design. **WARN** lines under a row are model-quality or +new-event-name notes that do not change the status. + +| Row | Turn shape | Asserts | Fixture | +|---|---|---|---| +| `plain_first_turn` | one short prompt, no tools | I1 frame order: only `quota_*`/`agent_notice`/`citation`/`agent_status`/`session_title` before `message_start`; deltas after it; `message_stop` then only tail frames; `metadata` before `done`; `done` once and last; `session_title` ≤1 and never "New Conversation"; no `stream_error`; every JSON frame parses | — | +| `second_turn_cache` | second prompt, same session | I1 again; then `GET /admin/costs/sessions/{id}/calls`: last row not `partial_miss`, `toolConfigHash`/`systemPromptHash` equal across the two rows. `uncached` is normal for prompts this small. Warns if `agent_status.preparing` fired on a warm session | admin scope to read the rows; otherwise PASS on the frame contract alone | +| `restore_after_reload` | `GET /sessions/{id}/messages` | both user prompts and both streamed assistant texts come back byte-for-byte; ≥2 messages per turn | — | +| `tool_turn` | prompt forcing `--tool` (default `calculator`) | a `tool_use` and a `tool_result`; `agent_status` carries `tool_start`/`tool_end` when present; warns if the arithmetic is wrong | the tool granted to the user | +| `ask_user_question_resume` | prompt forcing `ask_user_question` | `user_question_required` **after** the final `message_stop` with `interruptId` + `questions`; resume with `{answers: {<header>: {selected, text}}}` finishes without re-raising; `pendingInterrupts` empty afterwards | `ASK_USER_QUESTION_ENABLED` (default on), tool granted | +| `tool_approval_resume` | prompt forcing `--approval-tool` | `tool_approval_required`, resume with `"approved"`, a `tool_result` follows | `--approval-tool <id>` for a tool flagged `needsApproval` (dev has an MCP server whose hello tool is) | +| `duplicate_send_409` | second POST while the first streams | HTTP 409 with the single-flight detail | app-api target | +| `stop_mid_answer` | abort the stream after 12 deltas, `POST /sessions/{id}/interrupt {reason: user_stopped}` | the next send is accepted within 45 s (lease released); reports whether the partial answer persisted. A short turn can finish server-side before the heartbeat and clear its own marker, which is by design | app-api target | +| `preview_session` | `preview-…` session id | completes; `GET /messages` empty or 404; absent from `GET /sessions` | — | +| `steer_mid_turn` | three-tool prompt, `POST /sessions/{id}/steer` on the first `tool_use` | OBSERVED: whether `steering_applied` landed or the steer lost the race (then #916's end-of-turn flush sends it as a normal turn). FAIL only if the endpoint errors; SKIP on 404 (flag off) | app-api target, `MID_TURN_STEERING_ENABLED` | +| `attach_pdf` | inline PDF generated in-script | completes; the restored user message carries a document block or the attachment marker; warns if the token was not read back | `--with-attachments` | +| `attach_csv` | inline CSV | completes; restored user text carries `[Attached files: …]` (diverted, not inlined); the model used `list_spreadsheets`/`analyze_spreadsheet` and a `tool_result` names the file, proving the diverted inline file was persisted as a session file | `--with-attachments` | +| `skill_invoke` | `enabled_skills` + `invoked_skills` | the `skills` tool is called | `--skill-id` of a granted skill | +| `kb_agent_first_turn` | `rag_assistant_id` | any `citation` frames precede `message_start`; warns if none | `--agent-id` of an agent with a KB | +| `quota_exceeded` | plain prompt as an exhausted user | `stopReason=quota_exceeded`, zero call rows | `--expect-quota-exceeded`, a test tier at its limit | + +The report's `agent_status` counts include a `preparing`/`prepared` pair on +**every** turn: the backend emits them unconditionally and the SPA delays the +render, so their presence is not a cache miss. The `prepared.durationMs` is; +the `second_turn_cache` row warns when it exceeds 200 ms on a warm session. + +**Observed-only events** are counted at the bottom of the report and never +asserted, because nothing outside the runtime can trigger them on demand: +`model_retry` (Bedrock throttling), `compaction` (needs a long history or a +threshold override), `tool_group_summary` (side-channel timing), +`steering_applied` (race). The report also prints client-clock first-token +latency per run; compare it against `turn_prelude` lines (turn-path-ttft §6C +R1), never against a different day's run. + +**Reading a failure.** The detail names the frame or field that broke the +contract. Three traps that look like product bugs: + +- `tool_turn` with no `tool_use`: the tool is not granted to the user, or the + model declined. Check `GET /tools` for the user before filing anything. +- `second_turn_cache` unreadable: the admin endpoint 403'd. Locally that is + `SKIP_AUTH_ROLES=DotNetDevelopers`; set it to `system_admin`. +- `stop_mid_answer` says the partial answer did not persist: the turn finished + server-side inside the heartbeat window and the coordinator cleared the + marker on purpose. Lengthen nothing; it is the documented control case. + +### 2.3 Adding a row + +A new SSE event or turn shape gets a row here, a line in CLAUDE.md's SSE table, +and a unit test that pins it. In the script: write `row_<name>(ctx)` returning +a `RowResult`, raise `RowFailure` for a contract breach and `RowSkip` when a +fixture is missing, append it to `ROWS`, and add the event name to +`KNOWN_EVENTS` so it stops surfacing as a WARN. + +--- + +## 3. L2 — the in-app browser + +For the rows that need a rendered surface. The in-app browser pane shares the +user's signed-in session on `https://dev.boisestate.ai`, so this runs against +the deployed artifact with real RBAC. Prefer `read_page` / `read_network_requests` +over screenshots; drive the composer with `form_input` and click +`button[aria-label="Submit message"]`. + +**Frame capture.** The SPA's SSE client resolves `window.fetch` at call time, +so wrapping it timestamps every frame without touching the app: + +```js +(() => { + window.__frames = []; + const real = window.fetch; + window.fetch = async (input, init) => { + const res = await real(input, init); + const url = typeof input === 'string' ? input : input.url; + if (!url.includes('/chat/stream') || !res.body) return res; + const t0 = performance.now(); + const [a, b] = res.body.tee(); + (async () => { + const r = a.getReader(); const dec = new TextDecoder(); let buf = ''; + for (;;) { const {value, done} = await r.read(); if (done) break; + buf += dec.decode(value, {stream: true}); + let i; while ((i = buf.indexOf('\n\n')) >= 0) { + const frame = buf.slice(0, i); buf = buf.slice(i + 2); + const m = frame.match(/^event:\s*(\S+)/m); + window.__frames.push({ms: Math.round(performance.now() - t0), name: m ? m[1] : '?', frame}); + } } + })(); + return new Response(b, {status: res.status, headers: res.headers}); + }; +})(); +``` + +Read `window.__frames.map(f => f.ms + ' ' + f.name)` after the turn. Any +non-GET call made by hand from the pane needs the `X-CSRF-Token` header mirrored +from the `__Host-bff_csrf` cookie, or it 403s. + +**Checklist** (each row: do it, confirm the expectation, reload the page, +confirm the conversation restores with the same content): + +| Row | Do | Expect | +|---|---|---| +| First turn | send a plain prompt in a new conversation | loader phases, then text; sidebar title replaces "New Conversation" during or right after the answer | +| Second turn | send again | no "Getting ready…" label (warm build); `second_turn_cache` in L1 covers the hashes | +| Agent with KB | pick an agent, ask about its corpus | citation chips render; answer uses the corpus | +| `@`-mention | mention an agent in a plain thread, then send a plain follow-up | the mention turn runs as that agent; the follow-up sees the mention's messages | +| Attach PDF / CSV / PPTX | attach, send | PDF inline; CSV and PPTX diverted with the guidance note; cards survive reload | +| Skill | enable a skill, invoke it with `/` | the skills tool call renders; the answer follows the skill | +| OAuth MCP tool | use a connector tool before consenting | Connect affordance after the answer; consent; the same turn resumes (not a new one) | +| Ask-user-question | prompt as in L1 | picker renders with Other + Skip; answering resumes; Skip resumes | +| Tool approval | call the approval-gated tool | Approve banner; one click resumes | +| Stop | send a long prompt, click Stop mid-answer | partial text stays; interrupted chip; the next send is accepted, no "already responding" | +| Steer | send a multi-tool prompt, type a follow-up and Enter while it streams | either the follow-up is injected at a tool boundary or sent as the next turn; never lost | +| Continue after `max_tokens` | force a long answer with a low max-tokens model setting | the continuation extends, does not restart | +| Quota exceeded | as a user on an exhausted test tier | conversational refusal, persisted, no spinner | +| Duplicate send | double-submit quickly | the SPA queues rather than erroring; a raw second POST 409s | +| Dark mode and mobile | toggle theme, `resize_window` mobile | tool rails, pickers and banners keep contrast and do not overflow | + +Driving dev writes real data for the signed-in user. Delete the smoke +conversations afterwards from Manage Conversations. + +--- + +## 4. When to run what + +| Change touches | Run | +|---|---| +| anything | L0 for the packages touched | +| `inference_api/chat/*`, `agents/main_agent/**`, `apis/shared/harness/**`, the SSE table | L0 + L1 against dev once the branch is deployed there (merge to `develop` auto-deploys dev), or against a local stack before | +| an interrupt flow (OAuth, approval, question, browser login) | + L1's resume rows + the matching L2 row | +| the composer, loader, tool rail, pickers, banners | + L2 rows for the surface | +| a release | L0 + full L1 against dev + the L2 checklist, once, on the release branch | + +Model-call-path changes also owe the prompt-cache and TTFT evidence described +in CLAUDE.md ("Token cost effectiveness", "TTFT is a hard budget"); +`second_turn_cache` and the first-token line in L1's report are the cheap +first look, not the whole proof. + +--- + +## 5. Where this runs in CI + +- The PR gate runs L0 path-scoped (`.github/workflows/`, see #1392). +- The nightly pipeline runs the health smoke (`scripts/nightly/smoke-test.sh`) + and, with `run-e2e`, the Playwright suite (`frontend/ai.client/e2e`) against + an ephemeral stack. The same job then runs L1 through + `scripts/nightly/turn-smoke.sh`: `smoke_turns.py --auth cognito + --with-attachments` against the stack's `/api` as the E2E user, after + `e2e-test.sh` has patched CORS and the Cognito callback list for the + CloudFront URL. The step is `continue-on-error` and uploads + `turn-smoke-report-<label>.json`; promote it to blocking once it has a green + track record. The E2E user's grants come from the `e2e_user` role that + `seed-e2e-users.sh` writes through `backend/scripts/seed_e2e_role.py` + (Cognito group `e2e-users` → `calculator`, `ask_user_question`, all + models), because the nightly stack never runs the bootstrap seeding and its + `default` role grants nothing. The pool must serve the classic Hosted UI + login page (managed login v2 exits 2, "could not start"). +- L2 is agent-run, not CI. + +--- + +## 6. Related + +- `docs/specs/turn-path-ttft.md` §6 — the invariants (I1–I3), the per-change + checklist and the dev recipes (`turn_prelude` logs, cache rows, A/B). Its + former §6D matrix now lives here as §2.2 and §3. +- `tests/load/` — the Locust suite; its Hosted UI login is the one + `smoke_turns.py --auth cognito` ports. +- `backend/scripts/spike_headless_run.py`, + `backend/scripts/experiment_agent_cache_arms.py` — the other headless-grant + drivers against dev. +- `docs/TESTING_POSTURE_REPORT.md` — coverage inventory. diff --git a/frontend/ai.client/package-lock.json b/frontend/ai.client/package-lock.json index 0d2c4e0f5..806056c06 100644 --- a/frontend/ai.client/package-lock.json +++ b/frontend/ai.client/package-lock.json @@ -1,12 +1,12 @@ { "name": "ai.client", - "version": "1.25.1", + "version": "1.26.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ai.client", - "version": "1.25.1", + "version": "1.26.0", "dependencies": { "@angular/cdk": "21.2.14", "@angular/common": "21.2.20", diff --git a/frontend/ai.client/package.json b/frontend/ai.client/package.json index 52f2ec08f..0b2920b68 100644 --- a/frontend/ai.client/package.json +++ b/frontend/ai.client/package.json @@ -1,6 +1,6 @@ { "name": "ai.client", - "version": "1.25.1", + "version": "1.26.0", "scripts": { "ng": "ng", "prestart": "tsx scripts/branding/generate-brand-theme.ts && tsx scripts/branding/generate-surface-theme.ts && tsx scripts/branding/generate-surface-colors.ts && tsx scripts/branding/generate-favicons.ts", diff --git a/frontend/ai.client/src/app/admin/manage-models/model-catalog.page.spec.ts b/frontend/ai.client/src/app/admin/manage-models/model-catalog.page.spec.ts index 7d71a0441..9ea6ff44d 100644 --- a/frontend/ai.client/src/app/admin/manage-models/model-catalog.page.spec.ts +++ b/frontend/ai.client/src/app/admin/manage-models/model-catalog.page.spec.ts @@ -255,12 +255,27 @@ describe('ModelCatalogPage', () => { }); it('derives cache rates from base input at Bedrock\'s published multipliers', () => { + // Published exceptions to the 1.25x write / 0.1x read default, each from + // a source outside this file. Anything else that drifts is a typo. + // - Claude Opus 5.5 reads at 0.05x: Price List API, $0.22 against $4.40 + // Regional ($0.20 against $4.00 Global), 2026-09-25. + // - GPT-5.5 has no write bucket: its card's cache-write cell is an em + // dash, and a live probe reported `cache_write_tokens: 0` on every turn + // while reading the prefix back from cache. + const WRITE_MULTIPLIER: Record<string, number> = { 'gpt-5-5': 0 }; + const READ_MULTIPLIER: Record<string, number> = { 'claude-opus-5-5': 0.05 }; for (const model of tieredModels) { const t = model.template; if (!t.supportsCaching) continue; const input = t.inputPricePerMillionTokens; - expect(t.cacheWritePricePerMillionTokens).toBeCloseTo(input * 1.25, 6); - expect(t.cacheReadPricePerMillionTokens).toBeCloseTo(input * 0.1, 6); + expect(t.cacheWritePricePerMillionTokens, model.key).toBeCloseTo( + input * (WRITE_MULTIPLIER[model.key] ?? 1.25), + 6, + ); + expect(t.cacheReadPricePerMillionTokens, model.key).toBeCloseTo( + input * (READ_MULTIPLIER[model.key] ?? 0.1), + 6, + ); } }); }); @@ -467,6 +482,45 @@ describe('ModelCatalogPage', () => { expect(gpt54?.template.cacheWritePricePerMillionTokens).toBe(0); }); + describe('curated Nova 2 Sonic row', () => { + // Voice mode's model. It is curated so voice sessions PRICE — with no + // catalog row for its id every voice turn was $0 against quota — and it + // is kept out of the chat picker by its SPEECH output modality. + const sonic = CURATED_BEDROCK_MODELS.find(m => m.key === 'nova-2-sonic')!; + + it('names the id the voice agent connects with', () => { + expect(sonic.template.modelId).toBe('amazon.nova-2-sonic-v1:0'); + }); + + it('declares speech in its output modalities, which is what hides it from the picker', () => { + expect(sonic.template.outputModalities).toContain('SPEECH'); + expect(sonic.template.inputModalities).toContain('SPEECH'); + }); + + it('carries both rate cards: text on input/output, speech on its own fields', () => { + // us-west-2, Price List API `AmazonBedrock`, 2026-09-30. + expect(sonic.template.inputPricePerMillionTokens).toBeCloseTo(0.319, 6); + expect(sonic.template.outputPricePerMillionTokens).toBeCloseTo(2.651, 6); + expect(sonic.template.speechInputPricePerMillionTokens).toBeCloseTo(3.0, 6); + expect(sonic.template.speechOutputPricePerMillionTokens).toBeCloseTo(12.0, 6); + }); + + it('has no prompt cache, so declares no cache rates', () => { + expect(sonic.template.supportsCaching).toBe(false); + expect(sonic.template.cacheWritePricePerMillionTokens).toBeUndefined(); + expect(sonic.template.cacheReadPricePerMillionTokens).toBeUndefined(); + }); + + it('is the only curated row with speech rates', () => { + const withSpeech = [ + ...CURATED_BEDROCK_MODELS, + ...CURATED_MANTLE_MODELS, + ...CURATED_BEDROCK_RESPONSES_MODELS, + ].filter(m => m.template.speechInputPricePerMillionTokens != null); + expect(withSpeech.map(m => m.key)).toEqual(['nova-2-sonic']); + }); + }); + describe('curated picker placement', () => { const ALL = [ ...CURATED_BEDROCK_MODELS, @@ -484,7 +538,15 @@ describe('ModelCatalogPage', () => { // provider an install may not use at all. Demoting it left Mantle with no // featured model but a specialist coding one, which the family check below // now catches. A template default cannot assume what else gets added. - const DEMOTED = ['claude-sonnet-4-6', 'qwen3-coder-30b', 'gpt-5-6-luna']; + const DEMOTED = [ + 'claude-opus-4-7', + 'claude-sonnet-4-6', + 'qwen3-coder-30b', + 'gpt-5-5', + 'gpt-5-6-sol', + 'gpt-5-6-terra', + 'gpt-5-6-luna', + ]; it('demotes exactly the superseded and specialist models', () => { const demoted = ALL.filter(m => m.template.isFeatured === false) diff --git a/frontend/ai.client/src/app/admin/manage-models/model-form.page.html b/frontend/ai.client/src/app/admin/manage-models/model-form.page.html index b9295e333..2991c47d9 100644 --- a/frontend/ai.client/src/app/admin/manage-models/model-form.page.html +++ b/frontend/ai.client/src/app/admin/manage-models/model-form.page.html @@ -703,6 +703,69 @@ <h2 class="text-base/7 font-semibold text-gray-900 dark:text-white">Pricing</h2> </div> </div> + <!-- + Speech rates. Only a speech-to-speech row (Nova 2 Sonic) has them: + its `input`/`output` above are the TEXT rates, and spoken tokens + bill ~10x higher on their own card. Left blank on a text model. + --> + <div class="space-y-4"> + <h3 class="text-sm/6 font-semibold text-gray-900 dark:text-white"> + Speech pricing + </h3> + <p class="text-sm/6 text-gray-600 dark:text-gray-400"> + Speech-to-speech models only. Spoken tokens are billed separately from the text rates above; + voice mode prices each bucket on its own rate. + </p> + <div class="grid grid-cols-1 gap-4 sm:grid-cols-2"> + <div> + <label for="speechInputPricePerMillionTokens" class="block text-sm/6 font-medium text-gray-700 dark:text-gray-300"> + Speech Input Price (per 1M tokens) + </label> + <div class="relative mt-1"> + <span class="pointer-events-none absolute inset-y-0 left-0 flex items-center pl-3 text-gray-500 dark:text-gray-400"> + $ + </span> + <input + type="number" + id="speechInputPricePerMillionTokens" + formControlName="speechInputPricePerMillionTokens" + step="0.01" + min="0" + placeholder="0.00 (optional)" + class="block w-full rounded-2xl border border-gray-300 bg-white py-2 pl-7 pr-3 text-sm/6 text-gray-900 placeholder:text-gray-400 focus:border-primary-500 focus:outline-none focus:ring-2 focus:ring-primary-500 dark:border-gray-600 dark:bg-gray-800 dark:text-white dark:placeholder:text-gray-500" + [class.border-state-danger-500]="modelForm.controls.speechInputPricePerMillionTokens.invalid && modelForm.controls.speechInputPricePerMillionTokens.touched" + /> + </div> + @if (modelForm.controls.speechInputPricePerMillionTokens.invalid && modelForm.controls.speechInputPricePerMillionTokens.touched) { + <p class="mt-1 text-sm/6 text-state-danger-600 dark:text-state-danger-400">Enter a valid price (0 or greater)</p> + } + </div> + <div> + <label for="speechOutputPricePerMillionTokens" class="block text-sm/6 font-medium text-gray-700 dark:text-gray-300"> + Speech Output Price (per 1M tokens) + </label> + <div class="relative mt-1"> + <span class="pointer-events-none absolute inset-y-0 left-0 flex items-center pl-3 text-gray-500 dark:text-gray-400"> + $ + </span> + <input + type="number" + id="speechOutputPricePerMillionTokens" + formControlName="speechOutputPricePerMillionTokens" + step="0.01" + min="0" + placeholder="0.00 (optional)" + class="block w-full rounded-2xl border border-gray-300 bg-white py-2 pl-7 pr-3 text-sm/6 text-gray-900 placeholder:text-gray-400 focus:border-primary-500 focus:outline-none focus:ring-2 focus:ring-primary-500 dark:border-gray-600 dark:bg-gray-800 dark:text-white dark:placeholder:text-gray-500" + [class.border-state-danger-500]="modelForm.controls.speechOutputPricePerMillionTokens.invalid && modelForm.controls.speechOutputPricePerMillionTokens.touched" + /> + </div> + @if (modelForm.controls.speechOutputPricePerMillionTokens.invalid && modelForm.controls.speechOutputPricePerMillionTokens.touched) { + <p class="mt-1 text-sm/6 text-state-danger-600 dark:text-state-danger-400">Enter a valid price (0 or greater)</p> + } + </div> + </div> + </div> + <!-- Prompt Caching (Bedrock Converse + bedrock-runtime Responses) --> @if (supportsCachingControls()) { <div class="space-y-4"> diff --git a/frontend/ai.client/src/app/admin/manage-models/model-form.page.ts b/frontend/ai.client/src/app/admin/manage-models/model-form.page.ts index 0d4df5edd..eefe4cf28 100644 --- a/frontend/ai.client/src/app/admin/manage-models/model-form.page.ts +++ b/frontend/ai.client/src/app/admin/manage-models/model-form.page.ts @@ -262,6 +262,8 @@ interface ModelFormGroup { outputPricePerMillionTokens: FormControl<number>; cacheWritePricePerMillionTokens: FormControl<number | null>; cacheReadPricePerMillionTokens: FormControl<number | null>; + speechInputPricePerMillionTokens: FormControl<number | null>; + speechOutputPricePerMillionTokens: FormControl<number | null>; knowledgeCutoffDate: FormControl<string | null>; supportsCaching: FormControl<boolean>; mantleApiMode: FormControl<MantleApiMode>; @@ -400,6 +402,8 @@ export class ModelFormPage implements OnInit { outputPricePerMillionTokens: this.fb.control(0, { nonNullable: true, validators: [Validators.required, Validators.min(0)] }), cacheWritePricePerMillionTokens: this.fb.control<number | null>(null, { validators: [Validators.min(0)] }), cacheReadPricePerMillionTokens: this.fb.control<number | null>(null, { validators: [Validators.min(0)] }), + speechInputPricePerMillionTokens: this.fb.control<number | null>(null, { validators: [Validators.min(0)] }), + speechOutputPricePerMillionTokens: this.fb.control<number | null>(null, { validators: [Validators.min(0)] }), knowledgeCutoffDate: this.fb.control<string | null>(null), // Seeded for the form's initial provider ('bedrock') and re-derived on // every provider switch. A hardcoded `false` here disagreed with the @@ -944,6 +948,8 @@ export class ModelFormPage implements OnInit { outputPricePerMillionTokens: model.outputPricePerMillionTokens, cacheWritePricePerMillionTokens: model.cacheWritePricePerMillionTokens ?? null, cacheReadPricePerMillionTokens: model.cacheReadPricePerMillionTokens ?? null, + speechInputPricePerMillionTokens: model.speechInputPricePerMillionTokens ?? null, + speechOutputPricePerMillionTokens: model.speechOutputPricePerMillionTokens ?? null, knowledgeCutoffDate: model.knowledgeCutoffDate, supportsCaching: model.supportsCaching ?? true, mantleApiMode: this.coerceMantleApiMode(model.apiMode), @@ -990,6 +996,8 @@ export class ModelFormPage implements OnInit { outputPricePerMillionTokens: template.outputPricePerMillionTokens, cacheWritePricePerMillionTokens: template.cacheWritePricePerMillionTokens ?? null, cacheReadPricePerMillionTokens: template.cacheReadPricePerMillionTokens ?? null, + speechInputPricePerMillionTokens: template.speechInputPricePerMillionTokens ?? null, + speechOutputPricePerMillionTokens: template.speechOutputPricePerMillionTokens ?? null, knowledgeCutoffDate: template.knowledgeCutoffDate ?? null, supportsCaching: template.supportsCaching ?? true, mantleApiMode: this.coerceMantleApiMode(template.apiMode), @@ -1247,6 +1255,8 @@ export class ModelFormPage implements OnInit { outputPricePerMillionTokens: v.outputPricePerMillionTokens, cacheWritePricePerMillionTokens: v.cacheWritePricePerMillionTokens, cacheReadPricePerMillionTokens: v.cacheReadPricePerMillionTokens, + speechInputPricePerMillionTokens: v.speechInputPricePerMillionTokens, + speechOutputPricePerMillionTokens: v.speechOutputPricePerMillionTokens, knowledgeCutoffDate: v.knowledgeCutoffDate, supportsCaching: supportsCachingForProvider(v.provider, v.supportsCaching), // Only meaningful on an OpenAI-compatible Bedrock surface; null diff --git a/frontend/ai.client/src/app/admin/manage-models/models/curated-models.ts b/frontend/ai.client/src/app/admin/manage-models/models/curated-models.ts index 7fa721018..5d5d9912b 100644 --- a/frontend/ai.client/src/app/admin/manage-models/models/curated-models.ts +++ b/frontend/ai.client/src/app/admin/manage-models/models/curated-models.ts @@ -51,7 +51,7 @@ import { ManagedModelFormData, ModelProvider, SupportedParams } from './managed- * written for one silently returns nothing for the other. Match both. * * The one real gap: the **hosted OpenAI family** (`openai.gpt-5.4`, - * `us.openai.gpt-5.6-*`, `us.openai.gpt-6-astra`) is absent from BOTH offer + * `us.openai.gpt-5.[56]*`, `us.openai.gpt-6-*`) is absent from BOTH offer * files, so its cards remain the only source. Reading that absence as * "unpublished" once put three rows into the dev catalog at GovCloud prices, * over-charging by 20%. Note the old explanation for it — "those models bill @@ -167,9 +167,48 @@ const ratesWithDerivedCache = ( }; export const CURATED_BEDROCK_MODELS: CuratedModel[] = [ + { + key: 'claude-opus-5-5', + tagline: 'Anthropic\'s most capable model — coding, knowledge work and long-running tasks.', + capabilities: ['Adaptive thinking', 'Effort control', 'Vision', 'Long context', 'Prompt caching'], + pricingTier: 'regional', + template: { + ...claude4xDefaults(), + // `us.` because dev's SCP denies every `global.*` profile (verified for + // this id 2026-09-25). Prod has no SCP: `global.anthropic.claude-opus-5-5` + // at $4.00 / $20.00 is the cheaper id there. + modelId: 'us.anthropic.claude-opus-5-5', + // Card: 1M context, 128K output, June 2026 cutoff. + maxInputTokens: 1_000_000, + modelName: 'Claude Opus 5.5', + shortDescription: 'For your toughest challenges', + maxOutputTokens: 128_000, + // Regional (CRIS): $4.40 / $22.00, cache write $5.50 (1.25x) — Price List + // API, `AmazonBedrockFoundationModels`, 2026-09-25 (the card defers to the + // pricing page). Cache READ is $0.22, **0.05x** input, not the 0.1x + // default; the Global rows ($4.00 / $0.20) show the same ratio. + ...ratesWithDerivedCache(4.4, 22.0, 0.05), + knowledgeCutoffDate: '2026-06-01', + // Measured against `us.anthropic.claude-opus-5-5` 2026-09-25: temperature + // and top_p both 400 ("deprecated for this model"), and + // `thinking.type.disabled` 400s — adaptive thinking is always on, so no + // `thinking` param is declared. The effort enum is the endpoint's own + // 400 on a bogus level; `medium` is the card's published default. + supportedParams: { + params: { + max_tokens: { supported: true, min: 1, max: 128_000, default: 32_000 }, + effort: { + supported: true, + allowed: ['low', 'medium', 'high', 'xhigh', 'max'], + default: 'medium', + }, + }, + }, + }, + }, { key: 'claude-opus-4-7', - tagline: 'Anthropic\'s most capable model — for the hardest reasoning.', + tagline: 'Previous-generation Opus — for the hardest reasoning.', capabilities: ['Adaptive thinking', 'Effort control', 'Vision', 'Prompt caching'], pricingTier: 'regional', template: { @@ -181,7 +220,9 @@ export const CURATED_BEDROCK_MODELS: CuratedModel[] = [ // needs, paying a prefix re-write and a summarizer call each time. maxInputTokens: 1_000_000, modelName: 'Claude Opus 4.7', - shortDescription: 'For your toughest challenges', + shortDescription: 'Previous-generation Opus', + // Superseded by Claude Opus 5.5, which is also cheaper. + isFeatured: false, maxOutputTokens: 64_000, // Regional (CRIS): $5.50 / $27.50. Global is $5.00 / $25.00. ...ratesWithDerivedCache(5.5, 27.5), @@ -282,6 +323,56 @@ export const CURATED_BEDROCK_MODELS: CuratedModel[] = [ }, }, }, + { + key: 'nova-2-sonic', + tagline: 'Amazon\'s speech-to-speech model — what voice mode runs on. Priced, not picked.', + capabilities: ['Speech-to-speech', 'Polyglot voices', 'Tool use'], + pricingTier: 'regional', + template: { + // No shared defaults helper fits: `claude4xDefaults()` names Anthropic + // and turns caching on, and both are wrong here. + provider: 'bedrock', + responseStreamingSupported: true, + allowedAppRoles: [], + availableToRoles: [], + enabled: true, + isDefault: false, + // In-Region only (no CRIS profile exists); us-west-2 and us-east-1 both + // host it. NOT a chat model: it answers only `InvokeModelWithBidirectionalStream`, + // so the chat picker hides any row whose output modalities include + // SPEECH and the user-facing /models list never returns it. This row + // exists so voice sessions PRICE — with no catalog row for this id, + // every voice turn was $0 against quota (`UnmeteredModelCall`, prod + // 2026-09-26). Voice reaches it by id (`NOVA_SONIC_MODEL_ID`), not by + // RBAC grant, so the roles picker is irrelevant here. + modelId: 'amazon.nova-2-sonic-v1:0', + modelName: 'Nova 2 Sonic', + shortDescription: 'Voice mode (speech-to-speech)', + iconSlug: 'amazon', + providerName: 'Amazon', + inputModalities: ['SPEECH', 'TEXT'], + outputModalities: ['SPEECH', 'TEXT'], + // Card: 1M context, 64K output. + maxInputTokens: 1_000_000, + maxOutputTokens: 64_000, + // Sonic has no prompt cache; `false` keeps the cache-rate fields empty + // rather than deriving a write premium for a bucket Bedrock never bills. + supportsCaching: false, + // us-west-2, Price List API `AmazonBedrock` offer file, usagetypes + // `USW2-NovaSonic2.0-{text,speech}-{input,output}-tokens`, 2026-09-30. + // The TEXT rates ride `input`/`output`, the SPEECH rates their own two + // fields, and `CostCalculator.calculate_voice_cost` bills each bucket on + // its own card. us-east-1 is a hair higher on text ($0.33 / $2.75) and + // identical on speech ($3.00 / $12.00). Legacy Nova Sonic v1 reached EOL + // 2026-09-14 and is deliberately not curated. + inputPricePerMillionTokens: 0.319, + outputPricePerMillionTokens: 2.651, + speechInputPricePerMillionTokens: 3.0, + speechOutputPricePerMillionTokens: 12.0, + // Inference params are fixed by the voice agent (`sessionStart`), not + // by the catalog row, so none are declared. + }, + }, ]; /** @@ -474,7 +565,8 @@ const bedrockResponsesDefaults = (): Pick< * MEASURED — the bar this comment has always set. AWS still publishes no * parameter table for these models (`model-parameters-openai.html` documents * only the open-weight gpt-oss family), so the evidence is the endpoint's own - * responses, probed against all four ids in us-west-2: + * responses, probed against all four ids in us-west-2 (and re-probed, with + * identical results, against GPT-6 Sol, GPT-6 Luna and GPT-5.5 on 2026-09-25): * * - `reasoning.effort` — sending a deliberately invalid value returns a 400 * that ENUMERATES the enum: "Supported values are: 'none', 'low', @@ -670,9 +762,78 @@ export const CURATED_BEDROCK_RESPONSES_MODELS: CuratedModel[] = [ supportedParams: openaiResponsesParams(128_000), }, }, + { + key: 'gpt-6-sol', + tagline: 'Built for demanding development work — features, debugging, refactors and review.', + capabilities: ['Reasoning', 'Vision', 'Long context', 'Prompt caching'], + pricingTier: 'regional', + template: { + ...bedrockResponsesDefaults(), + modelId: 'us.openai.gpt-6-sol', + modelName: 'GPT-6 Sol', + shortDescription: 'Strong coding and agentic work', + // Card, US Geo CRIS Short Context: $2.20 / $11.00, cache write $2.75, + // cache read $0.22. Global CRIS is $2.00 / $10.00. Cheaper than the + // GPT-5.6 Sol and Terra rows it supersedes. + ...ratesWithDerivedCache(2.2, 11.0), + knowledgeCutoffDate: null, + supportedParams: openaiResponsesParams(), + }, + }, + { + key: 'gpt-6-luna', + tagline: 'Fast and affordable — summaries, extraction and focused questions at scale.', + capabilities: ['Reasoning', 'Vision', 'Long context', 'Prompt caching'], + pricingTier: 'regional', + template: { + ...bedrockResponsesDefaults(), + modelId: 'us.openai.gpt-6-luna', + modelName: 'GPT-6 Luna', + shortDescription: 'Fast and affordable, for focused tasks', + // Card, US Geo CRIS Short Context: $0.11 / $0.55, cache write $0.1375, + // cache read $0.011. Global CRIS is $0.10 / $0.50. + ...ratesWithDerivedCache(0.11, 0.55), + knowledgeCutoffDate: null, + supportedParams: openaiResponsesParams(), + }, + }, + { + key: 'gpt-5-5', + tagline: 'OpenAI GPT-5.5 — coding, research and long-running agentic work.', + capabilities: ['Reasoning', 'Vision', 'Long context', 'Prompt caching'], + pricingTier: 'regional', + template: { + ...bedrockResponsesDefaults(), + // The card lists `bedrock-mantle` only, In-Region in us-east-1/2. That is + // stale: `us.openai.gpt-5.5` is an ACTIVE profile in us-west-2 and + // answered over bedrock-runtime Responses on 2026-09-25, while Mantle + // us-west-2 404s the model. So it rides this transport like its siblings. + modelId: 'us.openai.gpt-5.5', + modelName: 'GPT-5.5', + shortDescription: 'Previous-generation GPT', + // A June 2026 model, older than the GPT-6 family and 2.5x GPT-6 Sol's + // price — available, but not a picker default. + isFeatured: false, + // ⚠️ SINGLE SOURCE. The card publishes only In-Region: $5.50 / $33.00, + // cache read $0.55, and an em dash for cache write. No US Geo CRIS row is + // published, and the hosted OpenAI family is in neither Price List offer + // file. Every sibling card prices US Geo CRIS identically to Mantle + // In-Region (both carry the same 10% premium), so these are that row. + // The write rate is a literal 0, as on GPT-5.4: measured 2026-09-25, a + // 5.9k-token prefix read back 5,470 cached tokens on turns 2 and 3 and + // reported `cache_write_tokens: 0` on every turn — there is no write + // bucket to bill. Audit against Cost Explorer on a single-model day. + inputPricePerMillionTokens: 5.5, + outputPricePerMillionTokens: 33.0, + cacheReadPricePerMillionTokens: 0.55, + cacheWritePricePerMillionTokens: 0, + knowledgeCutoffDate: null, + supportedParams: openaiResponsesParams(), + }, + }, { key: 'gpt-5-6-sol', - tagline: 'OpenAI\'s most capable model — frontier reasoning and agentic work.', + tagline: 'Previous-generation Sol — frontier reasoning and agentic work.', capabilities: ['Reasoning', 'Vision', 'Long context', 'Prompt caching'], pricingTier: 'regional', template: { @@ -680,6 +841,8 @@ export const CURATED_BEDROCK_RESPONSES_MODELS: CuratedModel[] = [ modelId: 'us.openai.gpt-5.6-sol', modelName: 'GPT-5.6 Sol', shortDescription: 'Strong reasoning and agentic work', + // Superseded by GPT-6 Sol, which is half the price. + isFeatured: false, // Geo CRIS: $4.40 / $22.00. Global CRIS is $4.00 / $20.00. ...ratesWithDerivedCache(4.4, 22.0), knowledgeCutoffDate: null, @@ -696,6 +859,9 @@ export const CURATED_BEDROCK_RESPONSES_MODELS: CuratedModel[] = [ modelId: 'us.openai.gpt-5.6-terra', modelName: 'GPT-5.6 Terra', shortDescription: 'Balanced performance per dollar', + // GPT-6 Sol undercuts it ($2.20 / $11.00 against $2.20 / $13.20), so + // there is no price tier left for Terra to hold at the top level. + isFeatured: false, // Geo CRIS: $2.20 / $13.20. Global CRIS is $2.00 / $12.00. ...ratesWithDerivedCache(2.2, 13.2), knowledgeCutoffDate: null, diff --git a/frontend/ai.client/src/app/admin/manage-models/models/managed-model.model.ts b/frontend/ai.client/src/app/admin/manage-models/models/managed-model.model.ts index b84b3c882..70ea6624e 100644 --- a/frontend/ai.client/src/app/admin/manage-models/models/managed-model.model.ts +++ b/frontend/ai.client/src/app/admin/manage-models/models/managed-model.model.ts @@ -228,6 +228,14 @@ export interface ManagedModel { cacheWritePricePerMillionTokens?: number | null; /** Cache read price per million tokens (in USD) - Bedrock only */ cacheReadPricePerMillionTokens?: number | null; + /** + * Speech input price per million tokens (in USD). Speech-to-speech models + * only (Nova 2 Sonic): `inputPricePerMillionTokens` is the TEXT rate on + * such a model; this is the rate for spoken input, ~10x higher. + */ + speechInputPricePerMillionTokens?: number | null; + /** Speech output price per million tokens (in USD). Speech-to-speech models only. */ + speechOutputPricePerMillionTokens?: number | null; /** Knowledge cutoff date for the model */ knowledgeCutoffDate?: string | null; /** Whether this model supports prompt caching (Bedrock only) */ @@ -352,6 +360,14 @@ export interface ManagedModelFormData { cacheWritePricePerMillionTokens?: number | null; /** Cache read price per million tokens (in USD) - Bedrock only */ cacheReadPricePerMillionTokens?: number | null; + /** + * Speech input price per million tokens (in USD). Speech-to-speech models + * only (Nova 2 Sonic): `inputPricePerMillionTokens` is the TEXT rate on + * such a model; this is the rate for spoken input, ~10x higher. + */ + speechInputPricePerMillionTokens?: number | null; + /** Speech output price per million tokens (in USD). Speech-to-speech models only. */ + speechOutputPricePerMillionTokens?: number | null; /** Knowledge cutoff date for the model */ knowledgeCutoffDate?: string | null; /** Whether this model supports prompt caching (Bedrock only) */ diff --git a/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.spec.ts b/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.spec.ts index 734356474..9605e2855 100644 --- a/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.spec.ts +++ b/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.spec.ts @@ -270,6 +270,47 @@ describe('ModelDropdownComponent', () => { expect(setEffort).toHaveBeenCalledWith('high'); }); + it('rotates the trigger chevron while the menu is open', () => { + // The open state comes from the trigger's cdkMenuOpened/cdkMenuClosed + // outputs. CdkMenu has no `opened` output, so an `(opened)` binding on the + // panel compiles to a DOM listener that never fires and the chevron sticks. + const { fixture } = setup(); + const trigger: HTMLButtonElement = fixture.nativeElement.querySelector( + 'button[aria-label="Select model"]', + ); + const chevron = () => trigger.querySelector('svg')!; + expect(chevron().classList).not.toContain('rotate-180'); + + trigger.click(); + fixture.detectChanges(); + expect(menuItems().length).toBeGreaterThan(0); + expect(chevron().classList).toContain('rotate-180'); + + // A second click on the trigger toggles the menu closed. + trigger.click(); + fixture.detectChanges(); + expect(menuItems().length).toBe(0); + expect(chevron().classList).not.toContain('rotate-180'); + }); + + it('un-rotates the chevron when choosing a model closes the menu', () => { + const { fixture, setSelectedModel } = setup({ + featured: [ + makeModel({ id: 'a', modelId: 'a', modelName: 'Alpha' }), + makeModel({ id: 'b', modelId: 'b', modelName: 'Beta' }), + ], + }); + const chevron = () => + fixture.nativeElement.querySelector('button[aria-label="Select model"] svg') as SVGElement; + openMenu(fixture); + expect(chevron().classList).toContain('rotate-180'); + + itemLabelled('Beta')!.click(); + fixture.detectChanges(); + expect(setSelectedModel).toHaveBeenCalled(); + expect(chevron().classList).not.toContain('rotate-180'); + }); + it('locks the picker to a plain label when an agent pins the model', () => { const { fixture, modelService } = setup(); modelService.agentModelLocked.set(true); diff --git a/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.ts b/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.ts index 4b0114856..4a8cd149b 100644 --- a/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.ts +++ b/frontend/ai.client/src/app/components/model-dropdown/model-dropdown.component.ts @@ -1,4 +1,4 @@ -import { Component, ChangeDetectionStrategy, computed, inject, input } from '@angular/core'; +import { Component, ChangeDetectionStrategy, computed, inject, input, signal } from '@angular/core'; import { Router } from '@angular/router'; import { CdkMenuTrigger, CdkMenu, CdkMenuItem } from '@angular/cdk/menu'; import { ConnectedPosition } from '@angular/cdk/overlay'; @@ -35,6 +35,8 @@ import { ModelOptionComponent } from './components/model-option.component'; type="button" [cdkMenuTriggerFor]="modelMenu" [cdkMenuPosition]="menuPositions()" + (cdkMenuOpened)="onMenuOpened()" + (cdkMenuClosed)="onMenuClosed()" [class]="triggerSizeClass()" class="flex items-center rounded-lg text-gray-600 transition-colors hover:bg-gray-100 hover:text-gray-700 focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-[var(--color-primary)] dark:text-gray-400 dark:hover:bg-white/5 dark:hover:text-gray-300" aria-label="Select model" @@ -65,8 +67,6 @@ import { ModelOptionComponent } from './components/model-option.component'; <ng-template #modelMenu> <div cdkMenu - (closed)="onMenuClosed()" - (opened)="onMenuOpened()" class="w-72 rounded-md bg-white p-1.5 shadow-lg ring-1 ring-black/5 focus:outline-hidden dark:bg-gray-800 dark:ring-white/10 animate-in fade-in slide-in-from-top-1 duration-200" role="menu" aria-orientation="vertical" @@ -258,8 +258,12 @@ export class ModelDropdownComponent { protected sessionService = inject(SessionService); private router = inject(Router); - // Internal state - protected menuOpen = false; + /** + * Whether the model menu is open, driving the trigger's chevron. Fed by the + * trigger's `cdkMenuOpened` / `cdkMenuClosed` outputs — `CdkMenu` itself has + * no `opened` output, so an `(opened)` binding on the panel never fires. + */ + private readonly menuOpen = signal(false); /** The effort control for the selected model, or null when it offers none. */ protected readonly effortControl = this.modelService.effortControl; @@ -335,15 +339,15 @@ export class ModelDropdownComponent { ]; protected isMenuOpen(): boolean { - return this.menuOpen; + return this.menuOpen(); } protected onMenuOpened(): void { - this.menuOpen = true; + this.menuOpen.set(true); } protected onMenuClosed(): void { - this.menuOpen = false; + this.menuOpen.set(false); } protected levelLabel(level: string): string { diff --git a/frontend/ai.client/src/app/components/sidenav/components/session-list/session-list.html b/frontend/ai.client/src/app/components/sidenav/components/session-list/session-list.html index b4abff1e0..16d7b21ee 100644 --- a/frontend/ai.client/src/app/components/sidenav/components/session-list/session-list.html +++ b/frontend/ai.client/src/app/components/sidenav/components/session-list/session-list.html @@ -31,7 +31,7 @@ <div class="sidenav-body-enter flex flex-col gap-y-3" style="animation-delay: 60ms"> @for (group of groupedSessions(); track group.label; let gi = $index) { <section> - <h3 class="px-2 pb-0.5 text-[11px]/5 font-medium text-gray-400 dark:text-gray-500">{{ group.label }}</h3> + <h3 class="px-2 pb-0.5 text-[11px]/5 font-medium text-gray-400 dark:text-gray-300">{{ group.label }}</h3> <ul role="list" class="flex flex-col gap-y-0.5"> @for (entry of group.entries; track entry.kind === 'session' ? entry.session.sessionId : 'project:' + entry.projectId) { @if (entry.kind === 'session') { diff --git a/frontend/ai.client/src/app/services/announcements/announcements.service.spec.ts b/frontend/ai.client/src/app/services/announcements/announcements.service.spec.ts index e36d84447..fe506dfe1 100644 --- a/frontend/ai.client/src/app/services/announcements/announcements.service.spec.ts +++ b/frontend/ai.client/src/app/services/announcements/announcements.service.spec.ts @@ -6,7 +6,8 @@ import { } from '@angular/common/http/testing'; import { provideHttpClient } from '@angular/common/http'; import { signal } from '@angular/core'; -import { AnnouncementsService } from './announcements.service'; +import { ACK_RETRY_DELAY_MS, AnnouncementsService } from './announcements.service'; +import { SUPPRESS_ERROR_TOAST } from '../../auth/error.interceptor'; import { Announcement, AnnouncementFeed } from './announcement.model'; import { ConfigService } from '../config.service'; @@ -124,6 +125,16 @@ describe('AnnouncementsService', () => { expect(service.unreadCount()).toBe(0); expect(service.hasUnread()).toBe(false); }); + + it('opts out of the global error toast', async () => { + service.feedResource.reload(); + service.panelItems(); + await vi.waitFor(() => { + const req = httpMock.expectOne(FEED_URL); + expect(req.request.context.get(SUPPRESS_ERROR_TOAST)).toBe(true); + req.flush(makeFeed()); + }); + }); }); describe('unread count', () => { @@ -229,6 +240,87 @@ describe('AnnouncementsService', () => { await expect(done).resolves.toBe(false); }); + it('opts out of the global error toast', async () => { + // A status-0 network drop on mobile surfaced as "An error occurred" + // even though the failure is swallowed here by design (§D7). + await loadFeed(); + + const done = service.ack('a1', 'acknowledged', 'modal'); + await vi.waitFor(() => { + const req = httpMock.expectOne(`${API}/announcements/a1/ack`); + expect(req.request.context.get(SUPPRESS_ERROR_TOAST)).toBe(true); + req.flush('nope', { status: 500, statusText: 'Server Error' }); + }); + + expect(await done).toBe(false); + }); + + describe('retry', () => { + const ACK_URL = `${API}/announcements/a1/ack`; + const networkDrop = (req: ReturnType<HttpTestingController['expectOne']>) => + req.error(new ProgressEvent('error'), { status: 0, statusText: 'Unknown Error' }); + + afterEach(() => vi.useRealTimers()); + + it('retries once after a network drop and reports success', async () => { + await loadFeed(); + vi.useFakeTimers(); + + const done = service.ack('a1', 'acknowledged', 'modal'); + networkDrop(httpMock.expectOne(ACK_URL)); + + // Nothing is resent until the delay has elapsed. + httpMock.expectNone(ACK_URL); + await vi.advanceTimersByTimeAsync(ACK_RETRY_DELAY_MS); + + const retried = httpMock.expectOne(ACK_URL); + expect(retried.request.body).toEqual({ action: 'acknowledged', surface: 'modal' }); + retried.flush(null); + + expect(await done).toBe(true); + }); + + it('retries a gateway 503', async () => { + await loadFeed(); + vi.useFakeTimers(); + + const done = service.ack('a1', 'acknowledged', 'modal'); + httpMock + .expectOne(ACK_URL) + .flush('nope', { status: 503, statusText: 'Service Unavailable' }); + await vi.advanceTimersByTimeAsync(ACK_RETRY_DELAY_MS); + httpMock.expectOne(ACK_URL).flush(null); + + expect(await done).toBe(true); + }); + + it('gives up after the one retry', async () => { + await loadFeed(); + vi.useFakeTimers(); + + const done = service.ack('a1', 'acknowledged', 'modal'); + networkDrop(httpMock.expectOne(ACK_URL)); + await vi.advanceTimersByTimeAsync(ACK_RETRY_DELAY_MS); + networkDrop(httpMock.expectOne(ACK_URL)); + await vi.advanceTimersByTimeAsync(ACK_RETRY_DELAY_MS * 5); + + httpMock.expectNone(ACK_URL); + expect(await done).toBe(false); + }); + + it('does not retry a 4xx', async () => { + await loadFeed(); + vi.useFakeTimers(); + + const done = service.ack('a1', 'acknowledged', 'modal'); + httpMock.expectOne(ACK_URL).flush('nope', { status: 404, statusText: 'Not Found' }); + await vi.advanceTimersByTimeAsync(ACK_RETRY_DELAY_MS * 5); + + httpMock.expectNone(ACK_URL); + expect(await done).toBe(false); + }); + }); + it('a local dismissal hides the modal too', async () => { await loadFeed( makeFeed({ diff --git a/frontend/ai.client/src/app/services/announcements/announcements.service.ts b/frontend/ai.client/src/app/services/announcements/announcements.service.ts index 1499cee85..f856cc8f8 100644 --- a/frontend/ai.client/src/app/services/announcements/announcements.service.ts +++ b/frontend/ai.client/src/app/services/announcements/announcements.service.ts @@ -1,6 +1,7 @@ import { Injectable, computed, inject, resource, signal } from '@angular/core'; -import { HttpClient } from '@angular/common/http'; -import { firstValueFrom } from 'rxjs'; +import { HttpClient, HttpContext, HttpErrorResponse } from '@angular/common/http'; +import { firstValueFrom, retry, throwError, timer } from 'rxjs'; +import { SUPPRESS_ERROR_TOAST } from '../../auth/error.interceptor'; import { ConfigService } from '../config.service'; import { Announcement, @@ -10,6 +11,22 @@ import { AnnouncementSurface, } from './announcement.model'; +/** How long to wait before the single ack retry. */ +export const ACK_RETRY_DELAY_MS = 1000; + +/** + * Failures worth one more attempt: the request never landed (status 0 — a + * mobile network blip, a backgrounded tab) or a gateway in front of app-api + * gave up. A 4xx is the server's considered answer and retrying won't change + * it. + */ +function isTransient(error: unknown): boolean { + return ( + error instanceof HttpErrorResponse && + (error.status === 0 || error.status === 502 || error.status === 503 || error.status === 504) + ); +} + const EMPTY_FEED: AnnouncementFeed = { panel: [], banner: null, @@ -46,6 +63,16 @@ export class AnnouncementsService { () => `${this.config.appApiUrl()}/announcements`, ); + /** + * Every request here fails open and swallows its own error, so the global + * error toast must not fire either — otherwise a dropped ack (status 0 on a + * flaky mobile connection, a transient 5xx) still puts "An error occurred" + * in front of the user for something they cannot act on. + */ + private readonly options = { + context: new HttpContext().set(SUPPRESS_ERROR_TOAST, true), + }; + /** * Loads on first read. The topnav only renders the user dropdown once the * session bootstrap has resolved, so the loader fires post-auth with the @@ -95,7 +122,12 @@ export class AnnouncementsService { * * Fails open (§D7): a rejected POST still hides the item locally and * resolves rather than throwing, so no caller has to remember to catch. - * Returns whether the server accepted it, for tests and for a future retry. + * Returns whether the server accepted it. + * + * A transient failure is retried once. The endpoint is idempotent and + * monotonic (§D2), so a duplicate that did land server-side is a no-op — + * and without the retry, a dropped `acknowledged` resurfaces the modal on + * the next load. */ async ack( announcementId: string, @@ -110,7 +142,17 @@ export class AnnouncementsService { const body: AnnouncementAckRequest = { action, surface }; try { await firstValueFrom( - this.http.post<void>(`${this.baseUrl()}/${announcementId}/ack`, body), + this.http.post<void>( + `${this.baseUrl()}/${announcementId}/ack`, + body, + this.options, + ).pipe( + retry({ + count: 1, + delay: error => + isTransient(error) ? timer(ACK_RETRY_DELAY_MS) : throwError(() => error), + }), + ), ); return true; } catch { @@ -145,7 +187,7 @@ export class AnnouncementsService { private async fetchFeed(): Promise<AnnouncementFeed> { try { return await firstValueFrom( - this.http.get<AnnouncementFeed>(`${this.baseUrl()}/`), + this.http.get<AnnouncementFeed>(`${this.baseUrl()}/`, this.options), ); } catch { // The surface is kill-switched off (404) or the backend is unhappy. diff --git a/frontend/ai.client/src/app/services/user-settings.service.ts b/frontend/ai.client/src/app/services/user-settings.service.ts index d89df02fd..10bc43671 100644 --- a/frontend/ai.client/src/app/services/user-settings.service.ts +++ b/frontend/ai.client/src/app/services/user-settings.service.ts @@ -12,6 +12,12 @@ export interface UserSettings { * Blank clears; the backend caps it at {@link MAX_PERSONAL_INSTRUCTIONS}. */ personalInstructions?: string | null; + /** + * The Nova 2 Sonic voice heard in voice mode — an id from + * `session/services/voice/voice-catalog.ts`. Null means the platform + * default; blank clears. The backend refuses an id not in the catalog. + */ + voiceId?: string | null; } /** `MAX_PERSONAL_INSTRUCTIONS_CHARS` in `apis/shared/user_settings/models.py`. */ diff --git a/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.css b/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.css index 9ee1b6c87..ae4e98468 100644 --- a/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.css +++ b/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.css @@ -80,13 +80,25 @@ max-width: 720px; margin-left: auto; margin-right: auto; - padding-left: 1rem; - padding-right: 1rem; + /* No side padding on phones: the app shell's content box already insets + by 1rem there, so adding another 1rem here (plus the assistant card's + own padding) left messages ~3rem from the screen edge while the fixed + composer sat at 1rem. From sm up the shell's inset grows and this + restores the original gutter. */ + padding-left: 0; + padding-right: 0; padding-top: 3rem; /* Clear the fixed header (4rem) + small gap */ padding-bottom: 7.5rem; /* Space for fixed input */ min-width: 0; } +@media (min-width: 640px) { + .chat-messages-container.full-page { + padding-left: 1rem; + padding-right: 1rem; + } +} + /* Sidenav-aware positioning for lg screens */ @media (min-width: 1024px) { .chat-input-footer.full-page.sidenav-expanded, diff --git a/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.html b/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.html index 5982df395..933103c95 100644 --- a/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.html +++ b/frontend/ai.client/src/app/session/components/chat-container/chat-container.component.html @@ -22,7 +22,7 @@ <!-- Skeleton loading state --> @if (showSkeleton()) { - <div class="relative pb-30 pt-16 max-w-[720px] mx-auto px-4 py-6 min-w-0"> + <div class="relative pb-30 pt-16 max-w-[720px] mx-auto sm:px-4 py-6 min-w-0"> <div class="min-w-0 animate-fade-in"> <app-paragraph-skeleton /> </div> diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css index 02feb4034..a202dd6db 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.css @@ -131,3 +131,70 @@ animation: none; } } + +/* + * Voice mode's waveform comes alive on hover. + * + * Each bar is its own <path> so it can move on its own beat: the outer bars + * swell the most and the centre the least, on a symmetric delay, so the icon + * reads as sound rather than a bar chart. `transform-box: fill-box` makes the + * scale run from each bar's own centre (an SVG element's default origin is the + * viewport's corner, which would slide the bars off the icon). The hover is on + * the whole split control — the chevron included — so the two halves read as + * one thing, and `focus-within` gives keyboard users the same cue. + * + * Idle only. A live session already animates its own state in the button. + */ +.voice-wave-bar { + transform-box: fill-box; + transform-origin: center; + --voice-amp: 1.5; + --voice-delay: 0ms; +} + +.voice-wave-bar:nth-child(1), +.voice-wave-bar:nth-child(5) { + --voice-amp: 2.2; +} + +.voice-wave-bar:nth-child(2), +.voice-wave-bar:nth-child(4) { + --voice-delay: 110ms; +} + +.voice-wave-bar:nth-child(3) { + --voice-amp: 1.15; + --voice-delay: 220ms; +} + +/* + * The delay rides a custom property because the `animation` shorthand below + * resets every longhand it does not name — a plain `animation-delay` on the + * nth-child rules was silently zeroed, and all five bars moved as one. + */ +.voice-control--idle:hover .voice-wave-bar, +.voice-control--idle:focus-within .voice-wave-bar { + animation: voice-wave 900ms ease-in-out var(--voice-delay) infinite; +} + +@keyframes voice-wave { + 0%, + 100% { + transform: scaleY(1); + } + 50% { + transform: scaleY(var(--voice-amp)); + } +} + +/* The chevron sits flush against the waveform; the negative margin closes the gap without a new grid. */ +.voice-menu-trigger { + margin-left: -0.125rem; +} + +@media (prefers-reduced-motion: reduce) { + .voice-control--idle:hover .voice-wave-bar, + .voice-control--idle:focus-within .voice-wave-bar { + animation: none; + } +} diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html index 2cf7b5275..05f91c2bd 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.html @@ -356,43 +356,253 @@ } @if (showVoiceSlot()) { - <!-- Voice Toggle Button --> - <button - type="button" - (click)="toggleVoice()" - [disabled]="isDictating()" - [class]="voiceButtonClass() + ' disabled:cursor-not-allowed disabled:opacity-40'" - [attr.aria-label]="voiceAriaLabel()" - [appTooltip]="voiceTooltip()" - appTooltipPosition="top" + <!-- + Voice mode: a split control. The waveform starts (or ends) a + conversation; the chevron beside it opens the options — which + voice answers, which microphone listens. The chevron leaves while + a session is live, because both options are fixed at connect + (the voice at Bedrock's promptStart, the mic at getUserMedia). + Hovering anywhere on the control sets the waveform moving, so the + icon reads as "sound both ways" before it is pressed. + --> + <div + class="voice-control flex items-center" + [class.voice-control--idle]="!isVoiceActive()" + role="group" + aria-label="Voice mode" > - @if (voiceStatus() === 'listening') { - <!-- Pulsing dot indicator for listening --> - <span class="relative flex size-3"> - <span class="absolute inline-flex size-full animate-ping rounded-full bg-state-danger-400 opacity-75"></span> - <span class="relative inline-flex size-3 rounded-full bg-state-danger-500"></span> - </span> - } @else if (voiceStatus() === 'speaking') { - <!-- Wave bars for speaking --> - <span class="flex items-center gap-0.5" aria-hidden="true"> - <span class="h-3 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 0ms;"></span> - <span class="h-4 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 150ms;"></span> - <span class="h-3 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 300ms;"></span> - </span> - } @else if (voiceStatus() === 'connecting') { - <!-- Spinner for connecting --> - <app-spinner size="md" label="Connecting" /> - } @else { + <button + type="button" + (click)="toggleVoice()" + [disabled]="isDictating()" + [class]="voiceButtonClass() + ' disabled:cursor-not-allowed disabled:opacity-40'" + [attr.aria-label]="voiceAriaLabel()" + [appTooltip]="voiceTooltip()" + appTooltipPosition="top" + > + @if (voiceStatus() === 'listening') { + <!-- Pulsing dot indicator for listening --> + <span class="relative flex size-3"> + <span class="absolute inline-flex size-full animate-ping rounded-full bg-state-danger-400 opacity-75"></span> + <span class="relative inline-flex size-3 rounded-full bg-state-danger-500"></span> + </span> + } @else if (voiceStatus() === 'speaking') { + <!-- Wave bars for speaking --> + <span class="flex items-center gap-0.5" aria-hidden="true"> + <span class="h-3 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 0ms;"></span> + <span class="h-4 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 150ms;"></span> + <span class="h-3 w-0.5 animate-bounce rounded-full bg-state-success-500" style="animation-delay: 300ms;"></span> + </span> + } @else if (voiceStatus() === 'connecting') { + <!-- Spinner for connecting --> + <app-spinner size="md" label="Connecting" /> + } @else { + <!-- + Waveform for idle. Voice mode used to wear the microphone; the mic now + means Dictate (speech into the composer), and a conversation is sound + both ways — hence the waveform. One path per bar so each can move on + its own beat (see `.voice-wave-bar`). + --> + <svg class="voice-wave size-5" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true"> + <path class="voice-wave-bar" d="M3 10.5v3" /> + <path class="voice-wave-bar" d="M7 7.5v9" /> + <path class="voice-wave-bar" d="M11 4v16" /> + <path class="voice-wave-bar" d="M15 7.5v9" /> + <path class="voice-wave-bar" d="M19 10.5v3" /> + </svg> + } + </button> + + @if (!isVoiceActive()) { <!-- - Waveform for idle. Voice mode used to wear the microphone; the mic now - means Dictate (speech into the composer), and a conversation is sound - both ways — hence the waveform. + The open/closed events are the TRIGGER's (`cdkMenuOpened` / + `cdkMenuClosed`); `cdkMenu` itself has only `closed`. The + trigger also owns `aria-haspopup` and `aria-expanded`. --> - <svg class="size-5" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true"> - <path d="M3 10.5v3M7 7.5v9M11 4v16M15 7.5v9M19 10.5v3" /> - </svg> + <button + type="button" + [cdkMenuTriggerFor]="voiceMenu" + [cdkMenuPosition]="voiceMenuPositions" + (cdkMenuOpened)="onVoiceMenuOpened()" + (cdkMenuClosed)="onVoiceMenuClosed()" + [disabled]="isDictating()" + class="voice-menu-trigger flex h-10 w-5 items-center justify-center rounded-lg text-gray-500 transition-colors hover:bg-gray-100 hover:text-gray-700 focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-[var(--color-primary)] disabled:cursor-not-allowed disabled:opacity-40 dark:text-gray-400 dark:hover:bg-white/5 dark:hover:text-gray-300" + aria-label="Voice mode options" + appTooltip="Voice options" + appTooltipPosition="top" + > + <svg + viewBox="0 0 20 20" + fill="currentColor" + aria-hidden="true" + class="size-3.5 transition-transform" + [class.rotate-180]="voiceMenuOpen()" + > + <path + fill-rule="evenodd" + d="M5.22 8.22a.75.75 0 0 1 1.06 0L10 11.94l3.72-3.72a.75.75 0 1 1 1.06 1.06l-4.25 4.25a.75.75 0 0 1-1.06 0L5.22 9.28a.75.75 0 0 1 0-1.06Z" + clip-rule="evenodd" + /> + </svg> + </button> } - </button> + </div> + + <ng-template #voiceMenu> + <div + cdkMenu + class="voice-menu w-64 rounded-md bg-white p-1.5 shadow-lg ring-1 ring-black/5 focus:outline-hidden dark:bg-gray-800 dark:ring-white/10" + role="menu" + aria-orientation="vertical" + aria-label="Voice mode options" + > + <p class="px-3 pb-1 pt-1.5 text-[11px]/4 font-medium uppercase tracking-wide text-gray-500 dark:text-gray-400"> + Voice mode + </p> + + <!-- Which voice answers --> + <button + cdkMenuItem + type="button" + [cdkMenuTriggerFor]="voicePickerMenu" + [cdkMenuPosition]="voiceSubmenuPositions" + class="flex w-full items-center justify-between gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitem" + data-testid="voice-picker-trigger" + > + <span class="flex min-w-0 flex-col text-left"> + <span>Voice</span> + <span class="truncate text-xs/4 text-gray-500 dark:text-gray-400"> + {{ selectedVoice().name }} · {{ selectedVoice().language }} + </span> + </span> + <ng-icon name="heroChevronRight" class="size-4 shrink-0 text-gray-400 dark:text-gray-500" aria-hidden="true" /> + </button> + + <!-- Which microphone listens --> + <button + cdkMenuItem + type="button" + [cdkMenuTriggerFor]="microphoneMenu" + [cdkMenuPosition]="voiceSubmenuPositions" + class="flex w-full items-center justify-between gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitem" + data-testid="microphone-picker-trigger" + > + <span class="flex min-w-0 flex-col text-left"> + <span>Microphone</span> + <span class="truncate text-xs/4 text-gray-500 dark:text-gray-400">{{ selectedMicrophoneLabel() }}</span> + </span> + <ng-icon name="heroChevronRight" class="size-4 shrink-0 text-gray-400 dark:text-gray-500" aria-hidden="true" /> + </button> + + <div class="my-1.5 border-t border-gray-200 dark:border-gray-700"></div> + + <button + cdkMenuItem + type="button" + (click)="toggleVoice()" + [disabled]="isDictating()" + class="flex w-full items-center gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 disabled:opacity-40 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitem" + data-testid="voice-menu-start" + > + <svg class="size-4 shrink-0" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true"> + <path d="M3 10.5v3M7 7.5v9M11 4v16M15 7.5v9M19 10.5v3" /> + </svg> + <span>Start voice conversation</span> + </button> + </div> + </ng-template> + + <ng-template #voicePickerMenu> + <div + cdkMenu + class="max-h-96 w-72 overflow-y-auto rounded-md bg-white p-1.5 shadow-lg ring-1 ring-black/5 focus:outline-hidden dark:bg-gray-800 dark:ring-white/10" + role="menu" + aria-orientation="vertical" + aria-label="Voice" + data-testid="voice-picker" + > + <p class="px-3 py-2 text-xs/4 text-gray-500 dark:text-gray-400"> + Tiffany and Matthew speak every language; the rest answer in their own. + </p> + @for (group of voiceGroups; track group.language) { + <p class="px-3 pb-1 pt-2 text-[11px]/4 font-medium uppercase tracking-wide text-gray-500 dark:text-gray-400"> + {{ group.language }} + </p> + @for (voice of group.voices; track voice.id) { + <button + cdkMenuItem + type="button" + (click)="selectVoice(voice.id)" + class="flex w-full items-center justify-between gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitemradio" + [attr.aria-checked]="voice.id === selectedVoiceId()" + [attr.data-voice-id]="voice.id" + > + <span class="flex min-w-0 flex-col text-left"> + <span>{{ voice.name }}</span> + <span class="text-xs/4 text-gray-500 dark:text-gray-400">{{ voiceDescription(voice) }}</span> + </span> + @if (voice.id === selectedVoiceId()) { + <ng-icon name="heroCheck" class="size-4 shrink-0 text-primary-500 dark:text-slate-400" aria-hidden="true" /> + } + </button> + } + } + </div> + </ng-template> + + <ng-template #microphoneMenu> + <div + cdkMenu + class="w-72 rounded-md bg-white p-1.5 shadow-lg ring-1 ring-black/5 focus:outline-hidden dark:bg-gray-800 dark:ring-white/10" + role="menu" + aria-orientation="vertical" + aria-label="Microphone" + data-testid="microphone-picker" + > + <button + cdkMenuItem + type="button" + (click)="selectMicrophone(null)" + class="flex w-full items-center justify-between gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitemradio" + [attr.aria-checked]="!inputDeviceId()" + > + <span>System default</span> + @if (!inputDeviceId()) { + <ng-icon name="heroCheck" class="size-4 shrink-0 text-primary-500 dark:text-slate-400" aria-hidden="true" /> + } + </button> + @if (microphones().length === 0) { + <p class="px-3 py-2 text-sm/5 text-gray-500 dark:text-gray-400">No devices found</p> + } @else { + @if (!microphoneLabelsKnown()) { + <p class="px-3 py-1.5 text-xs/4 text-gray-500 dark:text-gray-400"> + Allow microphone access once to see device names. + </p> + } + @for (mic of microphones(); track mic.deviceId; let i = $index) { + <button + cdkMenuItem + type="button" + (click)="selectMicrophone(mic.deviceId)" + class="flex w-full items-center justify-between gap-2 rounded-xs px-3 py-2 text-sm/5 text-gray-700 outline-hidden hover:bg-gray-50 focus:bg-gray-50 dark:text-gray-300 dark:hover:bg-gray-700 dark:focus:bg-gray-700" + role="menuitemradio" + [attr.aria-checked]="mic.deviceId === inputDeviceId()" + [attr.data-device-id]="mic.deviceId" + > + <span class="truncate">{{ mic.label || 'Microphone ' + (i + 1) }}</span> + @if (mic.deviceId === inputDeviceId()) { + <ng-icon name="heroCheck" class="size-4 shrink-0 text-primary-500 dark:text-slate-400" aria-hidden="true" /> + } + </button> + } + } + </div> + </ng-template> } @if (showStop()) { diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.spec.ts b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.spec.ts index d013b830b..6ed94c090 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.spec.ts +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.spec.ts @@ -8,7 +8,8 @@ import { FileMetadata, FileUploadService } from '../../../services/file-upload'; import { SystemPromptsService } from '../../../services/system-prompts/system-prompts.service'; import { ToastService } from '../../../services/toast/toast.service'; import { ToolService } from '../../../services/tool/tool.service'; -import { VoiceChatService } from '../../services/voice'; +import { AudioRecorderService, NOVA_SONIC_VOICES, VoiceChatService, findVoice } from '../../services/voice'; +import { CdkMenu, CdkMenuItem, CdkMenuTrigger } from '@angular/cdk/menu'; import { DictationService, DictationUnavailableError, @@ -2233,3 +2234,182 @@ describe('ChatInputComponent — dictation', () => { expect(toast.info).toHaveBeenCalledWith('Dictation', 'Dictation stopped at its time limit.'); }); }); + +describe('ChatInputComponent voice mode options', () => { + let fixture: ComponentFixture<ChatInputComponent>; + let component: ChatInputComponent; + let status: ReturnType<typeof signal<'idle' | 'listening' | 'speaking' | 'connecting'>>; + let isVoiceActive: ReturnType<typeof signal<boolean>>; + let voiceId: ReturnType<typeof signal<string>>; + let setVoice: ReturnType<typeof vi.fn>; + let inputDeviceId: ReturnType<typeof signal<string | null>>; + let setInputDevice: ReturnType<typeof vi.fn>; + let devices: { deviceId: string; label: string }[]; + + beforeEach(async () => { + status = signal<'idle' | 'listening' | 'speaking' | 'connecting'>('idle'); + isVoiceActive = signal(false); + voiceId = signal('tiffany'); + setVoice = vi.fn(async (id: string) => voiceId.set(id)); + inputDeviceId = signal<string | null>(null); + setInputDevice = vi.fn((id: string | null) => inputDeviceId.set(id)); + devices = []; + + await TestBed.configureTestingModule({ + imports: [ChatInputComponent], + providers: [ + { provide: AgentMentionService, useClass: MentionServiceStub }, + { provide: SkillCommandService, useClass: SkillCommandServiceStub }, + { + provide: FileUploadService, + useValue: { + pendingUploadsList: signal([]), + hasActivePendingUploads: signal(false), + readyUploadIds: signal([]), + readyUploads: signal([]), + clearReadyUploads: () => undefined, + clearPendingUpload: () => undefined, + listSessionFiles: async () => [], + }, + }, + { provide: ToastService, useValue: { error: () => undefined, warning: () => undefined, info: () => undefined } }, + { provide: ToolService, useValue: {} }, + { + provide: VoiceChatService, + useValue: { + status, + isVoiceActive, + agentTranscript: signal(''), + voiceId, + voice: computed(() => findVoice(voiceId())!), + setVoice, + connect: vi.fn(async () => undefined), + disconnect: vi.fn(async () => undefined), + }, + }, + { + provide: AudioRecorderService, + useValue: { + // DictationService (real, root-provided) reads these two. + isSupported: signal(false), + isRecording: signal(false), + inputDeviceId, + setInputDevice, + listInputDevices: vi.fn(async () => devices), + }, + }, + { provide: SystemPromptsService, useValue: { activePrompt: signal(null) } }, + { provide: Router, useValue: { navigate: () => Promise.resolve(true) } }, + { provide: SteeringService, useClass: SteeringServiceStub }, + ], + }) + // The CDK menu directives stay real so the options menu actually opens; + // every child component is stubbed out as in the other suites. + .overrideComponent(ChatInputComponent, { + set: { imports: [CdkMenu, CdkMenuItem, CdkMenuTrigger], schemas: [NO_ERRORS_SCHEMA] }, + }) + .compileComponents(); + + fixture = TestBed.createComponent(ChatInputComponent); + component = fixture.componentInstance; + fixture.componentRef.setInput('showFileControls', false); + fixture.componentRef.setInput('showVoiceControl', true); + fixture.componentRef.setInput('autoFocus', false); + fixture.componentRef.setInput('sessionId', 's1'); + fixture.detectChanges(); + }); + + afterEach(() => { + document.querySelectorAll('.cdk-overlay-container').forEach((el) => el.remove()); + }); + + const control = () => fixture.nativeElement.querySelector('.voice-control') as HTMLElement; + const trigger = () => fixture.nativeElement.querySelector('.voice-menu-trigger') as HTMLButtonElement | null; + const menu = () => document.querySelector('.voice-menu') as HTMLElement | null; + + async function openMenu(): Promise<void> { + trigger()!.click(); + fixture.detectChanges(); + await fixture.whenStable(); + // The device list lands a microtask after the menu opens. + await new Promise((resolve) => setTimeout(resolve, 0)); + fixture.detectChanges(); + } + + it('is a split control: the waveform and an options chevron, one bar per path so they can animate', () => { + expect(control()).not.toBeNull(); + expect(control().classList).toContain('voice-control--idle'); + expect(control().querySelectorAll('.voice-wave-bar').length).toBe(5); + expect(trigger()).not.toBeNull(); + expect(trigger()!.getAttribute('aria-haspopup')).toBe('menu'); + expect(trigger()!.getAttribute('aria-expanded')).toBe('false'); + }); + + it('drops the chevron while a session is live, because the options are fixed at connect', () => { + status.set('listening'); + isVoiceActive.set(true); + fixture.detectChanges(); + expect(control().classList).not.toContain('voice-control--idle'); + expect(trigger()).toBeNull(); + }); + + it('opens the options menu naming the current voice and microphone', async () => { + await openMenu(); + expect(trigger()!.getAttribute('aria-expanded')).toBe('true'); + const items = menu()!; + expect(items.querySelector('[data-testid="voice-picker-trigger"]')!.textContent).toContain('Tiffany · English (US)'); + expect(items.querySelector('[data-testid="microphone-picker-trigger"]')!.textContent).toContain('System default'); + }); + + it('lists every voice grouped by language and persists the choice through the voice service', async () => { + await openMenu(); + (menu()!.querySelector('[data-testid="voice-picker-trigger"]') as HTMLButtonElement).click(); + fixture.detectChanges(); + await fixture.whenStable(); + fixture.detectChanges(); + + const picker = document.querySelector('[data-testid="voice-picker"]')!; + const options = Array.from(picker.querySelectorAll('[data-voice-id]')); + expect(options.length).toBe(NOVA_SONIC_VOICES.length); + expect(picker.textContent).toContain('French'); + expect(picker.querySelector('[data-voice-id="tiffany"]')!.getAttribute('aria-checked')).toBe('true'); + + (picker.querySelector('[data-voice-id="carlos"]') as HTMLButtonElement).click(); + expect(setVoice).toHaveBeenCalledWith('carlos'); + }); + + it('shows "No devices found" when there is no microphone, and the devices when there are', async () => { + await openMenu(); + (menu()!.querySelector('[data-testid="microphone-picker-trigger"]') as HTMLButtonElement).click(); + fixture.detectChanges(); + await fixture.whenStable(); + fixture.detectChanges(); + expect(document.querySelector('[data-testid="microphone-picker"]')!.textContent).toContain('No devices found'); + }); + + it('lets the user pick a microphone, and says when labels need a permission grant first', async () => { + devices = [ + { deviceId: 'dev-1', label: '' }, + { deviceId: 'dev-2', label: '' }, + ]; + await openMenu(); + (menu()!.querySelector('[data-testid="microphone-picker-trigger"]') as HTMLButtonElement).click(); + fixture.detectChanges(); + await fixture.whenStable(); + fixture.detectChanges(); + + const picker = document.querySelector('[data-testid="microphone-picker"]')!; + expect(picker.textContent).toContain('Allow microphone access once'); + expect(picker.querySelector('[data-device-id="dev-2"]')!.textContent).toContain('Microphone 2'); + + (picker.querySelector('[data-device-id="dev-2"]') as HTMLButtonElement).click(); + expect(setInputDevice).toHaveBeenCalledWith('dev-2'); + }); + + it('can start the conversation from the menu too', async () => { + const voice = TestBed.inject(VoiceChatService); + await openMenu(); + (menu()!.querySelector('[data-testid="voice-menu-start"]') as HTMLButtonElement).click(); + expect(voice.connect).toHaveBeenCalledWith('s1'); + }); +}); diff --git a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts index 26b03f620..20280429c 100644 --- a/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts +++ b/frontend/ai.client/src/app/session/components/chat-input/chat-input.component.ts @@ -16,11 +16,14 @@ import { import { FormsModule } from '@angular/forms'; import { v4 as uuidv4 } from 'uuid'; import { Router } from '@angular/router'; +import { CdkMenu, CdkMenuItem, CdkMenuTrigger } from '@angular/cdk/menu'; +import { ConnectedPosition } from '@angular/cdk/overlay'; import { NgIcon, provideIcons } from '@ng-icons/core'; import { heroPlus, heroArrowTurnDownRight, heroCheck, + heroChevronRight, heroClock, heroMicrophone, heroXMark, @@ -45,7 +48,14 @@ import { } from '../../../services/file-upload'; import { ToastService } from '../../../services/toast/toast.service'; import { ToolService } from '../../../services/tool/tool.service'; -import { VoiceChatService, type VoiceStatus } from '../../services/voice'; +import { + AudioRecorderService, + VoiceChatService, + VOICE_GROUPS, + voiceDescription, + type AudioInputDevice, + type VoiceStatus, +} from '../../services/voice'; import { DictationService, DictationUnavailableError, @@ -247,7 +257,7 @@ function dedupeAttachments(attachments: StoredAttachment[]): StoredAttachment[] @Component({ selector: 'app-chat-input', - imports: [AgentNoticeBannerComponent, AnnouncementBannerComponent, FormsModule, ModelDropdownComponent, NgIcon, QuotaWarningBannerComponent, StorageQuotaBannerComponent, TooltipDirective, FileCardComponent, AgentMentionMenuComponent, SkillCommandMenuComponent, SpinnerComponent], + imports: [AgentNoticeBannerComponent, AnnouncementBannerComponent, CdkMenu, CdkMenuItem, CdkMenuTrigger, FormsModule, ModelDropdownComponent, NgIcon, QuotaWarningBannerComponent, StorageQuotaBannerComponent, TooltipDirective, FileCardComponent, AgentMentionMenuComponent, SkillCommandMenuComponent, SpinnerComponent], // `relative` is the anchor the announcement banner floats against — it sits // `bottom-full` of this host, above the quota tabs and clear of the composer. host: { class: 'relative block' }, @@ -256,6 +266,7 @@ function dedupeAttachments(attachments: StoredAttachment[]): StoredAttachment[] heroPlus, heroArrowTurnDownRight, heroCheck, + heroChevronRight, heroClock, heroMicrophone, heroXMark, @@ -275,6 +286,7 @@ export class ChatInputComponent { private readonly draftStorage = inject(ComposerDraftStorageService); private readonly toolService = inject(ToolService); private readonly voiceChatService = inject(VoiceChatService); + private readonly audioRecorder = inject(AudioRecorderService); private readonly dictation = inject(DictationService); protected readonly systemPromptsService = inject(SystemPromptsService); private readonly router = inject(Router); @@ -776,6 +788,70 @@ export class ChatInputComponent { } }); + // ---- Voice mode options (the chevron beside the waveform) ---- + + /** Every catalog voice, grouped by language, in catalog order. */ + protected readonly voiceGroups = VOICE_GROUPS; + protected readonly voiceDescription = voiceDescription; + /** The voice the next session speaks with, and its catalog entry. */ + protected readonly selectedVoiceId = this.voiceChatService.voiceId; + protected readonly selectedVoice = this.voiceChatService.voice; + + /** The microphone the recorder will open, or null for the browser default. */ + protected readonly inputDeviceId = this.audioRecorder.inputDeviceId; + /** Microphones seen the last time the menu opened; refreshed on each open. */ + protected readonly microphones = signal<AudioInputDevice[]>([]); + /** Labels stay blank until mic access has been granted once. */ + protected readonly microphoneLabelsKnown = computed(() => this.microphones().some((m) => !!m.label)); + protected readonly selectedMicrophoneLabel = computed(() => { + const id = this.inputDeviceId(); + if (!id) return 'System default'; + const index = this.microphones().findIndex((m) => m.deviceId === id); + if (index < 0) return 'Chosen device'; + return this.microphones()[index].label || `Microphone ${index + 1}`; + }); + + protected readonly voiceMenuOpen = signal(false); + + /** + * The options menu opens upward from the control's right edge: the composer + * sits at the bottom of the viewport, and the control at its right end. + */ + protected readonly voiceMenuPositions: ConnectedPosition[] = [ + { originX: 'end', originY: 'top', overlayX: 'end', overlayY: 'bottom', offsetY: -8 }, + { originX: 'end', originY: 'bottom', overlayX: 'end', overlayY: 'top', offsetY: 8 }, + ]; + + /** Submenus fly out to the left of their row (the menu hugs the right edge), flipping right when cramped. */ + protected readonly voiceSubmenuPositions: ConnectedPosition[] = [ + { originX: 'start', originY: 'bottom', overlayX: 'end', overlayY: 'bottom', offsetX: -4 }, + { originX: 'end', originY: 'bottom', overlayX: 'start', overlayY: 'bottom', offsetX: 4 }, + { originX: 'start', originY: 'top', overlayX: 'end', overlayY: 'top', offsetX: -4 }, + { originX: 'end', originY: 'top', overlayX: 'start', overlayY: 'top', offsetX: 4 }, + ]; + + protected onVoiceMenuOpened(): void { + this.voiceMenuOpen.set(true); + // Enumerate on open, not at construction: the list changes when a headset + // is plugged in, and enumerating is cheap. Labels appear once the user has + // granted mic access, so the second open after a first session names them. + void this.audioRecorder.listInputDevices().then((devices) => this.microphones.set(devices)); + } + + protected onVoiceMenuClosed(): void { + this.voiceMenuOpen.set(false); + } + + /** Persisted on the account; applies to the next voice session. */ + protected selectVoice(voiceId: string): void { + void this.voiceChatService.setVoice(voiceId); + } + + /** Per-browser; applies to the next capture (voice mode and dictation both). */ + protected selectMicrophone(deviceId: string | null): void { + this.audioRecorder.setInputDevice(deviceId); + } + // ========================================================================= // Dictation // diff --git a/frontend/ai.client/src/app/session/components/context-meter/context-meter.component.ts b/frontend/ai.client/src/app/session/components/context-meter/context-meter.component.ts index bb34c2b26..5fecc05d5 100644 --- a/frontend/ai.client/src/app/session/components/context-meter/context-meter.component.ts +++ b/frontend/ai.client/src/app/session/components/context-meter/context-meter.component.ts @@ -154,11 +154,14 @@ function formatUsd(value: number): string { </button> @if (open()) { + <!-- z-40, not z-20: the panel shares chat-input's stacking context with + the announcement pill (z-30), which floats over the composer, so + anything lower renders the pill across the panel. --> <div id="context-meter-panel" role="region" aria-label="Context and cost details" - class="meter-panel absolute right-0 bottom-full z-20 mb-2 w-[min(20rem,calc(100vw-2rem))] rounded-xl border border-gray-200 bg-white p-3 text-left text-xs/5 text-gray-700 shadow-lg dark:border-gray-700 dark:bg-gray-800 dark:text-gray-300" + class="meter-panel absolute right-0 bottom-full z-40 mb-2 w-[min(20rem,calc(100vw-2rem))] rounded-xl border border-gray-200 bg-white p-3 text-left text-xs/5 text-gray-700 shadow-lg dark:border-gray-700 dark:bg-gray-800 dark:text-gray-300" > @if (showContext()) { <div class="flex items-baseline justify-between gap-3"> diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.css b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.css index 829efb75d..b5a6ef2e7 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.css +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.css @@ -121,6 +121,63 @@ -webkit-text-fill-color: inherit; } +/* Detail labels sit above their value, not in a gutter beside it. */ +.detail-label { + margin-bottom: 0.25rem; + font-size: 11px; + font-weight: 500; + line-height: 1rem; +} + +/* The bounded box every input/output/result value renders in. Long payloads + scroll rather than being truncated; text wraps anywhere, so a long URL or + token cannot force a horizontal scrollbar. + + Styled here rather than with utilities because two of these boxes are + <pre>, and the message block's global prose rules (styles.css, unlayered) + restyle every <pre> — padding, size, background — beating any Tailwind + utility. The encapsulated selectors below out-rank those rules in both + themes. */ +.detail-scroll { + overflow: auto; + margin: 0; + max-width: 100%; + border: 1px solid var(--color-gray-100); + border-radius: 0.125rem; + background-color: var(--color-gray-50); + padding: 0.375rem 0.5rem; + font-size: 0.75rem; + line-height: 1rem; + white-space: pre-wrap; + overflow-wrap: anywhere; +} + +:host-context(.dark) .detail-scroll { + border-color: color-mix(in oklab, var(--color-gray-700) 30%, transparent); + background-color: color-mix(in oklab, var(--color-gray-800) 60%, transparent); +} + +.detail-scroll:focus-visible { + outline: 2px solid var(--color-primary-500); + outline-offset: 1px; +} + +/* Result items are <pre> for whitespace fidelity; inside the box they are + plain flow, with a little air between consecutive items. */ +.result-block pre { + margin: 0; + padding: 0; + border-radius: 0; + background: transparent; + font-size: inherit; + white-space: pre-wrap; + overflow-wrap: anywhere; +} + +.result-block pre + pre { + margin-top: 0.5rem; +} + /* Muted JSON syntax highlighting for result blocks */ .result-block .json-key { color: #6b7280; /* gray-500 */ diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.html b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.html index 723ac4a19..b5a879ded 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.html +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.html @@ -126,81 +126,83 @@ reachable by screen readers on rows nobody opened. The enter animation supplies the accordion feel without that cost. --> @if (isCallDetailOpen(call)) { - <div class="detail-panel space-y-1 border-t border-gray-200/60 bg-gray-50/60 px-2.5 py-2 dark:border-white/5 dark:bg-black/20"> + <!-- Labels sit ABOVE their values, not in a gutter beside them: a + gutter column spends width on every line of a payload that is + usually the widest thing in the rail. Long values scroll inside + a bounded box rather than being cut off behind a "show more" — + the whole payload is always there to check the summary against, + and the box keeps one big result from pushing the answer off + screen. Each box is focusable so a keyboard user can scroll it. --> + <div class="detail-panel space-y-2.5 border-t border-gray-200/60 bg-gray-50/60 px-2.5 py-2 dark:border-white/5 dark:bg-black/20"> <!-- Input --> @if (call.input && (call.input | keyvalue)?.length) { - <div class="flex items-baseline gap-1.5"> - <span class="text-xs text-gray-400 dark:text-gray-500 shrink-0">input:</span> - <span class="text-xs text-gray-400 dark:text-gray-500 font-mono truncate"> - {{ formatInput(call.input) }} - </span> + <div class="call-input"> + <div class="detail-label text-gray-500 dark:text-gray-400">Input</div> + <pre + class="detail-scroll max-h-40 font-mono text-gray-500 dark:text-gray-400" + tabindex="0" + role="group" + [attr.aria-label]="'Input for ' + call.toolName" + >{{ formatInput(call.input) }}</pre> </div> } <!-- Live streaming output (e.g. artifact being generated) --> @if (isGenerating(call)) { - <div class="flex items-start gap-1.5"> - <span class="text-xs text-gray-400 dark:text-gray-500 shrink-0 mt-1">output:</span> - <div class="w-full"> - <div class="flex items-center gap-1.5 mb-1 text-xs text-gray-400 dark:text-gray-500"> - <span class="status-dot bg-state-warning-400 shimmer"></span> - <span>Generating output…</span> - </div> - <pre - class="max-h-56 overflow-auto whitespace-pre-wrap break-all font-mono text-xs text-gray-500 dark:text-gray-400 bg-gray-50 dark:bg-gray-800/60 border border-gray-100 dark:border-gray-700/30 rounded-xs px-2 py-1.5" - aria-label="Tool output being generated" - [appPinScrollToBottom]="call.streamingContent ?? ''" - >{{ call.streamingContent }}</pre> + <div> + <div class="detail-label flex items-center gap-1.5 text-gray-500 dark:text-gray-400"> + <span>Output</span> + <span class="status-dot bg-state-warning-400 shimmer"></span> + <span class="font-normal">Generating…</span> </div> + <pre + class="detail-scroll max-h-56 font-mono text-gray-500 dark:text-gray-400" + tabindex="0" + role="group" + aria-label="Tool output being generated" + [appPinScrollToBottom]="call.streamingContent ?? ''" + >{{ call.streamingContent }}</pre> </div> } <!-- Result --> @if (call.result) { - <div class="flex items-start gap-1.5"> - <span class="text-xs shrink-0 mt-1" - [class.text-gray-400]="call.result.status !== 'error'" - [class.dark:text-gray-500]="call.result.status !== 'error'" - [class.text-state-danger-500]="call.result.status === 'error'" + <div class="call-result"> + <div + class="detail-label" + [class.text-gray-500]="call.result.status !== 'error'" + [class.dark:text-gray-400]="call.result.status !== 'error'" + [class.text-state-danger-600]="call.result.status === 'error'" [class.dark:text-state-danger-400]="call.result.status === 'error'" - >result:</span> - <div class="result-block text-xs text-gray-500 dark:text-gray-400 bg-gray-50 dark:bg-gray-800/60 border border-gray-100 dark:border-gray-700/30 rounded-xs px-2 py-1.5 w-full overflow-hidden"> - @if (isResultExpanded(call.id)) { - <!-- Full result with syntax highlighting --> + >{{ call.result.status === 'error' ? 'Error' : 'Result' }}</div> + @if (hasResultBody(call)) { + <div + class="result-block detail-scroll max-h-80 text-gray-500 dark:text-gray-400" + tabindex="0" + role="group" + [attr.aria-label]="'Result from ' + call.toolName" + > @for (item of call.result.content; track $index) { @if (item.json) { - <pre class="font-mono text-xs overflow-x-auto"><code [innerHTML]="formatResultContent(item) | jsonSyntaxHighlight"></code></pre> + <pre class="font-mono"><code [innerHTML]="formatResultContent(item) | jsonSyntaxHighlight"></code></pre> } @else if (item.text) { - <div class="whitespace-pre-wrap break-all">{{ item.text }}</div> + <pre class="font-mono">{{ item.text }}</pre> } } - } @else { - <!-- Truncated result --> - <span class="font-mono break-all whitespace-pre-wrap text-gray-500 dark:text-gray-400">{{ truncateResult(getResultText(call)) }}</span> - } + </div> + } - @if (getResultText(call).length > 200) { - <button - type="button" - (click)="toggleFullResult(call.id)" - class="text-primary-accessible dark:text-primary-accessible-dark hover:underline mt-1 block text-xs cursor-pointer" - > - {{ isResultExpanded(call.id) ? 'Show less' : 'Show full result' }} - </button> - } - </div> + <!-- Images (if any) --> + @for (item of getResultImages(call); track $index) { + <div class="mt-1 rounded-xs overflow-hidden bg-gray-100 dark:bg-gray-800 p-1 border border-gray-200 dark:border-gray-700"> + <img + [src]="getImageDataUrl(item)" + alt="Tool result image" + class="max-w-full h-auto rounded-xs" + /> + </div> + } </div> - - <!-- Images (if any) --> - @for (item of getResultImages(call); track $index) { - <div class="mt-1 rounded-xs overflow-hidden bg-gray-100 dark:bg-gray-800 p-1 border border-gray-200 dark:border-gray-700"> - <img - [src]="getImageDataUrl(item)" - alt="Tool result image" - class="max-w-full h-auto rounded-xs" - /> - </div> - } } </div> } diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.spec.ts b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.spec.ts index a3a69aa43..aabacb94e 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.spec.ts +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.spec.ts @@ -72,17 +72,12 @@ describe('ToolRailComponent', () => { } @if (call.result) { <div class="call-result"> - @if (isResultExpanded(call.id)) { - @for (item of call.result.content; track $index) { - @if (item.text) { <div>{{ item.text }}</div> } - } - } @else { - <span>{{ truncateResult(getResultText(call)) }}</span> - } - @if (getResultText(call).length > 200) { - <button type="button" class="toggle-result" (click)="toggleFullResult(call.id)"> - {{ isResultExpanded(call.id) ? 'Show less' : 'Show full result' }} - </button> + @if (hasResultBody(call)) { + <div class="result-body"> + @for (item of call.result.content; track $index) { + @if (item.text) { <pre>{{ item.text }}</pre> } + } + </div> } </div> @for (item of getResultImages(call); track $index) { @@ -402,29 +397,29 @@ describe('ToolRailComponent', () => { expect(fixture.nativeElement.querySelector('.call-input')).toBeNull(); }); - // Split into two cases on purpose: the detail is now a real `@if`, so a - // second `toggle*` call in one case folds it back up and the assertion - // silently measures the wrong state. - it('offers no "show full result" for a short result', () => { + it('renders a long result in full, with no "show full result" toggle', () => { + // Long results scroll inside a bounded box instead of being cut off + // behind a toggle — the whole payload is always there to check. + const long = 'A'.repeat(300); render( makeGroup({ - calls: [makeCall({ result: { status: 'success', content: [{ text: 'short' }] } })], + calls: [makeCall({ result: { status: 'success', content: [{ text: long }] } })], }), ); component.toggleExpanded(); component.toggleCallDetail('tool-1'); fixture.detectChanges(); - expect(fixture.nativeElement.querySelector('.call-result')).toBeTruthy(); - expect(fixture.nativeElement.querySelector('.toggle-result')).toBeNull(); + expect(fixture.nativeElement.querySelector('.result-body').textContent).toContain(long); + expect(fixture.nativeElement.textContent).not.toContain('Show full result'); }); - it('offers "show full result" for a truncated one', () => { + it('renders no result body for an image-only result', () => { render( makeGroup({ calls: [ makeCall({ - result: { status: 'success', content: [{ text: 'A'.repeat(300) }] }, + result: { status: 'success', content: [{ image: { format: 'png', data: 'x' } }] }, }), ], }), @@ -433,19 +428,8 @@ describe('ToolRailComponent', () => { component.toggleCallDetail('tool-1'); fixture.detectChanges(); - expect(fixture.nativeElement.querySelector('.toggle-result')).toBeTruthy(); - }); - - it('tracks result expansion per call', () => { - render( - makeGroup({ - calls: [makeCall({ id: 'a' }), makeCall({ id: 'b' })], - }), - ); - - component.toggleFullResult('a'); - expect(component.isResultExpanded('a')).toBe(true); - expect(component.isResultExpanded('b')).toBe(false); + expect(fixture.nativeElement.querySelector('.call-result')).toBeTruthy(); + expect(fixture.nativeElement.querySelector('.result-body')).toBeNull(); }); it('renders images from result content', () => { @@ -634,20 +618,10 @@ describe('ToolRailComponent', () => { expect(component.formatDuration(250)).toBe('250ms'); }); - it('formats input as key-value pairs', () => { - const result = component.formatInput({ query: 'test', limit: 5 }); - expect(result).toContain('query: "test"'); - expect(result).toContain('limit: 5'); - }); - - it('truncates long text', () => { - const truncated = component.truncateResult('A'.repeat(300), 200); - expect(truncated.length).toBe(203); - expect(truncated.endsWith('...')).toBe(true); - }); - - it('leaves short text alone', () => { - expect(component.truncateResult('Hello')).toBe('Hello'); + it('formats input as one key-value pair per line', () => { + expect(component.formatInput({ query: 'test', limit: 5 })).toBe( + 'query: "test"\nlimit: 5', + ); }); it('builds an image data URL', () => { @@ -660,27 +634,25 @@ describe('ToolRailComponent', () => { expect(component.getImageDataUrl({ text: 'hello' })).toBe(''); }); - it('combines text and json result items', () => { - const text = component.getResultText( - makeCall({ - result: { - status: 'success', - content: [{ text: 'hello' }, { json: { key: 'value' } }], - }, - }), - ); - expect(text).toContain('hello'); - expect(text).toContain('"key"'); + it('has a result body for text or json items', () => { + expect( + component.hasResultBody( + makeCall({ result: { status: 'success', content: [{ json: { key: 'value' } }] } }), + ), + ).toBe(true); }); - it('represents an image item as [image]', () => { + it('has no result body for an image-only or empty result', () => { expect( - component.getResultText( + component.hasResultBody( makeCall({ result: { status: 'success', content: [{ image: { format: 'png', data: 'x' } }] }, }), ), - ).toBe('[image]'); + ).toBe(false); + expect(component.hasResultBody(makeCall({ result: { status: 'success', content: [] } }))).toBe( + false, + ); }); }); }); diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.ts b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.ts index 3ffc4293d..8a3a68c4c 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.ts +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-rail/tool-rail.component.ts @@ -38,9 +38,6 @@ export class ToolRailComponent { */ expandedCallIds = signal<Set<string>>(new Set()); - /** Track which individual tool results are fully expanded (for long results in fallback mode) */ - expandedResultIds = signal<Set<string>>(new Set()); - /** * The group's calls segmented by backend batch. * @@ -146,24 +143,6 @@ export class ToolRailComponent { return this.isGenerating(call) || this.expandedCallIds().has(call.id); } - /** Toggle full result display for a specific tool call */ - toggleFullResult(callId: string): void { - this.expandedResultIds.update(ids => { - const next = new Set(ids); - if (next.has(callId)) { - next.delete(callId); - } else { - next.add(callId); - } - return next; - }); - } - - /** Check if a tool call's result is fully expanded */ - isResultExpanded(callId: string): boolean { - return this.expandedResultIds().has(callId); - } - /** CSS class for status dot */ statusDotClass(call: ToolCallDisplay): string { switch (call.status) { @@ -188,31 +167,16 @@ export class ToolRailComponent { return ms >= 1000 ? `${(ms / 1000).toFixed(1)}s` : `${ms}ms`; } - /** Compact one-line display of tool input params */ + /** Tool input params, one `key: value` per line */ formatInput(inputObj: Record<string, unknown>): string { return Object.entries(inputObj) .map(([k, v]) => `${k}: ${JSON.stringify(v)}`) - .join(', '); - } - - /** Get combined text from result content array, for truncation */ - getResultText(call: ToolCallDisplay): string { - if (!call.result?.content) return ''; - return call.result.content - .map(item => { - if (item.text) return item.text; - if (item.json) return JSON.stringify(item.json, null, 2); - if (item.image) return '[image]'; - return ''; - }) - .filter(Boolean) .join('\n'); } - /** Truncate result text for collapsed display */ - truncateResult(text: string, maxLen = 200): string { - if (text.length <= maxLen) return text; - return text.substring(0, maxLen) + '...'; + /** Whether the result carries any text or JSON to show (images render separately). */ + hasResultBody(call: ToolCallDisplay): boolean { + return !!call.result?.content?.some(item => item.text || item.json); } /** Get image items from result content */ diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.spec.ts b/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.spec.ts new file mode 100644 index 000000000..d35319df5 --- /dev/null +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.spec.ts @@ -0,0 +1,42 @@ +import { describe, it, expect } from 'vitest'; +import { DomSanitizer } from '@angular/platform-browser'; +import { JsonSyntaxHighlightPipe } from './json-syntax-highlight.pipe'; + +const passthrough = { bypassSecurityTrustHtml: (html: string) => html } as unknown as DomSanitizer; + +function highlight(value: unknown): string { + return new JsonSyntaxHighlightPipe(passthrough).transform( + JSON.stringify(value, null, 2), + ) as string; +} + +function text(html: string): string { + return html.replace(/<[^>]+>/g, ''); +} + +describe('JsonSyntaxHighlightPipe', () => { + it('leaves the text of string values untouched', () => { + // The chained-replace version rewrote "09:30:17" as "09: 30: 17". + const value = { current_time: '2026-09-30T09:30:17-06:00', note: 'a: true, b: null {x}' }; + expect(text(highlight(value))).toBe(JSON.stringify(value, null, 2)); + }); + + it('classifies keys, strings, numbers, booleans, null and brackets', () => { + const html = highlight({ k: 'v', n: -1.5, t: true, z: null, a: [1] }); + expect(html).toContain('<span class="json-key">"k"</span>:'); + expect(html).toContain('<span class="json-string">"v"</span>'); + expect(html).toContain('<span class="json-number">-1.5</span>'); + expect(html).toContain('<span class="json-boolean">true</span>'); + expect(html).toContain('<span class="json-null">null</span>'); + expect(html).toContain('<span class="json-bracket">[</span>'); + }); + + it('escapes HTML inside values', () => { + expect(highlight({ x: '<img src=x>' })).toContain('<img src=x>'); + }); + + it('keeps escaped quotes inside a string', () => { + const value = { q: 'say "hi": 5' }; + expect(text(highlight(value))).toBe(JSON.stringify(value, null, 2)); + }); +}); diff --git a/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.ts b/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.ts index b27411516..12a9862ad 100644 --- a/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.ts +++ b/frontend/ai.client/src/app/session/components/message-list/components/tool-use/json-syntax-highlight.pipe.ts @@ -18,38 +18,26 @@ export class JsonSyntaxHighlightPipe implements PipeTransform { .replace(/</g, '<') .replace(/>/g, '>'); - // Apply syntax highlighting - const highlighted = escaped - // Highlight keys (property names) - .replace( - /"([^"]+)"(?=\s*:)/g, - '<span class="json-key">"$1"</span>' - ) - // Highlight string values - .replace( - /:\s*"([^"]*)"/g, - ': <span class="json-string">"$1"</span>' - ) - // Highlight numbers - .replace( - /:\s*(\d+\.?\d*)/g, - ': <span class="json-number">$1</span>' - ) - // Highlight booleans - .replace( - /:\s*(true|false)/g, - ': <span class="json-boolean">$1</span>' - ) - // Highlight null - .replace( - /:\s*(null)/g, - ': <span class="json-null">$1</span>' - ) - // Highlight brackets - .replace( - /([{}\[\]])/g, - '<span class="json-bracket">$1</span>' - ); + // One pass over the whole document, so each token is classified exactly + // once. Chained per-kind replaces re-scanned text already inside a string + // and highlighted it again — a timestamp value like "09:30:17" came out as + // "09: 30: 17", because the number rule matched the `:30` inside it. + const highlighted = escaped.replace( + /("(?:\\.|[^"\\])*")(\s*:)?|\b(?:true|false)\b|\bnull\b|-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?|[{}[\]]/g, + (match: string, str?: string, colon?: string) => { + if (str) { + return colon + ? `<span class="json-key">${str}</span>${colon}` + : `<span class="json-string">${str}</span>`; + } + if (match === 'true' || match === 'false') { + return `<span class="json-boolean">${match}</span>`; + } + if (match === 'null') return `<span class="json-null">${match}</span>`; + if (/[{}[\]]/.test(match)) return `<span class="json-bracket">${match}</span>`; + return `<span class="json-number">${match}</span>`; + }, + ); return this.sanitizer.bypassSecurityTrustHtml(highlighted); } diff --git a/frontend/ai.client/src/app/session/components/message-list/message-list.component.html b/frontend/ai.client/src/app/session/components/message-list/message-list.component.html index 271faec7f..840150353 100644 --- a/frontend/ai.client/src/app/session/components/message-list/message-list.component.html +++ b/frontend/ai.client/src/app/session/components/message-list/message-list.component.html @@ -33,7 +33,7 @@ class="group relative min-w-0" role="group" aria-label="Assistant message with metadata"> - <div class="rounded-2xl bg-gray-50 px-4 py-3 text-base/6 text-gray-800 dark:bg-gray-900 dark:text-gray-200 overflow-hidden"> + <div class="rounded-2xl bg-gray-50 px-0 sm:px-4 py-3 text-base/6 text-gray-800 dark:bg-gray-900 dark:text-gray-200 overflow-hidden"> <app-assistant-message class="min-w-0 block" [messages]="segment.messages" diff --git a/frontend/ai.client/src/app/session/services/chat/chat-http.service.spec.ts b/frontend/ai.client/src/app/session/services/chat/chat-http.service.spec.ts index 23ec7c97c..b34655ef6 100644 --- a/frontend/ai.client/src/app/session/services/chat/chat-http.service.spec.ts +++ b/frontend/ai.client/src/app/session/services/chat/chat-http.service.spec.ts @@ -30,7 +30,7 @@ describe('ChatHttpService', () => { // attaching a Bearer manually. { provide: BffSessionService, useValue: { csrfHeaders: vi.fn().mockReturnValue({}), handleUnauthorized: vi.fn() } }, { provide: SessionService, useValue: { currentSession: signal({ sessionId: 's1' }), updateSessionTitleInCache: vi.fn(), getSessionMetadata: vi.fn().mockResolvedValue({}), isNewSession: vi.fn().mockReturnValue(false) } }, - { provide: StreamParserService, useValue: { getCurrentStreamId: vi.fn().mockReturnValue('stream-1'), parseEventSourceMessage: vi.fn() } }, + { provide: StreamParserService, useValue: { getCurrentStreamId: vi.fn().mockReturnValue('stream-1'), parseEventSourceMessage: vi.fn(), hasReceivedDone: vi.fn().mockReturnValue(false) } }, { provide: ChatStateService, useValue: { abortRequest: vi.fn(), setChatLoading: vi.fn(), setLastTurnInterrupted: vi.fn(), seedSessionAggregates: vi.fn(), createAbortController: vi.fn().mockReturnValue(new AbortController()), releaseAbortController: vi.fn(), streamingSessionIds: vi.fn().mockReturnValue([]) } }, { provide: MessageMapService, useValue: { endStreaming: vi.fn() } }, { provide: ErrorService, useValue: { handleHttpError: vi.fn(), addError: vi.fn() } }, @@ -212,6 +212,228 @@ describe('ChatHttpService', () => { vi.restoreAllMocks(); }); + describe('Stop or page-hide racing the end of the turn', () => { + // `done` means the server's turn is over, but loading (and so the Stop + // button) only clears on the transport's close, which can trail it — on a + // first turn `session_title` may arrive after `done`. A Stop in that + // window must not stamp a false "interrupted" marker on a finished turn, + // and neither may a page departure: the session is still listed as + // streaming until close, so page-hide attribution would otherwise pick it. + + const encoder = new TextEncoder(); + let controller: AbortController; + let parser: any; + let fetchSpy: any; + + /** + * Serve `/chat/stream` from a body that emits `frames` and then stays + * open — the transport has not closed, so `onclose` has not run. + */ + function openStream(frames: string): void { + fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input) => { + if (!String(input).endsWith('/chat/stream')) { + return new Response(null, { status: 204 }); + } + const body = new ReadableStream<Uint8Array>({ + start(c) { + c.enqueue(encoder.encode(frames)); + }, + }); + return new Response(body, { + status: 200, + headers: { 'content-type': 'text/event-stream' }, + }); + }); + } + + function interruptPosts(): [string, RequestInit][] { + return fetchSpy.mock.calls.filter(([url]: [string]) => String(url).includes('/interrupt')); + } + + beforeEach(() => { + controller = new AbortController(); + chatStateService.createAbortController.mockReturnValue(controller); + chatStateService.abortRequest.mockImplementation(() => controller.abort()); + + // Stand-in for the parser's own bookkeeping: record `done` per session. + parser = TestBed.inject(StreamParserService); + const doneFor = new Set<string>(); + parser.parseEventSourceMessage.mockImplementation((sessionId: string, event: string) => { + if (event === 'done') doneFor.add(sessionId); + }); + parser.hasReceivedDone.mockImplementation((sessionId: string) => doneFor.has(sessionId)); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('a Stop after done but before close tears down the transport without marking the turn interrupted', async () => { + const messageMap = TestBed.inject(MessageMapService) as any; + openStream('event: message_start\ndata: {"role":"assistant"}\n\nevent: done\ndata: {}\n\n'); + + const streaming = service.sendChatRequest({ session_id: 's1', message: 'hi' }); + await vi.waitFor(() => expect(parser.hasReceivedDone('s1')).toBe(true)); + // Still open: the stream's own teardown has not run. + expect(chatStateService.releaseAbortController).not.toHaveBeenCalled(); + expect(chatStateService.setChatLoading).not.toHaveBeenCalled(); + + service.cancelChatRequest('s1'); + await streaming; + + expect(interruptPosts()).toHaveLength(0); + expect(chatStateService.setLastTurnInterrupted).not.toHaveBeenCalled(); + // The transport is still torn down and the UI leaves its streaming state. + expect(controller.signal.aborted).toBe(true); + expect(messageMap.endStreaming).toHaveBeenCalledWith('s1'); + expect(chatStateService.setChatLoading).toHaveBeenCalledWith('s1', false); + }); + + it('a Stop mid-stream still marks the turn interrupted', async () => { + openStream('event: message_start\ndata: {"role":"assistant"}\n\n'); + + const streaming = service.sendChatRequest({ session_id: 's1', message: 'hi' }); + await vi.waitFor(() => + expect(parser.parseEventSourceMessage).toHaveBeenCalledWith( + 's1', + 'message_start', + expect.anything(), + 'stream-1', + ), + ); + + service.cancelChatRequest('s1'); + await streaming; + + const posts = interruptPosts(); + expect(posts).toHaveLength(1); + expect(JSON.parse(String(posts[0][1].body))).toEqual({ reason: 'user_stopped' }); + expect(chatStateService.setLastTurnInterrupted).toHaveBeenCalledWith('s1', true, 'user_stopped'); + expect(controller.signal.aborted).toBe(true); + }); + + it('a page-hide after done but before close sends no navigated_away', async () => { + chatStateService.streamingSessionIds.mockReturnValue(['s1']); + openStream('event: message_start\ndata: {"role":"assistant"}\n\nevent: done\ndata: {}\n\n'); + + const streaming = service.sendChatRequest({ session_id: 's1', message: 'hi' }); + await vi.waitFor(() => expect(parser.hasReceivedDone('s1')).toBe(true)); + expect(chatStateService.releaseAbortController).not.toHaveBeenCalled(); + + window.dispatchEvent(new Event('pagehide')); + + expect(interruptPosts()).toHaveLength(0); + // Attribution never intervenes, so the stream is left to close itself. + expect(controller.signal.aborted).toBe(false); + controller.abort(); + await streaming; + }); + + it('a page-hide mid-stream still signals navigated_away', async () => { + chatStateService.streamingSessionIds.mockReturnValue(['s1']); + openStream('event: message_start\ndata: {"role":"assistant"}\n\n'); + + const streaming = service.sendChatRequest({ session_id: 's1', message: 'hi' }); + await vi.waitFor(() => + expect(parser.parseEventSourceMessage).toHaveBeenCalledWith( + 's1', + 'message_start', + expect.anything(), + 'stream-1', + ), + ); + + window.dispatchEvent(new Event('pagehide')); + + const posts = interruptPosts(); + expect(posts).toHaveLength(1); + expect(String(posts[0][0])).toContain('/sessions/s1/interrupt'); + expect(JSON.parse(String(posts[0][1].body))).toEqual({ reason: 'navigated_away' }); + expect(posts[0][1].keepalive).toBe(true); + controller.abort(); + await streaming; + }); + + it('runs the title fallback the skipped onclose would have, and no interrupted-turn cost refresh', async () => { + vi.useFakeTimers(); + try { + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(null, { status: 204 })); + parser.hasReceivedDone.mockReturnValue(true); + const sessionSvc = TestBed.inject(SessionService) as any; + sessionSvc.isNewSession.mockReturnValue(true); + sessionSvc.applyServerTitle = vi.fn(); + sessionSvc.getSessionMetadata.mockResolvedValue({ title: 'Finished Turn Title' }); + + service.cancelChatRequest('s1'); + await vi.runAllTimersAsync(); + + // Aborting skips onclose, whose job on a first turn is to pull a + // title that `session_title` may not have delivered yet. + expect(sessionSvc.applyServerTitle).toHaveBeenCalledWith('s1', 'Finished Turn Title'); + expect(sessionSvc.getSessionMetadata).toHaveBeenCalledTimes(1); + expect(chatStateService.seedSessionAggregates).not.toHaveBeenCalled(); + } finally { + vi.useRealTimers(); + } + }); + }); + + describe('title fallback polling', () => { + // A stream that outran title generation closes before Nova Micro answers, + // so the first read sees the placeholder. Short retries pick the title up + // as soon as it is written instead of after one fixed 1.5s wait. + let sessionSvc: any; + + beforeEach(() => { + vi.useFakeTimers(); + sessionSvc = TestBed.inject(SessionService) as any; + sessionSvc.isNewSession.mockReturnValue(true); + sessionSvc.applyServerTitle = vi.fn(); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('applies the title on the first retry that sees it', async () => { + sessionSvc.getSessionMetadata = vi + .fn() + .mockResolvedValueOnce({ title: 'New Conversation' }) + .mockResolvedValueOnce({ title: 'New Conversation' }) + .mockResolvedValue({ title: 'Biology Syllabus' }); + + void (service as any).refreshTitleFromServer('s1'); + await vi.advanceTimersByTimeAsync(800); + + expect(sessionSvc.getSessionMetadata).toHaveBeenCalledTimes(3); + expect(sessionSvc.applyServerTitle).toHaveBeenCalledWith('s1', 'Biology Syllabus'); + await vi.runAllTimersAsync(); + expect(sessionSvc.getSessionMetadata).toHaveBeenCalledTimes(3); + }); + + it('gives up after a bounded number of retries', async () => { + sessionSvc.getSessionMetadata = vi.fn().mockResolvedValue({ title: 'New Conversation' }); + + void (service as any).refreshTitleFromServer('s1'); + await vi.runAllTimersAsync(); + + expect(sessionSvc.getSessionMetadata).toHaveBeenCalledTimes(6); + expect(sessionSvc.applyServerTitle).not.toHaveBeenCalled(); + }); + + it('stops polling once the title arrived another way', async () => { + sessionSvc.getSessionMetadata = vi.fn().mockResolvedValue({ title: 'New Conversation' }); + + void (service as any).refreshTitleFromServer('s1'); + await vi.advanceTimersByTimeAsync(0); + // A late `session_title` event applied the title and cleared the flag. + sessionSvc.isNewSession.mockReturnValue(false); + await vi.runAllTimersAsync(); + + expect(sessionSvc.getSessionMetadata).toHaveBeenCalledTimes(1); + }); + }); + describe('page-departure attribution', () => { // A refresh / tab close / navigation is the one interruption cause only // the browser witnesses. Unattested it lands server-side as diff --git a/frontend/ai.client/src/app/session/services/chat/chat-http.service.ts b/frontend/ai.client/src/app/session/services/chat/chat-http.service.ts index 57decfad3..4ebc472f2 100644 --- a/frontend/ai.client/src/app/session/services/chat/chat-http.service.ts +++ b/frontend/ai.client/src/app/session/services/chat/chat-http.service.ts @@ -11,6 +11,15 @@ import { SessionService } from '../session/session.service'; import { ErrorService } from '../../../services/error/error.service'; import { isPreviewSession } from '../../../shared/constants/session.constants'; +/** + * Title fallback polling. A stream that outruns title generation closes a few + * hundred ms before Nova Micro answers, so one early read usually sees the + * placeholder; short, bounded retries pick the title up as soon as it lands + * instead of after a single fixed wait. 5 x 400ms keeps the old ~1.5s window. + */ +const TITLE_REFRESH_RETRY_MS = 400; +const TITLE_REFRESH_MAX_RETRIES = 5; + class RetriableError extends Error { constructor(message?: string) { super(message); @@ -350,9 +359,25 @@ export class ChatHttpService { // that will never be loaded again. Abort the transport and tear down // locally; that is the whole of "stop" for a session nobody persists. if (isPreviewSession(sessionId)) { - this.chatStateService.abortRequest(sessionId); - this.messageMapService.endStreaming(sessionId); - this.chatStateService.setChatLoading(sessionId, false); + this.stopLocally(sessionId); + return; + } + + // The server already sent `done`: the turn finished, and the socket is + // only open because the transport's close hasn't landed yet (widest on a + // first turn, where `session_title` can trail `done`). The Stop button is + // still showing because loading clears on close, so a click here is + // real — but there is nothing left to interrupt. Signalling + // `user_stopped` would stamp a false "interrupted" marker on a complete + // answer (and a false interruption note on the next prompt), so just + // close the transport. Aborting skips `onclose`, so run its title + // fallback here; the aggregates re-fetch below is unneeded, because the + // turn's `metadata` event arrived before `done`. + if (this.streamParserService.hasReceivedDone(sessionId)) { + this.stopLocally(sessionId); + if (this.sessionService.isNewSession(sessionId)) { + void this.refreshTitleFromServer(sessionId); + } return; } @@ -370,9 +395,7 @@ export class ChatHttpService { // refresh-survival source of truth. this.chatStateService.setLastTurnInterrupted(sessionId, true, 'user_stopped'); - this.chatStateService.abortRequest(sessionId); - this.messageMapService.endStreaming(sessionId); - this.chatStateService.setChatLoading(sessionId, false); + this.stopLocally(sessionId); // Aborting the fetch cut the socket before the stream's terminal // `metadata` SSE (usage / cost / context) could arrive, so the session @@ -385,6 +408,17 @@ export class ChatHttpService { setTimeout(() => void this.refreshAggregatesAfterStop(sessionId), 900); } + /** + * Abort a session's transport and tear its streaming state down here. + * fetch-event-source calls neither `onclose` nor `onerror` on abort, so + * the stream's own `finalizeStream` never runs for a stopped stream. + */ + private stopLocally(sessionId: string): void { + this.chatStateService.abortRequest(sessionId); + this.messageMapService.endStreaming(sessionId); + this.chatStateService.setChatLoading(sessionId, false); + } + /** * Re-seed a session's cost/context badge after a Stop, once the backend's * interruption teardown has persisted the partial turn's metadata (which @@ -465,6 +499,12 @@ export class ChatHttpService { if (typeof window === 'undefined') return; const onPageHide = () => { for (const sessionId of this.chatStateService.streamingSessionIds()) { + // A session stays "streaming" until the transport closes, which can + // trail `done`. A departure in that window interrupts nothing, and + // the backend's lease gate can't be relied on to drop it — the lease + // is released in the stream generator's `finally`, which can itself + // come after `done` — so skip it here, as the Stop path does. + if (this.streamParserService.hasReceivedDone(sessionId)) continue; this.signalInterrupt(sessionId, 'navigated_away'); } }; @@ -484,18 +524,22 @@ export class ChatHttpService { * and is normally PUSHED mid-stream as a `session_title` SSE event; this * fetch only runs when the stream closed while the session still looked * new (see onclose). On a "New Conversation" placeholder — generation - * still in flight or failed — we retry once after a short delay before - * giving up. + * still in flight or failed — we poll a few more times at a short interval + * before giving up, stopping early if the title arrived another way. */ - private async refreshTitleFromServer(sessionId: string, retried = false): Promise<void> { + private async refreshTitleFromServer(sessionId: string, attempt = 0): Promise<void> { try { const metadata = await this.sessionService.getSessionMetadata(sessionId); if (metadata.title && metadata.title !== 'New Conversation') { this.sessionService.applyServerTitle(sessionId, metadata.title); return; } - if (!retried) { - setTimeout(() => this.refreshTitleFromServer(sessionId, true), 1500); + if (attempt < TITLE_REFRESH_MAX_RETRIES) { + setTimeout(() => { + if (this.sessionService.isNewSession(sessionId)) { + void this.refreshTitleFromServer(sessionId, attempt + 1); + } + }, TITLE_REFRESH_RETRY_MS); } } catch (error) { console.error('Failed to refresh session title:', error); diff --git a/frontend/ai.client/src/app/session/services/chat/stream-parser.service.spec.ts b/frontend/ai.client/src/app/session/services/chat/stream-parser.service.spec.ts index ca2ee1c32..6b9cb2729 100644 --- a/frontend/ai.client/src/app/session/services/chat/stream-parser.service.spec.ts +++ b/frontend/ai.client/src/app/session/services/chat/stream-parser.service.spec.ts @@ -660,3 +660,78 @@ describe('StreamParserService - session_title events', () => { expect(applyServerTitle).not.toHaveBeenCalled(); }); }); + +describe('StreamParserService - hasReceivedDone (Stop guard)', () => { + // The Stop path reads this to decide whether a click can still interrupt + // anything. It must mean exactly "the server sent this stream's `done`": + // per session, per stream, and not tripped by a client-side parse error. + let service: StreamParserService; + + beforeEach(() => { + TestBed.resetTestingModule(); + TestBed.configureTestingModule({ + providers: [ + StreamParserService, + ChatStateService, + ErrorService, + QuotaWarningService, + { provide: SessionService, useValue: { applyServerTitle: vi.fn() } }, + ], + }); + service = TestBed.inject(StreamParserService); + service.reset('s1'); + }); + + afterEach(() => { + TestBed.resetTestingModule(); + }); + + it('is false mid-stream and true once done arrives', () => { + service.parseEventSourceMessage('s1', 'message_start', { role: 'assistant' }); + expect(service.hasReceivedDone('s1')).toBe(false); + + service.parseEventSourceMessage('s1', 'done', null); + expect(service.hasReceivedDone('s1')).toBe(true); + }); + + it('resets to false when the session starts a new stream', () => { + service.parseEventSourceMessage('s1', 'done', null); + service.reset('s1'); + expect(service.hasReceivedDone('s1')).toBe(false); + }); + + it('is scoped to its session', () => { + service.reset('s2'); + service.parseEventSourceMessage('s1', 'done', null); + expect(service.hasReceivedDone('s2')).toBe(false); + expect(service.hasReceivedDone('unknown')).toBe(false); + }); + + it('ignores a done from a superseded stream', () => { + const oldStreamId = service.getCurrentStreamId('s1'); + service.reset('s1'); + service.parseEventSourceMessage('s1', 'done', null, oldStreamId); + expect(service.hasReceivedDone('s1')).toBe(false); + }); + + it('is not set by a client parse error — the server turn may still be running', () => { + // A delta with no active message puts the parser in its Error state. + service.parseEventSourceMessage('s1', 'content_block_delta', { + contentBlockIndex: 0, + type: 'text', + text: 'x', + }); + expect(service.errorFor('s1')()).not.toBeNull(); + expect(service.hasReceivedDone('s1')).toBe(false); + }); + + it('still records done after a parse error, even though the state gate drops the event', () => { + service.parseEventSourceMessage('s1', 'content_block_delta', { + contentBlockIndex: 0, + type: 'text', + text: 'x', + }); + service.parseEventSourceMessage('s1', 'done', null); + expect(service.hasReceivedDone('s1')).toBe(true); + }); +}); diff --git a/frontend/ai.client/src/app/session/services/chat/stream-parser.service.ts b/frontend/ai.client/src/app/session/services/chat/stream-parser.service.ts index a2fde416e..dbd7522fb 100644 --- a/frontend/ai.client/src/app/session/services/chat/stream-parser.service.ts +++ b/frontend/ai.client/src/app/session/services/chat/stream-parser.service.ts @@ -136,9 +136,23 @@ interface ParserSessionState { /** Error state */ error: WritableSignal<string | null>; - /** Stream completion state */ + /** + * Parser-side completion: true once `done` was handled OR the parser gave + * up on the stream (`setError`). Drives rendering only — a client parse + * error says nothing about whether the server's turn is still running. + */ isStreamComplete: WritableSignal<boolean>; + /** + * True once the server sent this stream's `done` frame — the turn is over + * server-side, whatever the parser made of it. Recorded before the state + * gate, so a parser already in `Error` still learns the turn ended. + * Read by the Stop path: a Stop after this point must not mark the + * finished turn interrupted. Reset with the rest of the state on every + * new stream. + */ + doneReceived: boolean; + /** Metadata (usage, metrics) from the stream */ metadata: WritableSignal<MetadataEvent | null>; @@ -198,7 +212,6 @@ export class StreamParserService { private readonly lastEventAtCache = new Map<string, Signal<number>>(); private readonly citationsCache = new Map<string, Signal<Citation[]>>(); private readonly errorCache = new Map<string, Signal<string | null>>(); - private readonly isStreamCompleteCache = new Map<string, Signal<boolean>>(); // ========================================================================= // Public API @@ -248,9 +261,16 @@ export class StreamParserService { return this.cachedAccessor(this.errorCache, sessionId, (state) => state.error(), null); } - /** Stream completion state for a session. */ - isStreamCompleteFor(sessionId: string): Signal<boolean> { - return this.cachedAccessor(this.isStreamCompleteCache, sessionId, (state) => state.isStreamComplete(), false); + /** + * Whether a session's CURRENT stream has received the server's `done`. + * + * Scoped to the stream the parser was last reset for, so a new turn reads + * false until its own `done`. Deliberately not derived from + * `isStreamComplete`, which a client-side parse error also sets while the + * server turn keeps running — a Stop then is a real interruption. + */ + hasReceivedDone(sessionId: string): boolean { + return this.states().get(sessionId)?.doneReceived ?? false; } /** @@ -305,6 +325,12 @@ export class StreamParserService { // its replacement looking alive. state.lastEventAt.set(Date.now()); + // Like liveness, the end of the server's turn is a transport fact the + // state gate below must not swallow (a parser in `Error` drops `done`). + if (event === 'done') { + state.doneReceived = true; + } + // Validate inputs if (!event || typeof event !== 'string') { this.setError(state, 'parseEventSourceMessage: event must be a non-empty string'); @@ -429,6 +455,7 @@ export class StreamParserService { lastEventAt: signal<number>(Date.now()), error: signal<string | null>(null), isStreamComplete, + doneReceived: false, metadata: signal<MetadataEvent | null>(null), pendingCitations: signal<Citation[]>([]), currentMessage: computed<Message | null>(() => { @@ -1019,6 +1046,7 @@ export class StreamParserService { private handleDone(state: ParserSessionState): void { this.finalizeCurrentMessage(state); state.isStreamComplete.set(true); + state.doneReceived = true; state.modelRetry.set(null); // "Using list_courses" on a finished turn is a lie, not a stale nicety. // Durations and summaries already recorded are untouched. diff --git a/frontend/ai.client/src/app/session/services/model/model.service.spec.ts b/frontend/ai.client/src/app/session/services/model/model.service.spec.ts index e46fb3af1..298685a3a 100644 --- a/frontend/ai.client/src/app/session/services/model/model.service.spec.ts +++ b/frontend/ai.client/src/app/session/services/model/model.service.spec.ts @@ -22,7 +22,11 @@ describe('ModelService', () => { let sessionStore: Record<string, string> = {}; - async function setup() { + // `beforeEach(setup)` hands setup the test context, so the response is a + // separate entry point rather than an optional parameter. + const setup = () => setupWith(mockResponse); + + async function setupWith(response: { models: ManagedModel[]; totalCount: number }) { sessionStore = {}; vi.stubGlobal('sessionStorage', { getItem: vi.fn((k: string) => sessionStore[k] ?? null), @@ -48,7 +52,7 @@ describe('ModelService', () => { httpMock = TestBed.inject(HttpTestingController); await vi.waitFor(() => { - httpMock.expectOne('http://localhost:8000/models').flush(mockResponse); + httpMock.expectOne('http://localhost:8000/models').flush(response); }); } @@ -61,6 +65,24 @@ describe('ModelService', () => { TestBed.resetTestingModule(); }); + describe('speech-to-speech rows', () => { + it('never reach the picker, even when enabled', async () => { + const sonic: ManagedModel = { + ...mockModels[0], + id: 'm-sonic', + modelId: 'amazon.nova-2-sonic-v1:0', + modelName: 'Nova 2 Sonic', + inputModalities: ['SPEECH', 'TEXT'], + outputModalities: ['SPEECH', 'TEXT'], + enabled: true, + }; + await setupWith({ models: [...mockModels, sonic], totalCount: 3 }); + await vi.waitFor(() => { + expect(service.availableModels().map((m) => m.modelId)).toEqual(['claude-haiku', 'claude-sonnet']); + }); + }); + }); + describe('retired inference-param overrides', () => { // The drawer's Advanced form is gone (step 3 of // docs/specs/customize-surface.md), but this store is sessionStorage — a tab diff --git a/frontend/ai.client/src/app/session/services/model/model.service.ts b/frontend/ai.client/src/app/session/services/model/model.service.ts index f2ed982ac..f077f2a69 100644 --- a/frontend/ai.client/src/app/session/services/model/model.service.ts +++ b/frontend/ai.client/src/app/session/services/model/model.service.ts @@ -17,6 +17,11 @@ interface ManagedModelsListResponse { totalCount: number; } +/** A speech-to-speech row: voice mode's model, never a chat one. */ +export function isSpeechModel(model: { outputModalities?: string[] }): boolean { + return (model.outputModalities ?? []).some((m) => m.toUpperCase() === 'SPEECH'); +} + @Injectable({ providedIn: 'root' }) @@ -227,9 +232,11 @@ export class ModelService { ); // Filter to enabled models a user may select. Retired rows are held apart: - // never offered, only followed to their successor. + // never offered, only followed to their successor. Speech-to-speech rows + // (Nova 2 Sonic) are catalogued so voice sessions price, but they answer + // no chat API — the backend already omits them; this is the belt. const enabledModels = response.models.filter( - model => model.enabled && model.status !== 'retired', + model => model.enabled && model.status !== 'retired' && !isSpeechModel(model), ); this.retiredModels.set(response.models.filter(model => model.status === 'retired')); diff --git a/frontend/ai.client/src/app/session/services/voice/audio-recorder.service.ts b/frontend/ai.client/src/app/session/services/voice/audio-recorder.service.ts index 35174de52..f7c3b9bd7 100644 --- a/frontend/ai.client/src/app/session/services/voice/audio-recorder.service.ts +++ b/frontend/ai.client/src/app/session/services/voice/audio-recorder.service.ts @@ -2,6 +2,19 @@ import { Injectable, signal } from '@angular/core'; import { float32ToPcm16, pcm16ToBase64, resampleLinear } from './pcm-utils'; import { VOICE_SAMPLE_RATE, SAMPLES_PER_CHUNK } from './voice.config'; +/** A microphone the browser will let us capture from. */ +export interface AudioInputDevice { + deviceId: string; + /** Empty until the user has granted microphone access once. */ + label: string; +} + +/** + * Per-device, not per-account: a device id is only meaningful in the browser + * that enumerated it, so it lives in localStorage like the view-mode prefs. + */ +const INPUT_DEVICE_KEY = 'voice-input-device'; + /** * Audio capture service using Web Audio API. * @@ -19,6 +32,13 @@ export class AudioRecorderService { readonly isRecording = this._isRecording.asReadonly(); readonly isSupported = this._isSupported.asReadonly(); + /** + * The microphone to capture from, or null for the browser's default. Shared + * by voice mode and dictation, which both record through this service. + */ + private readonly _inputDeviceId = signal<string | null>(this.loadInputDeviceId()); + readonly inputDeviceId = this._inputDeviceId.asReadonly(); + private audioContext: AudioContext | null = null; private mediaStream: MediaStream | null = null; private sourceNode: MediaStreamAudioSourceNode | null = null; @@ -45,6 +65,37 @@ export class AudioRecorderService { return hasGetUserMedia && hasAudioContext; } + /** Choose the microphone for the next capture; null returns to the browser default. */ + setInputDevice(deviceId: string | null): void { + this._inputDeviceId.set(deviceId); + try { + if (deviceId) { + localStorage.setItem(INPUT_DEVICE_KEY, deviceId); + } else { + localStorage.removeItem(INPUT_DEVICE_KEY); + } + } catch { + // Private mode or blocked storage: the choice still applies this visit. + } + } + + /** + * The microphones this browser can see. Labels are blank until the user has + * granted microphone access once, which is why the picker says so instead + * of listing sixteen "Microphone" rows. + */ + async listInputDevices(): Promise<AudioInputDevice[]> { + if (typeof navigator.mediaDevices?.enumerateDevices !== 'function') return []; + try { + const devices = await navigator.mediaDevices.enumerateDevices(); + return devices + .filter((d) => d.kind === 'audioinput' && d.deviceId) + .map((d) => ({ deviceId: d.deviceId, label: d.label })); + } catch { + return []; + } + } + /** * Start capturing audio from the microphone. * Requests microphone permission if not already granted. @@ -56,15 +107,7 @@ export class AudioRecorderService { } try { - this.mediaStream = await navigator.mediaDevices.getUserMedia({ - audio: { - sampleRate: { ideal: VOICE_SAMPLE_RATE }, - channelCount: { exact: 1 }, - echoCancellation: true, - noiseSuppression: true, - autoGainControl: true, - }, - }); + this.mediaStream = await this.openMicrophone(); const AudioContextClass = window.AudioContext || (window as any).webkitAudioContext; this.audioContext = new AudioContextClass({ sampleRate: VOICE_SAMPLE_RATE }); @@ -90,6 +133,44 @@ export class AudioRecorderService { } } + /** + * Open the chosen microphone, falling back to the default when it is gone. + * + * A stored device id outlives the device: unplug the headset and `exact` + * throws OverconstrainedError, which would turn every later voice session + * into an error toast until the user found the picker. The fallback clears + * the stale choice so the picker shows the truth. + */ + private async openMicrophone(): Promise<MediaStream> { + const base: MediaTrackConstraints = { + sampleRate: { ideal: VOICE_SAMPLE_RATE }, + channelCount: { exact: 1 }, + echoCancellation: true, + noiseSuppression: true, + autoGainControl: true, + }; + const deviceId = this._inputDeviceId(); + if (deviceId) { + try { + return await navigator.mediaDevices.getUserMedia({ + audio: { ...base, deviceId: { exact: deviceId } }, + }); + } catch (err) { + if (!(err instanceof DOMException) || err.name !== 'OverconstrainedError') throw err; + this.setInputDevice(null); + } + } + return navigator.mediaDevices.getUserMedia({ audio: base }); + } + + private loadInputDeviceId(): string | null { + try { + return localStorage.getItem(INPUT_DEVICE_KEY); + } catch { + return null; + } + } + /** Stop capturing and release resources. */ async stop(): Promise<void> { if (!this._isRecording()) return; diff --git a/frontend/ai.client/src/app/session/services/voice/index.ts b/frontend/ai.client/src/app/session/services/voice/index.ts index 7b63c005e..148fb512c 100644 --- a/frontend/ai.client/src/app/session/services/voice/index.ts +++ b/frontend/ai.client/src/app/session/services/voice/index.ts @@ -1,5 +1,6 @@ -export { AudioRecorderService } from './audio-recorder.service'; +export { AudioRecorderService, type AudioInputDevice } from './audio-recorder.service'; export { AudioPlayerService } from './audio-player.service'; export { VoiceChatService, type VoiceStatus, type VoiceTranscriptEntry, type VoiceTokenUsage } from './voice-chat.service'; export * from './pcm-utils'; export * from './voice.config'; +export * from './voice-catalog'; diff --git a/frontend/ai.client/src/app/session/services/voice/voice-catalog.spec.ts b/frontend/ai.client/src/app/session/services/voice/voice-catalog.spec.ts new file mode 100644 index 000000000..fa6895a5a --- /dev/null +++ b/frontend/ai.client/src/app/session/services/voice/voice-catalog.spec.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from 'vitest'; +import { + DEFAULT_VOICE_ID, + NOVA_SONIC_VOICES, + VOICE_GROUPS, + findVoice, + isVoiceId, + voiceDescription, +} from './voice-catalog'; + +// Every voiceId the Nova 2 user guide lists for `audioOutputConfiguration` +// (sonic-input-events, verified 2026-09-30). The backend validates against +// the same list (`apis/shared/voice_catalog.py`); the two must agree. +const PUBLISHED_VOICE_IDS = [ + 'matthew', 'tiffany', 'amy', 'olivia', 'lupe', 'carlos', 'ambre', 'florian', + 'lennart', 'beatrice', 'lorenzo', 'tina', 'carolina', 'leo', 'kiara', 'arjun', +].sort(); + +describe('voice catalog', () => { + it('matches the published voice ids exactly', () => { + expect(NOVA_SONIC_VOICES.map((v) => v.id).sort()).toEqual(PUBLISHED_VOICE_IDS); + }); + + it('defaults to a polyglot voice, so a language switch keeps the voice', () => { + expect(findVoice(DEFAULT_VOICE_ID)?.polyglot).toBe(true); + expect(NOVA_SONIC_VOICES.filter((v) => v.polyglot).map((v) => v.id).sort()).toEqual(['matthew', 'tiffany']); + }); + + it('groups by language in catalog order, English first', () => { + expect(VOICE_GROUPS[0].language).toBe('English (US)'); + expect(VOICE_GROUPS[0].voices.map((v) => v.id)).toEqual(['tiffany', 'matthew']); + // Every voice lands in exactly one group. + const grouped = VOICE_GROUPS.flatMap((g) => g.voices.map((v) => v.id)); + expect(grouped.sort()).toEqual(PUBLISHED_VOICE_IDS); + }); + + it('looks voices up case- and whitespace-insensitively, and refuses the rest', () => { + expect(findVoice(' Matthew ')?.id).toBe('matthew'); + expect(isVoiceId('AMY')).toBe(true); + expect(findVoice('siri')).toBeUndefined(); + expect(isVoiceId('')).toBe(false); + expect(isVoiceId(null)).toBe(false); + }); + + it('describes a polyglot voice as speaking every language', () => { + expect(voiceDescription(findVoice('tiffany')!)).toBe('Feminine · Speaks every language'); + expect(voiceDescription(findVoice('leo')!)).toBe('Masculine'); + }); +}); diff --git a/frontend/ai.client/src/app/session/services/voice/voice-catalog.ts b/frontend/ai.client/src/app/session/services/voice/voice-catalog.ts new file mode 100644 index 000000000..a4040d9ca --- /dev/null +++ b/frontend/ai.client/src/app/session/services/voice/voice-catalog.ts @@ -0,0 +1,89 @@ +/** + * The Nova 2 Sonic voice catalog. + * + * Mirrors `apis/shared/voice_catalog.py` — the backend validates whatever id + * the SPA sends against the same list, so the two must agree. Source: the + * Nova 2 user guide, "Language support and multilingual capabilities" + * (verified 2026-09-30). + * + * `tiffany` and `matthew` are the polyglot voices: they speak every supported + * language, which is why `tiffany` is the default — a user who switches + * language mid-conversation keeps the same voice. Every other voice is tuned + * to one language and answers in that accent whatever the user speaks. + */ + +export type VoiceGender = 'feminine' | 'masculine'; + +export interface NovaSonicVoice { + /** The `voiceId` Bedrock accepts in `promptStart`. */ + readonly id: string; + /** Display name. */ + readonly name: string; + /** The language the voice is tuned to. */ + readonly language: string; + /** BCP-47 locale, for the group header. */ + readonly locale: string; + /** How the voice sounds; presentation only. */ + readonly gender: VoiceGender; + /** Speaks every supported language, not just its own. */ + readonly polyglot?: boolean; +} + +export const DEFAULT_VOICE_ID = 'tiffany'; + +export const NOVA_SONIC_VOICES: readonly NovaSonicVoice[] = [ + { id: 'tiffany', name: 'Tiffany', language: 'English (US)', locale: 'en-US', gender: 'feminine', polyglot: true }, + { id: 'matthew', name: 'Matthew', language: 'English (US)', locale: 'en-US', gender: 'masculine', polyglot: true }, + { id: 'amy', name: 'Amy', language: 'English (UK)', locale: 'en-GB', gender: 'feminine' }, + { id: 'olivia', name: 'Olivia', language: 'English (Australia)', locale: 'en-AU', gender: 'feminine' }, + { id: 'kiara', name: 'Kiara', language: 'English (India) / Hindi', locale: 'en-IN', gender: 'feminine' }, + { id: 'arjun', name: 'Arjun', language: 'English (India) / Hindi', locale: 'en-IN', gender: 'masculine' }, + { id: 'ambre', name: 'Ambre', language: 'French', locale: 'fr-FR', gender: 'feminine' }, + { id: 'florian', name: 'Florian', language: 'French', locale: 'fr-FR', gender: 'masculine' }, + { id: 'beatrice', name: 'Beatrice', language: 'Italian', locale: 'it-IT', gender: 'feminine' }, + { id: 'lorenzo', name: 'Lorenzo', language: 'Italian', locale: 'it-IT', gender: 'masculine' }, + { id: 'tina', name: 'Tina', language: 'German', locale: 'de-DE', gender: 'feminine' }, + { id: 'lennart', name: 'Lennart', language: 'German', locale: 'de-DE', gender: 'masculine' }, + { id: 'lupe', name: 'Lupe', language: 'Spanish (US)', locale: 'es-US', gender: 'feminine' }, + { id: 'carlos', name: 'Carlos', language: 'Spanish (US)', locale: 'es-US', gender: 'masculine' }, + { id: 'carolina', name: 'Carolina', language: 'Portuguese (Brazil)', locale: 'pt-BR', gender: 'feminine' }, + { id: 'leo', name: 'Leo', language: 'Portuguese (Brazil)', locale: 'pt-BR', gender: 'masculine' }, +]; + +/** One language's voices, in catalog order, for a grouped picker. */ +export interface VoiceGroup { + readonly language: string; + readonly voices: readonly NovaSonicVoice[]; +} + +/** The catalog grouped by language, in catalog order (English first). */ +export const VOICE_GROUPS: readonly VoiceGroup[] = (() => { + const groups: VoiceGroup[] = []; + for (const voice of NOVA_SONIC_VOICES) { + const group = groups.find((g) => g.language === voice.language); + if (group) { + (group.voices as NovaSonicVoice[]).push(voice); + } else { + groups.push({ language: voice.language, voices: [voice] }); + } + } + return groups; +})(); + +/** Look a voice up by id; `undefined` for anything not in the catalog. */ +export function findVoice(id: string | null | undefined): NovaSonicVoice | undefined { + if (!id) return undefined; + const key = id.trim().toLowerCase(); + return NOVA_SONIC_VOICES.find((v) => v.id === key); +} + +/** Whether `id` names a voice Bedrock will accept. */ +export function isVoiceId(id: string | null | undefined): boolean { + return findVoice(id) !== undefined; +} + +/** A short line for the picker: "Feminine · Speaks every language" / "Masculine". */ +export function voiceDescription(voice: NovaSonicVoice): string { + const gender = voice.gender === 'feminine' ? 'Feminine' : 'Masculine'; + return voice.polyglot ? `${gender} · Speaks every language` : gender; +} diff --git a/frontend/ai.client/src/app/session/services/voice/voice-chat.service.ts b/frontend/ai.client/src/app/session/services/voice/voice-chat.service.ts index 8825ae703..b2ff8d550 100644 --- a/frontend/ai.client/src/app/session/services/voice/voice-chat.service.ts +++ b/frontend/ai.client/src/app/session/services/voice/voice-chat.service.ts @@ -1,9 +1,11 @@ -import { Injectable, signal, computed, inject, OnDestroy } from '@angular/core'; +import { Injectable, signal, computed, effect, inject, untracked, OnDestroy } from '@angular/core'; import { v4 as uuidv4 } from 'uuid'; import { ConfigService } from '../../../services/config.service'; import { AudioRecorderService } from './audio-recorder.service'; import { AudioPlayerService } from './audio-player.service'; import { VoiceTicketService } from './voice-ticket.service'; +import { UserSettingsService } from '../../../services/user-settings.service'; +import { DEFAULT_VOICE_ID, findVoice, type NovaSonicVoice } from './voice-catalog'; import { Message } from '../models/message.model'; import { IDLE_TIMEOUT_MS, @@ -52,6 +54,24 @@ export class VoiceChatService implements OnDestroy { private readonly player = inject(AudioPlayerService); private readonly ticketService = inject(VoiceTicketService); private readonly config = inject(ConfigService); + private readonly userSettings = inject(UserSettingsService); + + // --- Voice choice --- + /** + * The Nova 2 Sonic voice the next session speaks with. Seeded from the + * user's saved settings once they load; a change is persisted silently, so + * a storage hiccup never interrupts starting the conversation. It applies + * on the next connect — a live session keeps the voice it opened with, + * because the voice is fixed at Bedrock's `promptStart`. + */ + private readonly _voiceId = signal<string>(DEFAULT_VOICE_ID); + readonly voiceId = this._voiceId.asReadonly(); + readonly voice = computed<NovaSonicVoice>( + () => findVoice(this._voiceId()) ?? (findVoice(DEFAULT_VOICE_ID) as NovaSonicVoice), + ); + /** The voice the live session actually opened with (the server confirms it). */ + private readonly _activeVoiceId = signal<string | null>(null); + readonly activeVoiceId = this._activeVoiceId.asReadonly(); // --- State signals --- private readonly _status = signal<VoiceStatus>('idle'); @@ -129,6 +149,32 @@ export class VoiceChatService implements OnDestroy { private sessionId: string | null = null; private idleTimer: ReturnType<typeof setTimeout> | null = null; + constructor() { + // Seed the voice from the saved settings. The resource is shared with + // ModelService's default-model read, so this costs no extra request. + effect(() => { + const saved = this.userSettings.settingsResource.value()?.voiceId; + if (saved && findVoice(saved)) { + untracked(() => this._voiceId.set(saved)); + } + }); + } + + /** + * Choose the voice for the next session and remember it on the account. + * Unknown ids are ignored rather than sent: the backend would refuse them. + */ + async setVoice(voiceId: string): Promise<void> { + const voice = findVoice(voiceId); + if (!voice || voice.id === this._voiceId()) return; + this._voiceId.set(voice.id); + try { + await this.userSettings.updateSettings({ voiceId: voice.id }, { silent: true }); + } catch { + // Kept in memory for this visit; the next load falls back to the saved one. + } + } + /** * Connect to the voice endpoint and start recording. * If no sessionId is provided, one is auto-generated. @@ -302,6 +348,7 @@ export class VoiceChatService implements OnDestroy { this.sendMessage({ type: 'config', session_id: sessionId, + voice: this._voiceId(), }); resolve(); }; @@ -346,7 +393,9 @@ export class VoiceChatService implements OnDestroy { switch (type) { case 'bidi_connection_start': - // Connection confirmed + // Connection confirmed. `voice` is the id the server actually used — + // it falls back to the default for an id it does not know. + this._activeVoiceId.set(typeof data['voice'] === 'string' ? (data['voice'] as string) : this._voiceId()); break; case 'bidi_audio_stream': @@ -615,6 +664,7 @@ export class VoiceChatService implements OnDestroy { private cleanupAll(): void { this.stopRevealTimer(); + this._activeVoiceId.set(null); if (this.idleTimer) { clearTimeout(this.idleTimer); diff --git a/frontend/ai.client/src/styles.css b/frontend/ai.client/src/styles.css index b9481f6ed..679d2e516 100644 --- a/frontend/ai.client/src/styles.css +++ b/frontend/ai.client/src/styles.css @@ -263,6 +263,24 @@ html.dark .message-block { a:hover { color: var(--color-primary-300); } + + /* Weight alone barely separates bold from body text on a dark surface, + and the assistant bubble's gray-200 body sits too close to white to + leave room above it. So response prose steps down to gray-300 while + headings and bold step up to white. Inside a link, bold keeps the + link's color. */ + &.text-block { + color: var(--color-gray-300); + } + + h1, h2, h3, h4, h5, h6, + strong { + color: var(--color-white); + } + + a strong { + color: inherit; + } } diff --git a/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts b/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts index 02e4fd4f0..af507797f 100644 --- a/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts +++ b/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts @@ -156,6 +156,34 @@ export class TurnLatencyObservabilityConstruct extends Construct { }), ); + // Row 1b — past the prelude to the first model output (`turn_first_token`, + // docs/specs/turn-path-ttft.md P1b). `FirstTokenMs` runs from handler + // entry, so it is the prelude plus the head of turn, the pre-model work + // (LTM retrieval, the user message's append) and the model's own time to + // first token — the server's whole share of the wait. It rides its own EMF + // record, so nothing here changes `PreludeTotalMs` or the unaccounted query. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'First token (handler entry → first model output)', + left: percentiles('FirstTokenMs'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'After the prelude (p90) — head of turn, pre-model, model', + left: [ + stage('HeadOfTurnMs', 'p90', 'head of turn'), + stage('HeadOfTurnHistoryCountMs', 'p90', 'history count (ListEvents)'), + stage('PreModelMs', 'p90', 'pre-model (append + LTM)'), + stage('ModelMs', 'p90', 'model to first token'), + ], + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + ); + // Row 2 — which sub-stage owns the preamble. Split by statistic rather than // stacking fifteen lines on one axis, which is a picture nobody reads. this.dashboard.addWidgets( @@ -232,6 +260,25 @@ export class TurnLatencyObservabilityConstruct extends Construct { }), ); + // Row 3c — inside `tools`, the largest cold sub-stage (P1a in + // docs/specs/turn-path-ttft.md). `AgentBuildToolsMs` above is now their + // sum. Which MCP server owned `mcp` is a log property (`mcpServers` on + // the `turn_prelude` line), not a metric — it has no percentile. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Inside tools (p90) — filter, gateway, MCP load + pre-flight, injected tools', + left: [ + stage('AgentBuildToolsFilterMs', 'p90', 'filter'), + stage('AgentBuildToolsGatewayMs', 'p90', 'gateway'), + stage('AgentBuildToolsMcpMs', 'p90', 'external MCP (catalog + pre-flight)'), + stage('AgentBuildToolsExtraMs', 'p90', 'injected tools'), + ], + leftYAxis: { min: 0 }, + width: 24, + height: 6, + }), + ); + // Row 4 — the slicing that dimensions would have done, done in Logs // Insights instead. A resume skips most of the preamble, so mixing the two // populations is what would make a traffic-mix shift look like a latency diff --git a/infrastructure/package-lock.json b/infrastructure/package-lock.json index 038de6bea..4681989b9 100644 --- a/infrastructure/package-lock.json +++ b/infrastructure/package-lock.json @@ -1,12 +1,12 @@ { "name": "infrastructure", - "version": "1.25.1", + "version": "1.26.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "infrastructure", - "version": "1.25.1", + "version": "1.26.0", "dependencies": { "aws-cdk-lib": "2.265.0", "constructs": "10.6.0" diff --git a/infrastructure/package.json b/infrastructure/package.json index 19310c420..25739354b 100644 --- a/infrastructure/package.json +++ b/infrastructure/package.json @@ -1,6 +1,6 @@ { "name": "infrastructure", - "version": "1.25.1", + "version": "1.26.0", "bin": { "infrastructure": "bin/infrastructure.js" }, diff --git a/infrastructure/test/turn-latency-observability.test.ts b/infrastructure/test/turn-latency-observability.test.ts index f25eb4ac3..09ac900d4 100644 --- a/infrastructure/test/turn-latency-observability.test.ts +++ b/infrastructure/test/turn-latency-observability.test.ts @@ -62,6 +62,17 @@ describe('TurnLatencyObservabilityConstruct', () => { 'AgentBuildPluginsMs', 'AgentBuildFinalizeMs', 'AgentBuildRestMs', + // Inside `tools` (docs/specs/turn-path-ttft.md P1a). + 'AgentBuildToolsFilterMs', + 'AgentBuildToolsGatewayMs', + 'AgentBuildToolsMcpMs', + 'AgentBuildToolsExtraMs', + // Past the prelude (docs/specs/turn-path-ttft.md P1b). + 'FirstTokenMs', + 'HeadOfTurnMs', + 'HeadOfTurnHistoryCountMs', + 'PreModelMs', + 'ModelMs', ]) { expect(body).toContain(metric); } diff --git a/scripts/build/build-one.sh b/scripts/build/build-one.sh index 5d430077d..c1f85ac5a 100755 --- a/scripts/build/build-one.sh +++ b/scripts/build/build-one.sh @@ -103,6 +103,7 @@ case "$SERVICE" in MANIFESTS=( "backend/src/apis/shared/__init__.py" "backend/src/apis/shared/timestamps.py" + "backend/src/apis/shared/aws_clients.py" ) # The RAG ingestion Lambda is arm64 (see the rag-ingestion CDK # construct), and the Dockerfile installs arm64 torch wheels. @@ -178,6 +179,7 @@ case "$SERVICE" in MANIFESTS=( "backend/src/apis/shared/__init__.py" "backend/src/apis/shared/timestamps.py" + "backend/src/apis/shared/aws_clients.py" ) # All four kb-migration Lambdas are arm64 (see the managed-kb # CDK construct). diff --git a/scripts/ci/classify-changes.sh b/scripts/ci/classify-changes.sh new file mode 100755 index 000000000..7019e9358 --- /dev/null +++ b/scripts/ci/classify-changes.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Map a pull request's changed paths to the test suites that can observe them. +# +# stdin: one repository-relative path per line (the PR's changed files; a +# rename is listed under both its old and new name). +# stdout: `key=true|false` lines for $GITHUB_OUTPUT, one per tests.yml suite: +# backend, backend_contracts, frontend, infra, load, scripts. +# +# Called by the `changes` job in .github/workflows/ci.yml. Before it existed +# the PR gate ran every suite on every pull request, so a two-line docs change +# paid the full backend suite (seven minutes on the runner) for a result that +# could not have changed. +# +# Fail-open by construction: +# * a path that matches no rule below turns EVERY suite on, so a new +# top-level directory is never silently untested; +# * an empty file list (no PR, or the API call failed) does the same; +# * only an explicit allowlist of paths that no suite reads turns nothing on. +# Which suite reads what is pinned by +# backend/tests/supply_chain/test_ci_path_filter.py — extend both together. +set -euo pipefail + +backend=false +backend_contracts=false +frontend=false +infra=false +load=false +scripts=false +all=false +seen=0 + +while IFS= read -r path; do + [ -z "$path" ] && continue + seen=$((seen + 1)) + case "$path" in + # Workflows, composite actions, dependabot and the retention doc are read + # by the supply-chain suites. scripts/ feeds the frontend coverage job + # (scripts/frontend/test.sh), the infra jest suite (scripts/release), the + # backend supply-chain suite (scripts/build) and the restore-data suite. + # Both change rarely and both reach several suites, so run everything. + .github/*|scripts/*) all=true ;; + # backend/scripts holds the backfills that infrastructure/test asserts + # exist by path (pending-backfills.test.ts). + backend/scripts/*) backend=true; load=true; infra=true ;; + # tests/load hand-encodes the SSE event names and the chat payload, so it + # runs with the backend: a rename in agents/main_agent/streaming should + # fail there rather than produce a load test that never sees a turn end. + backend/*) backend=true; load=true ;; + frontend/*) frontend=true ;; + # The backend supply-chain suite reads the CDK tree side by side with the + # env-var readers; pending-backfills.test.ts reads RELEASE_NOTES.md. + infrastructure/*|RELEASE_NOTES.md) infra=true; backend_contracts=true ;; + # gsi-update-limit.test.ts asserts against the cutting-a-release skill. + .claude/*) infra=true ;; + # The repo-root supply-chain suite runs inside the backend jobs. + tests/supply_chain/*) backend_contracts=true ;; + tests/load/*) load=true ;; + # No suite reads these. docs-site has its own build on push to main, and + # the TUI is not wired into CI at all. + docs/*|docs-site/*|tui/*) ;; + # Any other directory is unknown to this script: run everything. + */*) all=true ;; + # Root files no suite reads (RELEASE_NOTES.md was matched above). + *.md|*.MD|LICENSE|.gitignore|.editorconfig) ;; + *) all=true ;; + esac +done + +if [ "$seen" -eq 0 ] || [ "$all" = true ]; then + backend=true + backend_contracts=true + frontend=true + infra=true + load=true + scripts=true +fi + +printf 'backend=%s\n' "$backend" +printf 'backend_contracts=%s\n' "$backend_contracts" +printf 'frontend=%s\n' "$frontend" +printf 'infra=%s\n' "$infra" +printf 'load=%s\n' "$load" +printf 'scripts=%s\n' "$scripts" diff --git a/scripts/nightly/e2e-test.sh b/scripts/nightly/e2e-test.sh index 9d667e46b..58b2792c1 100755 --- a/scripts/nightly/e2e-test.sh +++ b/scripts/nightly/e2e-test.sh @@ -15,9 +15,10 @@ set -euo pipefail # USER_PASSWORD — Cognito regular user test account password # # The script resolves the frontend URL from: -# 1. SSM parameter /${CDK_PROJECT_PREFIX}/frontend/url (set by FrontendStack) -# 2. CloudFormation WebsiteUrl output from FrontendStack -# 3. CloudFormation DistributionDomainName output from FrontendStack +# 1. SSM parameter /${CDK_PROJECT_PREFIX}/frontend/url (written by the SPA +# distribution construct inside PlatformStack) +# 2. CloudFormation FrontendDnsRecordName output from PlatformStack (custom +# domain deployments only) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" @@ -38,7 +39,7 @@ log_success() { echo -e "${GREEN}[SUCCESS]${NC} $1"; } # Resolve the frontend URL of the deployed stack (CloudFront / S3) # --------------------------------------------------------------------------- get_base_url() { - # Try SSM parameter first (set by FrontendStack) + # Try SSM parameter first (set by the SPA distribution construct) local ssm_key="/${CDK_PROJECT_PREFIX}/frontend/url" local frontend_url frontend_url=$(aws ssm get-parameter \ @@ -57,37 +58,23 @@ get_base_url() { return 0 fi - # Fallback: query CloudFormation WebsiteUrl output from FrontendStack - local stack_name="${CDK_PROJECT_PREFIX}-FrontendStack" - frontend_url=$(aws cloudformation describe-stacks \ + # Fallback: the PlatformStack's custom-domain record name output. There is + # a single stack now; the former `-FrontendStack` WebsiteUrl / + # DistributionDomainName lookups referenced a stack that no longer exists. + local stack_name="${CDK_PROJECT_PREFIX}-PlatformStack" + local record_name + record_name=$(aws cloudformation describe-stacks \ --stack-name "${stack_name}" \ - --query "Stacks[0].Outputs[?OutputKey=='WebsiteUrl'].OutputValue" \ + --query "Stacks[0].Outputs[?OutputKey=='FrontendDnsRecordName'].OutputValue" \ --output text \ --region "${CDK_AWS_REGION}" 2>/dev/null || true) - if [ -n "${frontend_url}" ] && [ "${frontend_url}" != "None" ]; then - if [[ "${frontend_url}" == https://* ]]; then - echo "${frontend_url}" - else - echo "https://${frontend_url}" - fi - return 0 - fi - - # Last resort: query CloudFront distribution domain from FrontendStack - local cf_domain - cf_domain=$(aws cloudformation describe-stacks \ - --stack-name "${stack_name}" \ - --query "Stacks[0].Outputs[?OutputKey=='DistributionDomainName'].OutputValue" \ - --output text \ - --region "${CDK_AWS_REGION}" 2>/dev/null || true) - - if [ -n "${cf_domain}" ] && [ "${cf_domain}" != "None" ]; then - echo "https://${cf_domain}" + if [ -n "${record_name}" ] && [ "${record_name}" != "None" ]; then + echo "https://${record_name}" return 0 fi - log_error "Could not resolve frontend URL from SSM (${ssm_key}) or FrontendStack outputs" + log_error "Could not resolve frontend URL from SSM (${ssm_key}) or ${stack_name} outputs" return 1 } @@ -671,7 +658,7 @@ print(params.get('redirect_uri', [''])[0]) # a 301 redirect to HTTPS instead of reaching the BFF directly. log_warn " CloudFront /api/auth/login returned HTTP ${cf_status_code} redirect to: ${cf_login_redirect:0:120}" log_warn " This looks like an ALB HTTP→HTTPS redirect. CloudFront may be using HTTP_ONLY protocol." - log_warn " Fix: Ensure CDK_CERTIFICATE_ARN is set when deploying FrontendStack so CloudFront uses HTTPS to ALB." + log_warn " Fix: Ensure CDK_CERTIFICATE_ARN is set when deploying the PlatformStack so CloudFront uses HTTPS to ALB." else log_warn " CloudFront /api/auth/login returned HTTP ${cf_status_code} with no redirect (attempt $((cf_retries + 1))/${cf_max_retries}) — retrying in 10s..." fi diff --git a/scripts/nightly/seed-e2e-users.sh b/scripts/nightly/seed-e2e-users.sh index a8452e315..4fe607ae2 100644 --- a/scripts/nightly/seed-e2e-users.sh +++ b/scripts/nightly/seed-e2e-users.sh @@ -78,6 +78,65 @@ ensure_user() { # --------------------------------------------------------------------------- # Main # --------------------------------------------------------------------------- +# --------------------------------------------------------------------------- +# RBAC for the regular E2E user. +# +# The nightly stack never runs the bootstrap data seeding, so its `default` +# role grants no tools. The turn-path smoke matrix (turn-smoke.sh) needs +# `calculator` and `ask_user_question`, so the regular user joins a Cognito +# group that the `e2e_user` app role maps to. The group name lands in the +# access token's `cognito:groups`, which the BFF login reads as the user's +# roles; the role itself is written through the shared repository by +# backend/scripts/seed_e2e_role.py so every mapping row the resolver reads +# is created together. Both steps are idempotent. +# --------------------------------------------------------------------------- +E2E_USER_GROUP="${E2E_USER_GROUP:-e2e-users}" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +ensure_group_membership() { + local user_pool_id="$1" + local username="$2" + local group="$3" + + if ! aws cognito-idp get-group \ + --user-pool-id "${user_pool_id}" \ + --group-name "${group}" \ + --region "${CDK_AWS_REGION}" \ + --no-cli-pager > /dev/null 2>&1; then + log_info " Creating Cognito group ${group}..." + aws cognito-idp create-group \ + --user-pool-id "${user_pool_id}" \ + --group-name "${group}" \ + --description "Nightly E2E test accounts (maps to the e2e_user app role)" \ + --region "${CDK_AWS_REGION}" \ + --no-cli-pager > /dev/null + fi + + # admin-add-user-to-group is idempotent: re-adding a member is a no-op. + aws cognito-idp admin-add-user-to-group \ + --user-pool-id "${user_pool_id}" \ + --username "${username}" \ + --group-name "${group}" \ + --region "${CDK_AWS_REGION}" \ + --no-cli-pager > /dev/null + log_success " ${username} is in group ${group}" +} + +seed_e2e_role() { + if ! command -v uv > /dev/null 2>&1; then + log_warn " uv not found; skipping the e2e_user role seed (tool rows of the turn smoke will FAIL)" + return 0 + fi + log_info " Seeding the e2e_user app role (group ${E2E_USER_GROUP})..." + ( + cd "${PROJECT_ROOT}/backend" + AWS_REGION="${CDK_AWS_REGION}" uv run python scripts/seed_e2e_role.py \ + --prefix "${CDK_PROJECT_PREFIX}" \ + --region "${CDK_AWS_REGION}" \ + --group "${E2E_USER_GROUP}" + ) +} + main() { log_info "Seeding E2E test users in Cognito..." @@ -114,6 +173,9 @@ main() { ensure_user "${user_pool_id}" "${USER_USERNAME}" "${USER_PASSWORD}" "regular" ensure_user "${user_pool_id}" "${ADMIN_USERNAME}" "${ADMIN_PASSWORD}" "admin" + ensure_group_membership "${user_pool_id}" "${USER_USERNAME}" "${E2E_USER_GROUP}" + seed_e2e_role + log_success "E2E test users seeded successfully" } diff --git a/scripts/nightly/smoke-test.sh b/scripts/nightly/smoke-test.sh index 4df7871f0..d9d7f53a3 100755 --- a/scripts/nightly/smoke-test.sh +++ b/scripts/nightly/smoke-test.sh @@ -25,27 +25,42 @@ log_success() { echo -e "${GREEN}[SUCCESS]${NC} $1" } -# Get ALB URL from CDK outputs +# Resolve the ALB URL of the deployed stack. +# +# Order: the custom HTTPS subdomain when configured (avoids the HTTP 301), then +# the SSM parameter the ALB DNS construct writes, then the PlatformStack's +# `AlbUrl` output. There is one stack now; the old `-InfrastructureStack` +# lookup predates the single-stack architecture and always failed. get_alb_url() { - # Prefer the custom HTTPS subdomain when configured (avoids HTTP 301 redirect) if [ -n "${CDK_ALB_SUBDOMAIN:-}" ] && [ -n "${CDK_HOSTED_ZONE_DOMAIN:-}" ]; then echo "https://${CDK_ALB_SUBDOMAIN}.${CDK_HOSTED_ZONE_DOMAIN}" return 0 fi - local stack_name="${CDK_PROJECT_PREFIX}-InfrastructureStack" - local alb_dns=$(aws cloudformation describe-stacks \ + local alb_url + alb_url=$(aws ssm get-parameter \ + --name "/${CDK_PROJECT_PREFIX}/network/alb-url" \ + --query "Parameter.Value" \ + --output text \ + --region "${CDK_AWS_REGION}" 2>/dev/null || true) + if [ -n "${alb_url}" ] && [ "${alb_url}" != "None" ]; then + echo "${alb_url}" + return 0 + fi + + local stack_name="${CDK_PROJECT_PREFIX}-PlatformStack" + alb_url=$(aws cloudformation describe-stacks \ --stack-name "${stack_name}" \ - --query "Stacks[0].Outputs[?OutputKey=='AlbDnsName'].OutputValue" \ + --query "Stacks[0].Outputs[?OutputKey=='AlbUrl'].OutputValue" \ --output text \ - --region "${CDK_AWS_REGION}") - - if [ -z "${alb_dns}" ]; then - log_error "Could not retrieve ALB DNS name from stack ${stack_name}" - return 1 + --region "${CDK_AWS_REGION}" 2>/dev/null || true) + if [ -n "${alb_url}" ] && [ "${alb_url}" != "None" ]; then + echo "${alb_url}" + return 0 fi - - echo "https://${alb_dns}" + + log_error "Could not resolve the ALB URL from SSM (/${CDK_PROJECT_PREFIX}/network/alb-url) or ${stack_name} outputs" + return 1 } # Test health endpoint with retries diff --git a/scripts/nightly/turn-smoke.sh b/scripts/nightly/turn-smoke.sh new file mode 100755 index 000000000..59e5ffdb5 --- /dev/null +++ b/scripts/nightly/turn-smoke.sh @@ -0,0 +1,107 @@ +#!/bin/bash +set -euo pipefail + +# Script: Run the turn-path smoke matrix against a deployed nightly stack +# Description: Drives real agent turns through the stack's app-api (via the +# CloudFront /api path, exactly as the SPA does) as the nightly +# E2E user, and asserts the response contract on the wire: SSE +# frame order, interrupt/resume, the single-flight 409, Stop, +# preview sessions, attachments, restore, cache-hash stability. +# See docs/testing/smoke-regression.md §2 for the rows. +# +# Runs AFTER scripts/nightly/e2e-test.sh in the same job, which has already +# added the CloudFront URL to app-api's CORS origins and to the Cognito app +# client's callback list — both of which the scripted Hosted UI login needs. +# +# Required environment variables: +# CDK_PROJECT_PREFIX — CDK project prefix (e.g. nightly-develop) +# CDK_AWS_REGION — AWS region for the SSM lookup +# USER_USERNAME — Cognito regular-user test account (permanent password) +# USER_PASSWORD +# +# Optional: +# TURN_SMOKE_REPORT — path for the JSON report (default: turn-smoke-report.json) +# TURN_SMOKE_EXTRA_ARGS — appended to the smoke_turns.py invocation +# +# Exit status is the script's: 0 when no row FAILED, 1 otherwise, 2 for a +# setup problem (no base URL, login refused). The workflow step that calls this +# is non-blocking until the matrix has a track record on the nightly stack. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" +BACKEND_DIR="${PROJECT_ROOT}/backend" + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_success() { echo -e "${GREEN}[SUCCESS]${NC} $1"; } + +# Same primary lookup e2e-test.sh uses: the SSM parameter the SPA distribution +# construct writes. No CloudFormation fallback here — if the parameter is +# missing, the E2E step before this one has already failed loudly. +get_base_url() { + local frontend_url + frontend_url=$(aws ssm get-parameter \ + --name "/${CDK_PROJECT_PREFIX}/frontend/url" \ + --query "Parameter.Value" \ + --output text \ + --region "${CDK_AWS_REGION}" 2>/dev/null || true) + if [ -z "${frontend_url}" ] || [ "${frontend_url}" = "None" ]; then + log_error "Could not resolve the frontend URL from SSM (/${CDK_PROJECT_PREFIX}/frontend/url)" + return 1 + fi + if [[ "${frontend_url}" == https://* ]]; then + echo "${frontend_url}" + else + echo "https://${frontend_url}" + fi +} + +main() { + for var in CDK_PROJECT_PREFIX CDK_AWS_REGION USER_USERNAME USER_PASSWORD; do + if [ -z "${!var:-}" ]; then + log_error "${var} environment variable is required" + exit 2 + fi + done + + local base_url + base_url=$(get_base_url) || exit 2 + local api_url="${base_url}/api" + log_info "Project prefix: ${CDK_PROJECT_PREFIX}" + log_info "app-api via CloudFront: ${api_url}" + + local report="${TURN_SMOKE_REPORT:-${PROJECT_ROOT}/turn-smoke-report.json}" + + # The script reads the credentials from SMOKE_USERNAME / SMOKE_PASSWORD so + # they never appear on a command line. + export SMOKE_USERNAME="${USER_USERNAME}" + export SMOKE_PASSWORD="${USER_PASSWORD}" + + cd "${BACKEND_DIR}" + # shellcheck disable=SC2086 — TURN_SMOKE_EXTRA_ARGS is deliberately word-split + set +e + uv run python scripts/smoke_turns.py \ + --base-url "${api_url}" \ + --auth cognito \ + --with-attachments \ + --json "${report}" \ + ${TURN_SMOKE_EXTRA_ARGS:-} + local exit_code=$? + set -e + + case ${exit_code} in + 0) log_success "Turn smoke matrix passed (report: ${report})" ;; + 1) log_error "Turn smoke matrix has failing rows (report: ${report})" ;; + 2) log_warn "Turn smoke matrix could not start — a scripted Hosted UI login needs a classic (server-rendered) login page and a user with a permanent password" ;; + *) log_error "Turn smoke matrix exited ${exit_code}" ;; + esac + exit ${exit_code} +} + +main "$@" diff --git a/tui/pyproject.toml b/tui/pyproject.toml index 4b679bf73..df294bcc9 100644 --- a/tui/pyproject.toml +++ b/tui/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-tui" -version = "1.25.1" +version = "1.26.0" requires-python = ">=3.11" description = "Terminal client for the AgentCore Public Stack — streaming AI chat in your terminal" readme = "README.md" diff --git a/tui/src/agentcore_tui/__init__.py b/tui/src/agentcore_tui/__init__.py index 367e3d40e..5820c1821 100644 --- a/tui/src/agentcore_tui/__init__.py +++ b/tui/src/agentcore_tui/__init__.py @@ -7,6 +7,6 @@ from __future__ import annotations -__version__ = "1.25.1" +__version__ = "1.26.0" __all__ = ["__version__"] diff --git a/tui/uv.lock b/tui/uv.lock index 1adcfde87..a3f8933c0 100644 --- a/tui/uv.lock +++ b/tui/uv.lock @@ -8,7 +8,7 @@ resolution-markers = [ [[package]] name = "agentcore-tui" -version = "1.25.1" +version = "1.26.0" source = { editable = "." } dependencies = [ { name = "httpx" },