Release/1.26.0

.claude/skills/smoke-regression/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: smoke-regression
+description: >-
+  Run the smoke and regression pass for this stack after a change: the hermetic
+  suites, the turn-matrix script that drives real agent turns and asserts every
+  SSE code path (frame order, interrupt/resume, 409, Stop, preview, restore,
+  cache stability), and the in-app-browser checklist for rendered surfaces.
+  Use when the user says "smoke test", "regression test", "run the smoke pass",
+  "verify this on dev", "test all the response paths", "did we break
+  streaming", or before a release. Points at docs/testing/smoke-regression.md
+  as the single source of truth for rows, fixtures and failure reading.
+---
+
+# Smoke / regression pass
+
+The matrix, its fixtures and how to read a failure live in
+[docs/testing/smoke-regression.md](../../../docs/testing/smoke-regression.md).
+Read §2.2 before running L1 and §3 before L2. This skill is the procedure.
+
+## 0. Decide the layers
+
+From the diff, pick per §4 of the doc:
+
+- **L0** always, for the packages touched.
+- **L1** when the change touches `inference_api/chat/*`, `agents/main_agent/**`,
+  `apis/shared/harness/**`, the SSE table, or any interrupt flow.
+- **L2** when the change touches a rendered surface (composer, loader, tool
+  rail, pickers, banners) or an interrupt flow.
+
+Say which layers you are running and why in one line before starting.
+
+## 1. L0 — hermetic
+
+From the branch's checkout. In a worktree `PYTHONPATH` is mandatory or you test
+`develop`:
+
+```bash
+cd <checkout>/backend && PYTHONPATH=$PWD/src AWS_EC2_METADATA_DISABLED=true <main checkout>/backend/.venv/bin/python -m pytest tests/ -q
+```
+
+Frontend `cd frontend/ai.client && npx ng test --watch=false`; infra
+`cd infrastructure && npx tsc --noEmit && npx jest`. Never pass dummy AWS
+credentials to make a stall go away (see the doc §1).
+
+## 2. L1 — turn matrix
+
+Pick the target in this order and state which one you used:
+
+1. **Deployed dev through the Runtime** (always available to an agent with the
+   `dev-ai` profile and an active headless grant):
+   ```bash
+   cd backend && AWS_PROFILE=dev-ai uv run python scripts/smoke_turns.py --auth headless-grant --user-id <sub> --prefix dev-boisestateai-v2 --json /tmp/smoke.json
+   ```
+   The grant's owner id is in `dev-boisestateai-v2-bff-sessions` under
+   `PK=HEADLESS-GRANT#…` (`grant_user_id`). If the script reports no active
+   grant, the user must sign in and run a headless run once; do not try to
+   mint tokens another way (the doc explains why the other routes are dead ends).
+   This mode skips the app-api-only rows (409, Stop, steer). It deletes the
+   sessions it created; after a crash or `--keep`, pass `--delete-sessions id,id`.
+2. **Local stack** when the branch is not deployed yet and you need the
+   app-api rows: app-api on :8000 with `SKIP_AUTH=true`, inference-api on
+   :8001, then `--base-url http://127.0.0.1:8000`. Do not flip `SKIP_AUTH` in
+   the user's `backend/src/.env` without asking; use a worktree copy.
+3. **Deployed through app-api** with a Cognito password user
+   (`--auth cognito`), when one exists for the environment.
+
+Add `--with-attachments` when attachments are in scope, and the fixture flags
+(`--approval-tool`, `--skill-id`, `--agent-id`) when those rows matter. Never
+pass `--allow-prod`.
+
+Paste the report table into your summary. Treat SKIP as "not covered here",
+OBSERVED as information, WARN as a model-quality note. A FAIL is a finding:
+quote its detail, check the three traps in the doc §2.2 before calling it a
+bug, then fix or report.
+
+## 3. L2 — in-app browser
+
+Use the built-in browser pane on `https://dev.boisestate.ai`; it carries the
+user's session. Install the frame-capture snippet from the doc §3 with
+`javascript_tool` first, then work through the checklist rows that apply.
+Verify with `read_page` and `window.__frames`, not screenshots, except for a
+final visual proof. Mirror the CSRF cookie into `X-CSRF-Token` for any manual
+non-GET call. Delete the smoke conversations when done.
+
+## 4. Report
+
+One table per layer run, a one-line verdict, and the exact commands used so
+the run is reproducible. If a layer was skipped, say which and why.

.github/workflows/ci.yml

@@ -3,9 +3,22 @@ name: CI
 # Pre-merge test gate. The deploy workflows (backend.yml, platform.yml,
 # frontend-deploy.yml) only run on push to develop/main — they run the
 # test jobs as a gate *before deploying*, so unit tests never ran on the
-# PR itself. This workflow runs the full test suites on every pull request
-# into develop/main, with no deploy/build/AWS steps, so regressions are
-# caught before merge. Deploys remain push-only.
+# PR itself. This workflow runs the test suites on every pull request into
+# develop/main, with no deploy/build/AWS steps, so regressions are caught
+# before merge. Deploys remain push-only.
+#
+# Which suites run is decided per PR by the `changes` job from the PR's file
+# list (scripts/ci/classify-changes.sh): a docs-only PR runs nothing, a
+# frontend-only PR runs the two SPA jobs, an infrastructure PR runs jest plus
+# the backend suites that read the CDK tree. The mapping is fail-open — an
+# unrecognised path, a workflow_dispatch, or a failed file-list call runs
+# everything — and is pinned by backend/tests/supply_chain/test_ci_path_filter.py.
+#
+# The filtering lives on the JOBS, not on the trigger's `paths:`. A workflow
+# that does not trigger leaves a required status check "Expected" forever
+# and blocks the merge; a job that is skipped reports as skipped and does
+# not. develop has no required checks today, but this shape survives adding
+# them.
 on:
   workflow_dispatch:
   pull_request:
@@ -23,25 +36,71 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  # Classify the PR's changed files into the suites that can observe them.
+  # The file list comes from the REST API rather than a checkout diff: no
+  # fetch-depth games, no third-party action, and it is exact for renames
+  # (both names are fed through). Any failure to obtain the list yields an
+  # empty stdin, which the classifier treats as "run everything".
+  changes:
+    name: Classify changed paths
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      pull-requests: read
+    outputs:
+      backend: ${{ steps.classify.outputs.backend }}
+      backend_contracts: ${{ steps.classify.outputs.backend_contracts }}
+      frontend: ${{ steps.classify.outputs.frontend }}
+      infra: ${{ steps.classify.outputs.infra }}
+      load: ${{ steps.classify.outputs.load }}
+      scripts: ${{ steps.classify.outputs.scripts }}
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+          sparse-checkout: scripts/ci
+      - name: Classify
+        id: classify
+        env:
+          GH_TOKEN: ${{ github.token }}
+          EVENT_NAME: ${{ github.event_name }}
+          REPO: ${{ github.repository }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          files=""
+          if [ "$EVENT_NAME" = "pull_request" ]; then
+            files=$(gh api "repos/$REPO/pulls/$PR_NUMBER/files" --paginate \
+              --jq '.[] | .filename, (.previous_filename // empty)') || files=""
+            echo "$(printf '%s\n' "$files" | grep -c .) changed path(s) on PR #$PR_NUMBER"
+          else
+            echo "$EVENT_NAME is not a pull request: running every suite"
+          fi
+          printf '%s\n' "$files" | bash scripts/ci/classify-changes.sh | tee -a "$GITHUB_OUTPUT"
+
   tests:
     name: Tests
+    needs: changes
     permissions:
       contents: read
     uses: ./.github/workflows/tests.yml
     with:
-      run_backend: true
-      run_frontend: true
+      run_backend: ${{ needs.changes.outputs.backend == 'true' }}
+      # The supply-chain/architecture subset, for PRs that touch what those
+      # tests read (CDK, scripts, workflows) without touching backend/. A
+      # no-op inside tests.yml when run_backend is also true.
+      run_backend_contracts: ${{ needs.changes.outputs.backend_contracts == 'true' }}
+      run_frontend: ${{ needs.changes.outputs.frontend == 'true' }}
       # The frontend suite again under --coverage — the nightly's invocation.
       # Coverage changes how the specs are built (see tests.yml), so specs can
       # pass here and fail there. Only the PR gate pays for it; the deploy
       # workflows keep running the plain suite.
-      run_frontend_coverage: true
-      run_infra: true
+      run_frontend_coverage: ${{ needs.changes.outputs.frontend == 'true' }}
+      run_infra: ${{ needs.changes.outputs.infra == 'true' }}
       # Pure-logic tests for tests/load. Deliberately only on the PR gate: the
       # deploy workflows run test suites to protect a deploy, and the load
       # suite has no bearing on one.
-      run_load_unit: true
+      run_load_unit: ${{ needs.changes.outputs.load == 'true' }}
       # Offline unit tests for scripts/restore-data. PR gate only, like the
       # load suite: a deploy gains nothing from them.
-      run_scripts: true
+      run_scripts: ${{ needs.changes.outputs.scripts == 'true' }}
 

.github/workflows/nightly-deploy-pipeline.yml

@@ -11,7 +11,8 @@
 #   3. deploy-*-code          — API-driven code deploys (ECS / AgentCore Runtime
 #                               / Lambda) onto the freshly-deployed stack.
 #   4. deploy-frontend        — build SPA + S3 sync + CloudFront invalidation.
-#   5. smoke-test / e2e-tests — validate the deployment.
+#   5. smoke-test / e2e-tests — validate the deployment (e2e-tests also runs
+#      the turn-path smoke matrix, scripts/nightly/turn-smoke.sh, non-blocking).
 #   6. teardown               — delete the ephemeral stack + buckets/secrets/logs.
 #
 # Ephemeral deploys run with NO custom domain (CDK_DOMAIN_NAME=""); the SPA,
@@ -504,6 +505,17 @@ jobs:
           aws-role-arn: ${{ secrets.AWS_ROLE_ARN }}
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      # uv is needed before the E2E run: e2e-test.sh seeds the E2E users and
+      # (via backend/scripts/seed_e2e_role.py) the e2e_user RBAC role, and the
+      # turn smoke below reuses the same environment.
+      - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+        with:
+          version: '0.7.12'
+          enable-cache: true
+          cache-dependency-glob: 'backend/uv.lock'
+      - name: Sync backend deps
+        working-directory: backend
+        run: uv sync --extra agentcore --frozen
       - name: Run E2E tests
         run: bash scripts/nightly/e2e-test.sh
       - name: Upload Playwright report
@@ -513,6 +525,31 @@ jobs:
           name: playwright-report-${{ inputs.label }}
           path: frontend/ai.client/playwright-report/
           retention-days: 30
+      # ── Turn-path smoke matrix (docs/testing/smoke-regression.md §2) ──
+      # Real agent turns through the stack's app-api as the E2E user, asserting
+      # the SSE contract, interrupt/resume, 409, Stop, preview, attachments,
+      # restore and cache-hash stability. Runs after the Playwright step because
+      # e2e-test.sh has by then patched CORS and the Cognito callback list for
+      # the CloudFront URL, which the scripted Hosted UI login depends on.
+      # Non-blocking (continue-on-error) until it has a green track record on
+      # the nightly stack; promote by removing that line. The E2E user's
+      # `calculator` / `ask_user_question` grants come from the e2e_user role
+      # that e2e-test.sh seeds; a row that cannot run reports SKIP, never FAIL.
+      - name: Run turn smoke matrix
+        id: turn-smoke
+        if: always()
+        continue-on-error: true
+        env:
+          TURN_SMOKE_REPORT: ${{ github.workspace }}/turn-smoke-report.json
+        run: bash scripts/nightly/turn-smoke.sh
+      - name: Upload turn smoke report
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: turn-smoke-report-${{ inputs.label }}
+          path: turn-smoke-report.json
+          retention-days: 30
+          if-no-files-found: ignore
 
   # ── Auto-teardown ──────────────────────────────────────────────────────────
   # ALWAYS runs (unless skip-teardown), even if any deploy/test job above failed

.github/workflows/tests.yml

@@ -16,6 +16,11 @@ on:
         required: false
         default: false
         type: boolean
+      run_backend_contracts:
+        description: 'Run only the backend suites that read the CDK tree, scripts and workflows (supply_chain, architecture); a no-op when run_backend is also set'
+        required: false
+        default: false
+        type: boolean
       run_frontend:
         description: 'Run frontend test suite'
         required: false
@@ -67,10 +72,13 @@ jobs:
         run: uv sync --extra agentcore --extra dev
       - name: Run pytest
         working-directory: backend
-        # `-n auto` fans the suite across all runner cores (pytest-xdist).
-        # Verbosity/--tb live in backend/pytest.ini. The nightly coverage run
-        # (scripts/backend/test.sh) is deliberately left serial.
-        run: uv run pytest tests/ -n auto
+        # `-n logical` fans the suite across every vCPU (pytest-xdist). Not
+        # `auto`: with psutil installed, `auto` counts PHYSICAL cores, and the
+        # 4-vCPU runner reports 2, so it started 2 workers for ~11k tests
+        # (`created: 2/2 workers` in the log). Verbosity/--tb live in
+        # backend/pytest.ini. The nightly coverage run (scripts/backend/test.sh)
+        # is deliberately left serial.
+        run: uv run pytest tests/ -n logical
       # The repo-ROOT supply-chain suite (not backend/tests/supply_chain):
       # cross-package contracts that read CDK, the backup/restore scripts and
       # the python env-var readers side by side, so it lives above backend/.
@@ -81,6 +89,38 @@ jobs:
       - name: Run root supply-chain pytest
         run: uv run --project backend --no-sync pytest tests/supply_chain/ -v
 
+  # The subset of the backend suite that reads OUTSIDE backend/: the
+  # supply-chain and architecture tests assert against the CDK tree, the
+  # scripts and the workflows side by side with the Python that consumes
+  # them, so an infrastructure-only PR still needs them — and nothing else in
+  # backend/tests/. About a hundred tests, well under a minute, against seven
+  # minutes for the full suite. Skipped whenever run_backend is set, because
+  # the full run already covers it.
+  test-backend-contracts:
+    if: ${{ inputs.run_backend_contracts && !inputs.run_backend }}
+    name: Test backend contracts (pytest)
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          ref: ${{ inputs.ref }}
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+        with:
+          version: '0.7.12'
+          enable-cache: true
+          cache-dependency-glob: 'backend/uv.lock'
+      - name: Sync deps
+        working-directory: backend
+        run: uv sync --extra agentcore --extra dev
+      - name: Run backend contract suites
+        working-directory: backend
+        run: uv run pytest tests/supply_chain/ tests/architecture/ -v
+      - name: Run root supply-chain pytest
+        run: uv run --project backend --no-sync pytest tests/supply_chain/ -v
+
   test-frontend:
     if: ${{ inputs.run_frontend }}
     name: Test frontend