Release/1.24.0

.claude/skills/kaizen-research/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: kaizen-research
-description: Weekly Friday early-morning external + internal scan for emerging functionality, agentic trends, tools, and feature/UX improvements in the AgentCore Public Stack repo. Tracks AWS Bedrock + AgentCore announcements, Strands Agents releases, FastMCP (used by externally hosted MCP servers), the aws-samples/sample-strands-agent-with-agentcore reference repo, the MCP ecosystem (including MCP Apps + extensions), frontier model announcements, agent-harness patterns (including opencode (anomalyco/opencode) as an open-source coding-agent harness reference scanned through tooling, cost-effectiveness, and context-engineering lenses — releases-first, light touch), agentic UI/UX patterns (MCP Apps, Vercel AI SDK, assistant-ui, NN/g AI research, Linear/Cursor/Anthropic product blogs), and LibreChat as a parallel open-source agentic-platform reference (releases-first, light touch). Audits internal signals (recent commits, open PRs, CI failures, version-pin lag, dormant skills). Outputs a dated research doc + queues ideas in `docs/kaizen/review-queue.md` for that same morning's `kaizen-review-prep` (runs ~2 hours later) to rank into decisions. Opens a PR into `develop`. **Out of scope**: security advisories / Dependabot / CodeQL — those have dedicated tooling and don't need a weekly kaizen lens. Triggers: "kaizen research", "weekly research scan", "external scan", "what should we look at this week".
+description: Weekly Friday early-morning external + internal scan for emerging functionality, agentic trends, tools, and feature/UX improvements in the AgentCore Public Stack repo. Tracks AWS Bedrock + AgentCore announcements, Strands Agents releases, FastMCP (used by externally hosted MCP servers), the aws-samples/sample-strands-agent-with-agentcore reference repo, the MCP ecosystem (including MCP Apps + extensions), frontier model announcements, agent-harness patterns (including opencode (anomalyco/opencode) as an open-source coding-agent harness reference scanned through tooling, cost-effectiveness, and context-engineering lenses — releases-first, light touch), agentic UI/UX patterns (MCP Apps, Vercel AI SDK, assistant-ui, NN/g AI research, Linear/Cursor/Anthropic product blogs), and LibreChat as a parallel open-source agentic-platform reference (releases-first, light touch). Audits internal signals (recent commits, open PRs, CI failures, version-pin lag, dormant skills). Outputs a dated research doc + queues ideas in `docs/kaizen/review-queue.md` for that same morning's `kaizen-review-prep` (runs ~4 hours later) to rank into decisions. Opens a PR into `develop`. **Out of scope**: security advisories / Dependabot / CodeQL — those have dedicated tooling and don't need a weekly kaizen lens. Triggers: "kaizen research", "weekly research scan", "external scan", "what should we look at this week".
 ---
 
 # Kaizen Research
@@ -18,7 +18,7 @@ Friday early morning. The "what's the rest of the world learning that we should
 
 ## When to run
 
-Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am MT) so both docs are waiting when Phil sits down Friday morning. Phil reviews, picks 1–3 to ship over the coming week, and POCs additional items over the weekend. Last weekend's POC findings surface in *this* run's review-prep as Carried Over items (lifted from comments on the previous week's research PR).
+Friday early morning (~6am MT). `kaizen-review-prep` runs ~4 hours later (~10am MT), a gap sized to this scan's own runtime (2h18m on 2026-09-11) rather than to the 2 hours the two skills used to claim — on 2026-09-18 the prep ran 18 minutes after this scan started and produced a review with no research doc. Both docs are waiting when Phil sits down Friday late morning; he reviews, picks 1–3 to ship over the coming week, and the *outcomes* of those picks are what review-prep reads the following Friday.
 
 ## Sources
 
@@ -436,8 +436,7 @@ gh pr create --base develop --head "$BRANCH" \
 
 ## Review
 - Read the research doc.
-- Comment on the PR with reactions and any weekend POC findings — these become first-class signal for *next* Friday's `kaizen-review-prep`.
-- POC promising ideas over the weekend.
+- Ship, decline or defer individual ideas in their own PRs — a merged PR is the signal `kaizen-review-prep` reads next Friday. Deep analysis belongs in a `docs/kaizen/review-queue.md` entry on the PR branch that produced it, where the next review will find it.
 
 ## Decision
 Ship the doc to `develop`. Ranking into decisions happens in the kaizen-review-prep PR opened later this morning. Action on individual ideas happens in separate PRs the following week.

.claude/skills/kaizen-review-prep/SKILL.md

@@ -1,31 +1,34 @@
 ---
 name: kaizen-review-prep
-description: Friday late-morning synthesis. Runs ~2 hours after `kaizen-research` the same morning. Consumes this week's research doc, open items in `docs/kaizen/review-queue.md`, last weekend's POC findings (from comments on the previous week's research PR), and recent merges/reverts/CI signal — produces a ranked, decision-oriented agenda. Every item has a Ship / Decline / Defer recommendation. Opens a PR into `develop`. Triggers: "kaizen review prep", "weekly review prep", "friday review", "rank kaizen ideas".
+description: Friday late-morning synthesis. Runs ~4 hours after `kaizen-research` the same morning. Consumes this week's research doc, open items in `docs/kaizen/review-queue.md`, the outcomes of the prior review's proposals (read from merged PRs and from open PR branches), and recent merges/reverts/CI signal — produces a ranked, decision-oriented agenda. Every item has a Ship / Decline / Defer recommendation. Opens a PR into `develop`. Triggers: "kaizen review prep", "weekly review prep", "friday review", "rank kaizen ideas".
 ---
 
 # Kaizen Review Prep
 
-Friday late morning, after `kaizen-research` ran earlier the same morning. This skill consolidates this week's research + open queue items + last weekend's POC findings (lifted from PR comments on the previous week's research PR) + recent repo state into a ranked decision agenda. Phil reviews Friday morning, marks ✅/❌/⏸ on each item, ships 1–3 the following week, and POCs the next batch over the weekend.
+Friday late morning, after `kaizen-research` ran earlier the same morning. This skill consolidates this week's research + open queue items + the outcomes of the *prior* review's proposals + recent repo state into a ranked decision agenda. Phil reviews Friday morning, marks ✅/❌/⏸ on each item, and ships 1–3 the following week. What those 1–3 did is the feedback signal the next run opens with.
 
 ## Philosophy
 
 - **Review is a decision forum, not a status update.** Everything that lands in the output should be either: (a) actionable this week, (b) explicitly deferred with a reason and revisit date, or (c) declined. Nothing is "noted." Noted-and-forgotten is how systems accumulate friction.
 - **Subtraction first.** Every proposal ranks against "do nothing" and "retire something instead." If a proposal adds anything, it must explain what existing thing it either replaces or simplifies.
 - **Dual lens — impact + capability-unlock.** Rank proposals through *two* lenses, not one: (a) **impact on existing code** (does this change, simplify, or obsolete something we already have?) and (b) **capability unlock** (what *new* product capability or UX enhancement does this enable that we couldn't easily build before?). Subtraction-first applies to lens (a). But proposals that genuinely unlock new product surface — code-interpreter sandboxes, persistent agent state, multi-agent UI attribution, new SSE event types that enable inline UI, etc. — must be evaluated on their strategic merit, *not* auto-deferred because they don't intersect existing code. A proposal with no `Subtracts` value but a substantive `Unlocks` value can rank above a low-impact dep-bump. Don't penalize net-new capability for not being a cleanup.
 - **Multiple cycles.** Kaizen is small changes, weekly, compounding. If this week's review touches 3 things, next week's will touch 3 different things. Phil doesn't need a grand plan — he needs a reliable weekly cadence.
-- **One-week feedback lag is intentional.** Phil reviews Friday → POCs over the weekend → those POC findings surface in the *next* Friday's review-prep as Carried Over items. Don't try to fold same-day POC findings in — they don't exist yet.
+- **Outcomes are the feedback channel, not comments.** Open every run with a scorecard of the prior review's proposals: which shipped, which didn't, and what the merged PRs found. Shipping a proposal is the only reaction that has ever been recorded — see `decisions.md` [2026-09-21]. Do not ask for, wait on, or rank by PR comments.
+- **Read open PR branches, not just `develop`.** This repo's deepest analysis is written as `docs/kaizen/review-queue.md` entries *inside the feature PR that produced it*. A review that reads only `develop` misses them, re-derives them worse, and gets corrected by a branch that was sitting there the whole time. Before ranking, `gh pr list --state open` and diff each against `develop` for `docs/kaizen/` and any spec the review cites. Those entries are authoritative over anything re-derived; cite them rather than restating them, and flag the merge order when two PRs touch the top of `## Open`.
 - **No edits outside `docs/kaizen/`.** This skill writes one Markdown file under `docs/kaizen/reviews/` and updates `docs/kaizen/review-queue.md` (moves Open → Resolved post-review). It never touches source code, `CLAUDE.md`, or skill files. Those changes happen in separate PRs after the review.
 
 ## When to run
 
-Friday late morning (~8am MT), ~2 hours after `kaizen-research` runs. Phil reviews both docs Friday morning, picks 1–3 to ship over the coming week, and POCs additional items over the weekend. POC findings from last weekend's POC session surface here as Carried Over items (lifted from PR comments on the *previous* week's research PR — not this week's, which Phil hasn't seen yet).
+Friday late morning (~10am MT), ~4 hours after `kaizen-research` runs. Phil reviews both docs Friday, picks 1–3 to ship over the coming week; what those picks did is the scorecard this skill opens with the following Friday.
+
+**Preflight — check that this morning's research doc exists before doing anything else.** The gap is sized to the scan's runtime (2h18m on 2026-09-11), not guaranteed by it. If `docs/kaizen/research/<today>.md` is absent or the `kaizen-research` run is still in flight, say so **in the review doc's header** and proceed queue-first — the `## Open` entries are the primary source and research is one input. Degrading is fine; degrading silently is not. This failed on 2026-09-18, when prep ran 18 minutes after the scan started.
 
 ## Inputs
 
 1. **Most recent `docs/kaizen/research/YYYY-MM-DD.md`** — Friday's scan. Its Top 5 ideas are the primary candidate list.
 2. **`docs/kaizen/review-queue.md`** — `## Open` entries. Includes both this week's ideas (just appended by `kaizen-research`) and any prior-week items that weren't resolved.
 3. **Last 1–2 `docs/kaizen/reviews/*.md`** — what was proposed before, what was decided, anything deferred to "revisit by [date]".
-4. **PR comments on the *previous* week's kaizen-research PR.** `gh pr view <n> --comments` — Phil's reactions and weekend POC findings are first-class signal. The PR opened *this* morning by `kaizen-research` is too fresh; comments accumulate over the week as Phil POCs ideas. Pick the research PR from one week ago (or the most recent merged/closed kaizen-research PR), not today's.
+4. **Outcomes of the prior review's proposals.** For each proposal in the most recent `docs/kaizen/reviews/*.md`, establish on disk or in git what happened: shipped (name the commit/PR), not started, or superseded. This is the feedback channel — see the Philosophy bullet. Include **open PR branches**, which routinely carry queue entries that refute or narrow what `develop` alone would suggest.
 5. **`docs/kaizen/decisions.md`** (if it exists) — declined items with reasons. Don't re-propose without materially new context.
 6. **Recent activity since last review:**
    - `git log develop --since="<last review date>" --oneline --no-merges` — what shipped.
@@ -78,7 +81,7 @@ check before decisions.]
 - **Unlocks** (if applicable): [net-new product capability, UX pattern, or enhancement this enables — bulleted if multiple. Required for proposals where `Subtracts: no — addition only`; the unlock is the justification. Omit when purely a cleanup/dep-bump and not applicable.]
 - **Effort**: Low / Med / High
 - **Impact**: Low / Med / High
-- **POC findings (if Phil tried it)**: [summary or "not POCed"]
+- **Evidence**: [measured — cite the number and where it came from | verified on disk — cite file:line | asserted — say so plainly]
 - **Ship means**: [specific action — "open PR updating X to do Y" or "retire skill Z"]
 - **Decline means**: [what happens instead — usually "keep current behavior, revisit in N weeks"]
 - **Recommendation**: Ship / Decline / Defer N weeks — [one-sentence why]
@@ -158,12 +161,12 @@ After Phil reviews and the decisions are logged in the review doc, this skill (o
    - Last ~14 days of `CHANGELOG.md` and `RELEASE_NOTES.md`
    - `CLAUDE.md` (read-only — for context, not edits)
 
-3. **Pull PR comments on the latest research PR** (parallel with step 4):
+3. **Score the prior review's proposals, and read open PR branches** (parallel with step 4):
    ```
-   gh pr list --base develop --state all --search "kaizen/research" --limit 1 --json number,url
-   gh pr view <number> --comments
+   gh pr list --state open --json number,title,headRefName
+   git fetch origin && git diff origin/develop...origin/<branch> -- docs/kaizen/
    ```
-   Capture Phil's reactions. POC findings he mentions get folded into proposal entries.
+   For each proposal in the last review doc, resolve shipped / not started / superseded against git and disk — that scorecard opens the new review. Queue entries found on open branches are authoritative over anything re-derived here.
 
 4. **Pull recent activity** (parallel Bash):
    - `git log develop --since="<last review date>" --oneline --no-merges`
@@ -183,7 +186,7 @@ After Phil reviews and the decisions are logged in the review doc, this skill (o
    - Low-effort × High-impact first.
    - **Retirement candidates** get a +1 boost (subtraction bias).
    - **Capability-unlock items** (proposals with a substantive `Unlocks` field — new product capability, UX surface, or platform primitive adoption) rank on their strategic merit. Do not auto-defer just because `Subtracts: no`. A High-impact unlock can rank above a Low-impact subtraction.
-   - Items with **POC findings** rank above untested items at the same effort/impact.
+   - Items with **measured evidence** rank above asserted ones at the same effort/impact.
 
 8. **Cap the proposal count at 10.** If more than 10 candidates, defer the lowest-ranked to next week with a note. The review is supposed to take 10-15 minutes, not be exhaustive.
 
@@ -213,7 +216,7 @@ gh pr create --base develop --head "$BRANCH" \
 - N proposals ranked Effort × Impact (retirement candidates boosted).
 - Friction patterns from the week's commits, PRs, and CI surfaced.
 - Carried-over deferred items now due for re-decision.
-- POC findings (from kaizen-research PR comments) folded into proposals where Phil tried something.
+- Scorecard of the prior review's proposals: what shipped, what didn't, and what the merged PRs found.
 
 ## Review
 1. Read Friction (2 min).

.github/workflows/backend.yml

@@ -8,6 +8,10 @@ on:
     paths:
       - 'backend/**'
       - 'scripts/build/**'
+      # recover-evicted-deploy.sh runs in THIS workflow. A change to the
+      # guard that re-dispatches an evicted peer deploy must itself be
+      # exercised by a real run.
+      - 'scripts/ci/**'
       - '.github/workflows/backend.yml'
 
 permissions:
@@ -19,7 +23,13 @@ env:
 
 # Shared "deploy-<ref>" group with platform.yml so a CFN deploy and these
 # API-driven code deploys never mutate the same ECS service / AgentCore
-# Runtime / Lambda concurrently — they queue instead.
+# Runtime / Lambda concurrently.
+#
+# ⚠️ The group does NOT queue — it is a latch with a queue depth of one, so
+# a third run entering it cancels whichever run was pending, before that run
+# executes a job. See the longer note in platform.yml and the
+# `recover-evicted-peer` job below, which re-dispatches a peer deploy this
+# run evicted. Enforced by backend/tests/supply_chain/test_concurrency_config.py.
 concurrency:
   group: deploy-${{ github.ref }}
   cancel-in-progress: false
@@ -40,6 +50,45 @@ concurrency:
 # ------------------------------------------------------------
 
 jobs:
+  # Recover a peer deploy that GitHub silently evicted from the shared
+  # "deploy-<ref>" concurrency group when THIS run claimed the pending slot.
+  # A concurrency group is a latch with a queue depth of ONE, not a queue:
+  # "any existing pending job or workflow in the same concurrency group will
+  # be canceled and the new queued job or workflow will take its place." So a
+  # third run entering an occupied group kills whichever run was waiting,
+  # before it executes a single job, leaving a `cancelled` conclusion that is
+  # indistinguishable from a human pressing Cancel.
+  #
+  # That is harmless when the evicted run is superseded by a LATER run of the
+  # SAME workflow (both deploy the tree at HEAD, and git is cumulative). It is
+  # destructive across workflows: a Platform Stack run does not ship
+  # container images or Lambda code, so when it evicts a pending Backend
+  # Deploy run that code is simply never shipped and CI stays green.
+  #
+  # No `needs`, so the recovery dispatch goes out at the top of the run rather
+  # than behind the deploy it is unrelated to. The re-dispatched run queues
+  # behind this one, so the concurrency guarantee is preserved.
+  # See scripts/ci/recover-evicted-deploy.sh.
+  # Enforced by backend/tests/supply_chain/test_concurrency_config.py.
+  recover-evicted-peer:
+    name: Recover evicted peer deploy
+    runs-on: ubuntu-24.04
+
+    permissions:
+      contents: read
+      # `gh workflow run` re-dispatches the peer workflow.
+      actions: write
+
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Detect and recover an evicted Platform Stack run
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: bash scripts/ci/recover-evicted-deploy.sh platform.yml
+
   test-infra:
     name: Test infrastructure (jest)
     permissions: