Skip to content

feat: enable agent-driven QA workflows (WI-2184) - #518

Merged
stefan-ernst merged 4 commits into
mainfrom
feat/wi-2184-agent-qa
Nov 16, 2026
Merged

stefan-ernst merged 4 commits into
mainfrom
feat/wi-2184-agent-qa

Conversation

@stefan-ernst

@stefan-ernst stefan-ernst commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Delivers WI-2184 and its seven child stories (WI-2185–WI-2191):

  • Shared agent/MCP/Standard tools for case creation, editing, reversible archive, bounded lexical search, bulk operations, per-item coverage, and observed defect effectiveness.
  • ws test case create|edit with flags or JSON file/stdin; ws test standard exports the bundled skill, starter workspace-page standard, and setup/worked-example guide.
  • Atomic authored step/label and BDD updates, scope/permission/workspace checks, preserved step identities, and protection against deleting execution evidence.
  • V2-only bulk, coverage, and effectiveness endpoints; audit classification, canonical metadata, regenerated OpenAPI, and client-generation smoke validation.
  • Standard bindings snapshot attached skills and reviewed page content at admission and disclose them through get_agent_skill. Profile creation now preserves requested capability groups.

Review follow-up

  • Coding and Standard runtimes now share agentskills.SnapshotBinding, store only RunGrants.Skills, and fail closed rather than silently omitting a required skill. The whole serialized skill set is capped at 32 skills, 256 KiB and 65536 estimated tokens, including metadata/escaping; bounded queries detect oversized legacy bindings.
  • Source-aware tool admission hides Standard-only skill disclosure from MCP/chat. Coverage/report denials now name the correct subject.
  • Removed the unused BDD update method and duplicate parse; title synchronization and explicit-title precedence remain tested.
  • Effectiveness offset paging is explicitly documented as independent samples of a mutable ranking, not a consistent snapshot. The skill no longer promises an exhaustive concurrent sweep.
  • Authored labels are rechecked and locked in the write transaction on PostgreSQL (SQLite serializes writers), producing a clean not-found result if deleted after preflight.
  • Added the requested collection caps, duplicate labels/case IDs, foreign milestone, BDD steps/format, snapshot failure/budget, and label-deletion regression coverage.

Second review follow-up / upgrade note

  • Attachment changes now reject more than 32 distinct IDs (including disabled skills) with a configuration-time 400 before mutating settings. Duplicate IDs do not consume slots.
  • Studio readiness uses the same snapshot loader as run admission, checking aggregate size, metadata and saved page readability. Invalid snapshots return skill_snapshot_invalid with the underlying error and a recovery hint; activation returns 422. Previously Ready legacy bindings show needs_setup instead of being advertised as available.
  • Upgrade: existing attachments are preserved. Validate profiles before the next run; detach/disable excess enabled skills or reduce bodies/reviewed page snapshots, then validate and activate again. Replacement lists must contain at most 32 distinct IDs. Runtime validation remains necessary because skill content can change after readiness validation. The shipped setup guide includes this recovery procedure.
  • Label existence/locking uses two batched reads per authored case instead of two per label; PostgreSQL key-share locks and deletion-race rollback remain covered.
  • Renamed the bounded reader to ListEnabledForBindingSnapshot and documented SkillGrant.Error as compatibility with older persisted unavailable grants.

Contract decisions

  • Archive is the existing reversible inactive lifecycle; no new hard-delete tool.
  • Retrieval defaults to 50 summaries and caps at 100; lexical SQL LIKE search includes steps and Gherkin. Semantic retrieval is explicitly not implemented.
  • Direct child items represent explicit criteria; existing Tests links determine traceability. No prose parsing or inferred adequacy. Without children, the response labels coarse item-level coverage. Responses disclose truncation.
  • Effectiveness counts persisted executed run/case results in [since, until), distinct same-workspace builtin Bug links from failed results, and finding executions without link fan-out. Zero-execution cases have null rates; missing links are not evidence of ineffectiveness.
  • Synthetic data and versioned external fixture references use existing step.data/preconditions. Storage never executes generators or dereferences URLs; no secrets in case bodies.
  • tests:write remains absent from default coding-agent scopes and must be explicitly granted. Standard capabilities do not bypass current workspace permissions.

Validation

Companion tests PR: https://github.com/Windshiftapp/core-tests-internal/pull/54

Passed locally using the companion test worktree:

../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/aitooladapter ./internal/aitools ./internal/agentskills ./internal/standardagent ./internal/wscli ./internal/restapi/v2
../core-tests/overlay.sh . -- -tags=test -race -count=1 ./internal/aitooladapter ./internal/aitools ./internal/agentskills ./internal/standardagent ./internal/wscli ./internal/restapi/v2
../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestTest(Catalog|Run|Management)|TestBDD|TestRouteClassification' ./tests
../core-tests/overlay.sh . -- -tags=test -race -count=1 -run 'TestAgentQA' ./tests
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN='<local PostgreSQL DSN>' ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA' ./tests
../core-tests/overlay.sh . -- -tags=test -count=1 -run 'Test.*(TestCase|TestManagement|TestCoverage|TestReport|BDD|Standard|Skill|AgentProfile)' ./internal/repository ./internal/handlers
make openapi-v2-check
bash scripts/check-deprecated-v1-api.sh
golangci-lint run ./...
git diff --check
go build ./cmd/ws

The scripted-model test exercises two real Standard bindings, saved skill/page activation, persisted malformed drafts, independent rejection/repair, and parent-run lineage. It does not claim live-model behavioral qualification.

The initial regression tests failed on the original code because the tools did not exist and authored steps were rejected by the API. The companion also updates a mandatory-tool assertion and two stale audit-helper tests needed to compile the affected aitools suite.

Broader services-package testing was attempted but is blocked by pre-existing private-suite drift: a services→testutils/factory→services import cycle, followed by outdated timer/action service signatures when that cycle is locally repaired. Those unrelated migrations are not included. Full Go, browser, frontend, and live-model suites were not run. No UI source changed. Full Go lint used existing local generated frontend assets only to satisfy main.go's embed requirement; no frontend rebuild is claimed.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Reviewed feat/wi-2184-agent-qa together with the companion tests PR (core-tests-internal#54). I read the diff and the surrounding code but did not run the suites, so treat the findings below as code-reading observations.

Overall this is coherent and the contract decisions in the description are actually reflected in the code. The 404-not-403 envelope, the "authorize every reference before any write" preflight, the atomic edit validation, step-identity preservation, the "cannot remove steps with recorded results" guard, and the no-fan-out effectiveness denominator all look right. The companion tests exercise real HTTP/MCP/Standard paths and assert exact denial contracts rather than restating the implementation. The capability_groups forwarding fix on profile creation is a good catch.

Findings, roughly by severity.

Medium

  1. Two parallel skill-snapshot paths. Dispatcher.snapshotSkills (internal/standardagent/dispatcher.go:767) re-implements BindingService.enabledSkillsForBinding + skillGrants (internal/services/binding_service.go:1136/:1168), but writes to ProfileSnapshot.Skills while coding agents write the same data to RunGrants.Skills (whose SkillFor helper is currently unused). The two also disagree on failure handling: coding agents log and omit a bad skill, Standard fails the whole run on a page-ref/metadata error. Please consolidate on one helper and one representation so the trust/size rules can't drift apart.

  2. Unbounded snapshot size. A single activation can render up to MaxActivationBytes (256 KiB), and nothing caps how many skills a binding may attach, so agent_runs.profile_snapshot_json can reach multi-MB per run and is parsed on every execute. Consider a cap on attached skills and/or total activation bytes.

  3. Dead code. TestCaseService.UpdateBDDContent (internal/services/test_case_bdd_service.go:93) has no callers left after UpdateCase moved Gherkin handling into cases.Update. Remove it or add a note on why it stays.

  4. Effectiveness pagination is offset over a mutable ranking. ORDER BY findings DESC, defects DESC, id LIMIT ? OFFSET ? (internal/repository/test_qa_repository.go:140) can shift rows when new executions land between pages, yet SKILL.md tells the agent to "page until total is exhausted". Consider keyset pagination, or document that the ranking is a point-in-time sample.

Low

  1. get_agent_skill is advertised where it can't work. It's registered in the global Default registry under the mandatory CapabilityReadComment group (internal/aitools/skills.go), so it appears in MCP and in-product chat tool lists where it always returns skill not found. Gate registration to the Standard source or document the no-op.

  2. Misleading denial subject. testAuthoringResult (internal/aitools/test_authoring.go:55) collapses every not-found/forbidden error to "test case not found", including get_item_test_coverage (an item) and get_test_effectiveness (a workspace/report). A per-tool message or a generic not found would keep the agent from reporting the wrong subject on a denial.

  3. Bulk preflight skips authored-content checks. BulkCases passes nil steps to validateAuthoredContent (internal/services/test_qa_application_service.go:148), so BDD-with-steps and >100-step payloads are only caught per item. That's consistent with "references are checked before any write", but the asymmetry is easy to miss — a comment would help.

  4. The new workspace-existence check in require needs a why-comment. SELECT EXISTS(SELECT 1 FROM workspaces ...) (internal/services/test_management_application_service.go:315) looks redundant next to HasWorkspacePermission, but it isn't: a system admin passes the permission check for a nonexistent workspace, so the query is what preserves the 404 contract. Worth one line saying so, otherwise it reads like an unnecessary round-trip on every test-management call.

  5. Label existence is validated outside the write transaction. A concurrent label delete surfaces as an FK error instead of a clean validation error. Minor.

Nits

  • UpdateCase parses Gherkin with BuildScenarioSpec, then cases.Update parses it again; the validation contract now lives in two places.
  • list_test_cases now defaults to spanning all folders (previously root-only). It's documented in the tool description, but it is a behavior change for existing chat/MCP agents.

Test gaps (companion PR)

  • No coverage for the MaxTestCaseSteps / MaxTestCaseLabels (100) limits or duplicate label_ids.
  • No test that bulk update rejects duplicate test_case_ids.
  • No cross-workspace (non-global) milestone_id denial for effectiveness.
  • No assertion that a BDD case rejects steps on update, or that format is immutable via update.

None of these block the core contract. Items 1–3 are the ones I'd want resolved or explicitly deferred before merge.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Addressed the review in 227d331f (core) and c3086c06 (companion tests PR #54), pushed to origin in both repositories:

  1. Snapshot consolidation: coding and Standard admission both use agentskills.SnapshotBinding and persist the same RunGrants.Skills representation. Standard no longer copies skill bodies into ProfileSnapshot. get_agent_skill uses RunGrants.SkillFor. Both loaders now fail closed for invalid metadata, unavailable saved pages or invalid/oversized activation content; coding runs no longer silently omit a potentially required standard. Remote claims use the frozen grants without reloading current skills.
  2. Aggregate bounds: at most 32 enabled skills, with a 256 KiB / 65536 estimated-token limit on the complete serialized grant set (including metadata and JSON escaping). The repository reads at most 33 rows to detect oversized legacy bindings. Both admission and frozen-grant execution validate the aggregate. Tests cover the count boundary, aggregate tokens/Unicode/escaping, read failures and no partial snapshots.
  3. BDD cleanup: removed UpdateBDDContent and the unused intermediate activation-error field. Gherkin is parsed once in the atomic update path. Source-only edits synchronize the scenario title; explicit titles still win.
  4. Pagination: retained offset pagination but removed the skill's claim that paging yields an exhaustive ranking under concurrent activity. The tool, API definition, OpenAPI and skill now explicitly state that each page samples a mutable ranking; fixed timestamps do not freeze result edits or defect links. Agents must deduplicate IDs and report sampling gaps. A durable immutable reporting snapshot is not introduced in this PR.
  5. Tool advertisement: source-aware registry admission excludes get_agent_skill from MCP and chat lists/executors while keeping it in the mandatory Standard preset. MCP discovery and chat admission tests assert this.
  6. Denials: coverage uses item not found; effectiveness uses workspace not found. Case tools retain their case-specific denial. Permission and existence denials remain indistinguishable per tool, with exact assertions.
  7. Bulk preflight: added the requested comment explaining that all references are preflighted, while authored-content validation is intentionally per case so valid entries can succeed.
  8. Workspace existence: added the system-administrator bypass explanation beside the query.
  9. Concurrent label deletion: labels are rechecked in the write transaction and held with PostgreSQL FOR KEY SHARE; SQLite already serializes writers. Deterministic create/update regressions reproduced the old FK error before the fix and now assert ErrNotFound, rollback and no extra case. Both database engines pass.

The requested test gaps are covered: 100/101 steps and labels, duplicate labels, duplicate bulk-update IDs, foreign non-global milestones, BDD-with-steps updates, and immutable format. The all-folder default is intentional and now also called out in the setup guide's compatibility note.

Validation passed:

../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN='<local PostgreSQL DSN>' ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabelDeletionBeforeWriteReturnsNotFoundAndRollsBack|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -race -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabelDeletionBeforeWriteReturnsNotFoundAndRollsBack|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
make openapi-v2-check
golangci-lint run ./...
git diff --check

The previously documented broader services-suite compile drift remains outside this change; no full services/browser/frontend or live-model qualification is claimed.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Re-reviewed 227d331f together with the companion c3086c06. All the findings from my earlier review are addressed — nice work:

  • Shared skill snapshot — agentskills.SnapshotBinding/ValidateSnapshot now backs both runtimes, grants live only in RunGrants.Skills, and the duplicated helpers (enabledSkillsForBinding, skillGrants, skillsFromGrants, ActivationError) are gone.
  • Bounded snapshot — 32-skill cap plus an aggregate 256 KiB / 65536-token bound over the serialized set; the repo query reads one overflow sentinel, and the tests cover count, byte, token, Unicode and JSON-escaping budgets.
  • Dead code — UpdateBDDContent removed; Update now takes SyncBDDTitle, so the Gherkin parse happens once and the title still syncs when the patch doesn't override it.
  • Pagination — documented in the OpenAPI/contract metadata, the report Definition, the tool description and SKILL.md.
  • get_agent_skill gating — new Tool.Sources/Entry.AvailableFor filters chat, MCP, Standard and executor admission; the MCP smoke test asserts it isn't advertised.
  • Denial subjects — per-tool notFound messages (item not found, workspace not found).
  • Label race — re-checked inside the write transaction with FOR KEY SHARE on Postgres, with a regression test that deletes the label between preflight and Begin.
  • require comment and the bulk-preflight comment are in.

Core CI is green and the companion tests line up with the new Env.Grants shape.

New finding

The snapshot cap is neither enforced nor surfaced before a run starts. MaxSnapshotSkills/ValidateSnapshot are applied only at run start (agentskills.SnapshotBinding, ValidateSnapshot in execute/ResolveRunInputs). The attach path doesn't cap anything — ReplaceBindingSkills (internal/repository/workspace_agent_skill_repository.go:208) accepts any count — and ValidateStudioProfile (internal/services/agent_profile_service.go:521) has no skill check. So an admin can attach 40 skills, see the profile as Ready, and only find out when every run fails with ... exceeds 32 skills .... Consider enforcing the cap (and ideally the aggregate budget) when attaching skills and/or adding a ProfileValidationError, so the failure lands at configuration time with the same message.

This also matters for existing deployments: bindings that already carry more than 32 skills — which the new TestBindingSkillSnapshotQueryBoundsLegacyAttachments shows is expected — will start failing on their next run after deploy. A readiness error plus a release note would make that self-service.

Minor

  • Dead branch. SkillGrant.Error is no longer set anywhere in production (SnapshotBinding fails closed instead), so the skill activation unavailable branch in get_agent_skill (internal/aitools/skills.go:24) is unreachable outside the unit test. Either drop the field/branch or note it's kept for forward compatibility.
  • Per-label round trips. writeAuthoredContent now issues one existence / FOR KEY SHARE query per label (internal/services/test_case_authoring.go:55), on top of the per-label preflight in validateAuthoredContent, so a 50-case bulk update with 100 labels each is roughly 10k queries. A single SELECT id ... WHERE workspace_id = ? AND id IN (...) FOR KEY SHARE would validate and lock in one round trip. Not blocking, but worth batching given the bulk path.
  • Naming. ListEnabledForBinding now silently caps at MaxSnapshotSkills+1; a name like ListEnabledForBindingForSnapshot (or a stronger doc note) would keep future callers from treating it as a complete list.

The cap-surfacing item is the only one I'd want addressed or explicitly deferred; the rest are polish.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Addressed the new finding and all three polish comments in 4d2f51b9 (core) and d01364a3 (companion tests #54), both pushed to origin:

  1. Configuration/readiness: attachment replacement rejects more than 32 distinct IDs, counting disabled attachments too. Service preflight rejects oversized create/update requests before configuration mutation, and the repository enforces the same guard. Duplicate IDs do not consume extra slots. Studio readiness now calls the shared snapshot loader, so aggregate overflow, invalid metadata and unreadable saved pages produce skill_snapshot_invalid with the actual error and a recovery hint; activation returns 422. Existing Ready profiles with oversized legacy attachments are shown as needs_setup, not available. Aggregate content is checked at readiness rather than rejecting skill-library edits: a shared skill may be valid alone but exceed a particular binding's combined budget. Run-time validation remains the final guard against subsequent edits.
  2. Upgrade guidance: the shipped test-case-standard/setup.md and this PR's upgrade note explain that attachments are preserved, existing profiles should be validated before their next run, and admins can recover by detaching/disabling excess skills or shortening bodies/reviewed saved page snapshots, then validating/activating again. New replacement lists require at most 32 distinct IDs.
  3. Compatibility branch: SkillGrant.Error and the tool's unavailable branch are explicitly documented as compatibility with older persisted unavailable grants; new snapshots still fail closed instead of writing such grants.
  4. Label round trips: both preflight and transactional validation now use a single IN (...) read per case; the write-side query retains PostgreSQL FOR KEY SHARE, with deterministic ID ordering. The regression measured 200 label-validation reads on the previous code versus 2 now for 100 labels, for both create and update. Inserts are unchanged. The deletion-between-preflight-and-write rollback regression still passes.
  5. Naming: the capped reader is now ListEnabledForBindingSnapshot with an explicit incomplete-list/overflow-sentinel comment.

The new readiness tests failed on the previous implementation because over-budget bindings were Ready and 33-skill updates succeeded. They now cover exact boundaries, duplicate IDs, rejected settings remaining unchanged, aggregate overflow, previously Ready legacy bindings, 422 activation refusal, and recovery after reducing content/attachments.

Validation passed:

../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN='<local PostgreSQL DSN>' ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -race -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
golangci-lint run ./...
make openapi-v2-check
git diff --check

The previously documented broader services-suite compilation drift is unchanged. Full Go/browser/frontend and live-model suites were not run.

@github-actions

Copy link
Copy Markdown

This PR has merge conflicts that need to be resolved before it can be merged. Please rebase on the latest main branch.

@github-actions github-actions Bot removed the conflict label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Merge conflicts have been resolved.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Reviewed the current implementation and prior review follow-ups, then merged current main into both branches:

  • Core: 86d5ea1a
  • Companion tests (#54): 87abe647

Resolved the audit-registry and test route-classification conflicts by retaining both branches’ entries. Authorization, authored-content transactions, bulk preflight, bounded retrieval/reporting, shared skill snapshots, readiness limits, and source-specific tool admission were reviewed. No additional blocking findings. GitHub now reports this PR as MERGEABLE; refreshed CI is running.

Passed locally:

../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
../core-tests/overlay.sh . -- -tags=test -count=1 -run "TestAgentQA|TestTest(Catalog|Run|Management)|TestBDD|TestRouteClassification|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repository
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN="<local PostgreSQL DSN>" ../core-tests/overlay.sh . -- -tags=test -count=1 -run "TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -count=1 -run "Test.*(TestCase|TestManagement|TestCoverage|TestReport|BDD|Standard|Skill|AgentProfile)" ./internal/services ./internal/repository ./internal/handlers
make openapi-v2-check
make lint
git diff --check
go build -o /private/var/folders/k6/vw61445n0xx1zkdrb7g7kb1r0000gn/T/opencode/ws-pr518 ./cmd/ws

The previously reported services-suite compilation drift is resolved with current companion main: the focused services suite above passes. The full pre-commit hook also passed, including OpenAPI v1/v2, schema guards, npm run check, npm run typecheck, and npm run check:inline-styles. The first commit attempt stopped because frontend dependencies were absent; npm ci with Node 24.18.0/npm 11.16.0 resolved that environment issue. Typecheck reports zero errors and three warnings.

Race validation is still running. Full Go, frontend unit/browser, and live-model suites were not run.

@stefan-ernst

Copy link
Copy Markdown
Contributor Author

Race validation also passed:

../core-tests/overlay.sh . -- -tags=test -race -count=1 -run "TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repository

Both packages passed with no race reports. Refreshed GitHub CI remains under observation.

@stefan-ernst
stefan-ernst merged commit 1c41ae7 into main Nov 16, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant