Repository navigation
feat: enable agent-driven QA workflows (WI-2184) - #518
Conversation
|
Reviewed Overall this is coherent and the contract decisions in the description are actually reflected in the code. The 404-not-403 envelope, the "authorize every reference before any write" preflight, the atomic edit validation, step-identity preservation, the "cannot remove steps with recorded results" guard, and the no-fan-out effectiveness denominator all look right. The companion tests exercise real HTTP/MCP/Standard paths and assert exact denial contracts rather than restating the implementation. The Findings, roughly by severity. Medium
Low
Nits
Test gaps (companion PR)
None of these block the core contract. Items 1–3 are the ones I'd want resolved or explicitly deferred before merge. |
|
Addressed the review in
The requested test gaps are covered: 100/101 steps and labels, duplicate labels, duplicate bulk-update IDs, foreign non-global milestones, BDD-with-steps updates, and immutable format. The all-folder default is intentional and now also called out in the setup guide's compatibility note. Validation passed: ../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN='<local PostgreSQL DSN>' ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabelDeletionBeforeWriteReturnsNotFoundAndRollsBack|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -race -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabelDeletionBeforeWriteReturnsNotFoundAndRollsBack|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
make openapi-v2-check
golangci-lint run ./...
git diff --checkThe previously documented broader services-suite compile drift remains outside this change; no full services/browser/frontend or live-model qualification is claimed. |
|
Re-reviewed
Core CI is green and the companion tests line up with the new New findingThe snapshot cap is neither enforced nor surfaced before a run starts. This also matters for existing deployments: bindings that already carry more than 32 skills — which the new Minor
The cap-surfacing item is the only one I'd want addressed or explicitly deferred; the rest are polish. |
|
Addressed the new finding and all three polish comments in
The new readiness tests failed on the previous implementation because over-budget bindings were Ready and 33-skill updates succeeded. They now cover exact boundaries, duplicate IDs, rejected settings remaining unchanged, aggregate overflow, previously Ready legacy bindings, 422 activation refusal, and recovery after reducing content/attachments. Validation passed: ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN='<local PostgreSQL DSN>' ../core-tests/overlay.sh . -- -tags=test -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -race -count=1 -run 'TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshotQueryBoundsLegacyAttachments' ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
golangci-lint run ./...
make openapi-v2-check
git diff --checkThe previously documented broader services-suite compilation drift is unchanged. Full Go/browser/frontend and live-model suites were not run. |
|
This PR has merge conflicts that need to be resolved before it can be merged. Please rebase on the latest |
|
Merge conflicts have been resolved. |
|
Reviewed the current implementation and prior review follow-ups, then merged current main into both branches:
Resolved the audit-registry and test route-classification conflicts by retaining both branches’ entries. Authorization, authored-content transactions, bulk preflight, bounded retrieval/reporting, shared skill snapshots, readiness limits, and source-specific tool admission were reviewed. No additional blocking findings. GitHub now reports this PR as MERGEABLE; refreshed CI is running. Passed locally: ../core-tests/overlay.sh . -- -tags=test -count=1 ./internal/agentskills ./internal/aitooladapter ./internal/aitools ./internal/standardagent ./internal/mcp ./internal/restapi/v2 ./internal/wscli
../core-tests/overlay.sh . -- -tags=test -count=1 -run "TestAgentQA|TestTest(Catalog|Run|Management)|TestBDD|TestRouteClassification|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repository
TEST_DB_TYPE=postgres TEST_POSTGRES_DSN="<local PostgreSQL DSN>" ../core-tests/overlay.sh . -- -tags=test -count=1 -run "TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repository
../core-tests/overlay.sh . -- -tags=test -count=1 -run "Test.*(TestCase|TestManagement|TestCoverage|TestReport|BDD|Standard|Skill|AgentProfile)" ./internal/services ./internal/repository ./internal/handlers
make openapi-v2-check
make lint
git diff --check
go build -o /private/var/folders/k6/vw61445n0xx1zkdrb7g7kb1r0000gn/T/opencode/ws-pr518 ./cmd/wsThe previously reported services-suite compilation drift is resolved with current companion main: the focused services suite above passes. The full pre-commit hook also passed, including OpenAPI v1/v2, schema guards, Race validation is still running. Full Go, frontend unit/browser, and live-model suites were not run. |
|
Race validation also passed: ../core-tests/overlay.sh . -- -tags=test -race -count=1 -run "TestAgentQA|TestMCP_Smoke_ListTools|TestTestCaseLabel|TestBindingSkillSnapshot" ./tests ./internal/repositoryBoth packages passed with no race reports. Refreshed GitHub CI remains under observation. |
Summary
Delivers WI-2184 and its seven child stories (WI-2185–WI-2191):
ws test case create|editwith flags or JSON file/stdin;ws test standardexports the bundled skill, starter workspace-page standard, and setup/worked-example guide.get_agent_skill. Profile creation now preserves requested capability groups.Review follow-up
agentskills.SnapshotBinding, store onlyRunGrants.Skills, and fail closed rather than silently omitting a required skill. The whole serialized skill set is capped at 32 skills, 256 KiB and 65536 estimated tokens, including metadata/escaping; bounded queries detect oversized legacy bindings.Second review follow-up / upgrade note
skill_snapshot_invalidwith the underlying error and a recovery hint; activation returns 422. Previously Ready legacy bindings showneeds_setupinstead of being advertised as available.ListEnabledForBindingSnapshotand documentedSkillGrant.Erroras compatibility with older persisted unavailable grants.Contract decisions
inactivelifecycle; no new hard-delete tool.[since, until), distinct same-workspace builtin Bug links from failed results, and finding executions without link fan-out. Zero-execution cases have null rates; missing links are not evidence of ineffectiveness.tests:writeremains absent from default coding-agent scopes and must be explicitly granted. Standard capabilities do not bypass current workspace permissions.Validation
Companion tests PR: https://github.com/Windshiftapp/core-tests-internal/pull/54
Passed locally using the companion test worktree:
The scripted-model test exercises two real Standard bindings, saved skill/page activation, persisted malformed drafts, independent rejection/repair, and parent-run lineage. It does not claim live-model behavioral qualification.
The initial regression tests failed on the original code because the tools did not exist and authored steps were rejected by the API. The companion also updates a mandatory-tool assertion and two stale audit-helper tests needed to compile the affected aitools suite.
Broader services-package testing was attempted but is blocked by pre-existing private-suite drift: a services→testutils/factory→services import cycle, followed by outdated timer/action service signatures when that cycle is locally repaired. Those unrelated migrations are not included. Full Go, browser, frontend, and live-model suites were not run. No UI source changed. Full Go lint used existing local generated frontend assets only to satisfy main.go's embed requirement; no frontend rebuild is claimed.