From 2c656a681701ec8f91bf13be418c8e9b9b633c19 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adam=20Wr=C3=B3bel?= Date: Mon, 3 Aug 2026 16:09:13 +0200 Subject: [PATCH 01/11] feat(plugin): add SpecFlow 2.0 spec-refinement skills and oracle library Adds the local, plugin-only refinement loop: independent subagents simulate building a spec under different adversarial lenses, deterministic scripts merge and rank what they disagree about, and the user resolves only the decisions that need a human. No backend, no third party, no network in the measurement path. Four new published skills (specflow-refine, -simulate, -resolve, -contracts), two reworked (-analysis emits machine-checkable dimensions; -planning now runs after refinement rather than before), one repurposed (-report), and specflow-mutate as internal QA in .claude/skills. Orchestration is prose; every count, ranking and verdict is code. The oracle library is stdlib-only so the plugin needs no pip install. Its core is the totality gate: a prose blocker list is partial by nature, so lenses must fill a total structure instead, and every admitted gap (inferred anchor, undefined_in_spec outcome, unhandled failure mode) has to be paid for with a matching blocker. Without that rule the escape hatches become a quiet way past the hard cells. Planning moved after refinement deliberately. A plan built on an ambiguous spec encodes one arbitrary reading of it, which is how sync_plan_to_workspaces made 1.0's variance signal unmeasurable. 37 stdlib unittest tests; each maps to a defect the loop must keep catching. Verified end-to-end against a two-lens fixture and from a simulated install location. backend/ and mcp_server/ are untouched. Two fixes outside the plugin: - .gitignore: `lib/` (Python build output) was excluding the whole oracle library, so the plugin would have shipped referencing absent scripts. - pyrightconfig.json: extraPaths only, no typeCheckingMode, so existing code keeps the defaults it was written against. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/specflow-mutate/SKILL.md | 110 ++++ .gitignore | 7 + plans/specflow-2.0/specflow-plugin-plan.md | 270 ++++++++++ plugins/specflow/.claude-plugin/plugin.json | 13 +- plugins/specflow/lib/README.md | 117 +++++ plugins/specflow/lib/specflow/__init__.py | 36 ++ plugins/specflow/lib/specflow/artifacts.py | 140 +++++ plugins/specflow/lib/specflow/concordance.py | 275 ++++++++++ plugins/specflow/lib/specflow/contracts.py | 259 ++++++++++ .../specflow/lib/specflow/jsonschema_mini.py | 220 ++++++++ plugins/specflow/lib/specflow/mutate.py | 228 ++++++++ plugins/specflow/lib/specflow/rank.py | 145 ++++++ plugins/specflow/lib/specflow/saturation.py | 146 ++++++ .../lib/specflow/schema/blocker.schema.json | 74 +++ .../specflow/schema/dimensions.schema.json | 192 +++++++ .../schema/interpretation.schema.json | 188 +++++++ plugins/specflow/lib/specflow/totality.py | 228 ++++++++ plugins/specflow/lib/specflow_cli.py | 489 ++++++++++++++++++ plugins/specflow/lib/tests/test_oracles.py | 450 ++++++++++++++++ .../skills/specflow-analysis/SKILL.md | 116 ++++- .../skills/specflow-contracts/SKILL.md | 94 ++++ .../skills/specflow-planning/SKILL.md | 91 +++- .../specflow/skills/specflow-refine/SKILL.md | 187 +++++++ .../specflow-refine/lenses/auth-boundaries.md | 40 ++ .../specflow-refine/lenses/concurrency.md | 35 ++ .../specflow-refine/lenses/data-lifecycle.md | 40 ++ .../specflow-refine/lenses/idempotency.md | 38 ++ .../skills/specflow-refine/lenses/ordering.md | 41 ++ .../specflow-refine/lenses/partial-failure.md | 39 ++ .../specflow/skills/specflow-report/SKILL.md | 61 +++ .../specflow/skills/specflow-resolve/SKILL.md | 101 ++++ .../skills/specflow-simulate/SKILL.md | 78 +++ pyrightconfig.json | 8 + 33 files changed, 4551 insertions(+), 5 deletions(-) create mode 100644 .claude/skills/specflow-mutate/SKILL.md create mode 100644 plans/specflow-2.0/specflow-plugin-plan.md create mode 100644 plugins/specflow/lib/README.md create mode 100644 plugins/specflow/lib/specflow/__init__.py create mode 100644 plugins/specflow/lib/specflow/artifacts.py create mode 100644 plugins/specflow/lib/specflow/concordance.py create mode 100644 plugins/specflow/lib/specflow/contracts.py create mode 100644 plugins/specflow/lib/specflow/jsonschema_mini.py create mode 100644 plugins/specflow/lib/specflow/mutate.py create mode 100644 plugins/specflow/lib/specflow/rank.py create mode 100644 plugins/specflow/lib/specflow/saturation.py create mode 100644 plugins/specflow/lib/specflow/schema/blocker.schema.json create mode 100644 plugins/specflow/lib/specflow/schema/dimensions.schema.json create mode 100644 plugins/specflow/lib/specflow/schema/interpretation.schema.json create mode 100644 plugins/specflow/lib/specflow/totality.py create mode 100644 plugins/specflow/lib/specflow_cli.py create mode 100644 plugins/specflow/lib/tests/test_oracles.py mode change 120000 => 100644 plugins/specflow/skills/specflow-analysis/SKILL.md create mode 100644 plugins/specflow/skills/specflow-contracts/SKILL.md mode change 120000 => 100644 plugins/specflow/skills/specflow-planning/SKILL.md create mode 100644 plugins/specflow/skills/specflow-refine/SKILL.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/auth-boundaries.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/concurrency.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/data-lifecycle.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/idempotency.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/ordering.md create mode 100644 plugins/specflow/skills/specflow-refine/lenses/partial-failure.md create mode 100644 plugins/specflow/skills/specflow-report/SKILL.md create mode 100644 plugins/specflow/skills/specflow-resolve/SKILL.md create mode 100644 plugins/specflow/skills/specflow-simulate/SKILL.md create mode 100644 pyrightconfig.json diff --git a/.claude/skills/specflow-mutate/SKILL.md b/.claude/skills/specflow-mutate/SKILL.md new file mode 100644 index 0000000..57c6e18 --- /dev/null +++ b/.claude/skills/specflow-mutate/SKILL.md @@ -0,0 +1,110 @@ +--- +name: specflow-mutate +description: Internal QA. Inject a known ambiguity into a spec, run the refinement loop against the damaged copy, and verify the loop both detects and localizes the defect. This is how we validate the instrument — not a customer feature. +argument-hint: "(optional) spec_dir kind — kind defaults to drop_constraint" +--- + +# SpecFlow Mutate (internal) + +**This skill is for SpecFlow engineers and is deliberately not shipped in the +marketplace plugin.** It validates the product; it is not part of it. + +## The problem it solves + +SpecFlow 2.0 rests on an unproven hypothesis: that divergence between simulated +builds tracks real specification defects. With no real builds, there is nothing +to check that against. + +So we manufacture the ground truth. Take a spec that refines cleanly, +programmatically remove a constraint or introduce a contradiction, and assert two +things: + +1. **Detection** — the loop raises a blocker. +2. **Localization** — the blocker lands on the requirement we damaged. + +Localization is the part that matters. A loop that complained about everything +would score perfectly on detection alone and be worthless. Detection without +localization is not a pass. + +This is also the regression suite for the whole pipeline. A mutation that stops +being caught is a concrete bug with a reproducible input. + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +``` + +## Available mutations + +| Kind | What it does | +|---|---| +| `drop_constraint` | deletes a line carrying a hard constraint (`must`, `never`, `exactly`, `at least`) | +| `contradict` | inverts a modal in place, so the spec asserts both a rule and its negation | +| `vague_quantity` | replaces a specific number with "several" | +| `drop_error_case` | deletes a line describing failure handling | +| `blur_enum` | replaces an explicit list of allowed values with "an appropriate value" | + +Selection is index-based, not random, so any run is reproducible from its +manifest alone. + +## What to do + +### 1. Establish a clean baseline + +Run `/specflow-refine` against the unmodified spec first and let it converge. A +mutation test is only meaningful against a spec the loop already handles — if the +baseline has open blockers, you cannot tell your injected defect from the noise. + +### 2. Inject one defect + +```bash +python3 "$SF" mutate apply \ + --spec-dir specs \ + --into /tmp/specflow-mutation \ + --kind drop_constraint \ + --index 0 +``` + +This copies the spec tree, applies exactly one mutation, and writes +`mutation-manifest.json` recording what was damaged and where. + +Read the manifest and confirm the mutation is genuinely a defect. Some lines +match the pattern but carry no real constraint, and deleting one of those tests +nothing. If so, increment `--index` and try again. + +### 3. Run the loop against the damaged copy + +Run `/specflow-refine` with `spec_dir` pointed at the mutated tree, and a +separate `outputs_dir` so you do not overwrite the baseline run. + +### 4. Verify + +```bash +python3 "$SF" mutate verify \ + --outputs /tmp/specflow-mutation/docs \ + --manifest /tmp/specflow-mutation/mutation-manifest.json +``` + +Exit 0 means detected and localized. Non-zero means the loop missed it. + +### 5. Interpret a miss honestly + +A miss is a finding about our product, so resist explaining it away. Work out +which it is: + +- **The lens set has a blind spot.** No lens attacks the class of defect that was + injected. Fix: add or sharpen a lens. +- **The artifact does not force the question.** The structure let the lens skip + the damaged area. Fix: extend the schema or the totality checks — this is the + strongest kind of fix, because it applies mechanically to every future run. +- **The mutation was not really a defect.** The spec determined the value + elsewhere, so nothing was lost. Not a miss; pick another line. + +Record misses. A mutation that used to be caught and now is not is a regression, +and it is the cheapest signal we have that a prompt change made the loop worse. + +## Coverage + +One mutation proves one thing. Sweep the kinds, and several indices per kind, to +say anything general about the loop's sensitivity. Each run is a full fan-out, so +sweeps are the expensive part of developing this product — budget for them +deliberately rather than running them ad hoc. diff --git a/.gitignore b/.gitignore index cb8b9e2..a97fe95 100644 --- a/.gitignore +++ b/.gitignore @@ -30,6 +30,13 @@ wheels/ .installed.cfg *.egg venv/ + +# The SpecFlow plugin ships its oracle library at plugins/specflow/lib. The +# generic `lib/` rule above targets Python build output and would silently +# exclude it — the plugin would then publish with skills referencing scripts +# that are not in the repo. +!plugins/specflow/lib/ +!plugins/specflow/lib/** ENV/ env/ .venv diff --git a/plans/specflow-2.0/specflow-plugin-plan.md b/plans/specflow-2.0/specflow-plugin-plan.md new file mode 100644 index 0000000..7001a37 --- /dev/null +++ b/plans/specflow-2.0/specflow-plugin-plan.md @@ -0,0 +1,270 @@ +# SpecFlow 2.0 — Plugin Plan (final) + +**Status**: DRAFT for approval +**Date**: 2026-08-03 +**Supersedes**: `PLAN.md` and `PLUGIN-SKILLS.md` in this directory (earlier drafts, safe to delete once this is approved) +**Shape**: A Claude Code marketplace plugin. No backend, no MCP server, no Agent SDK, no hosted anything. + +--- + +## 1. The flow + +### Today (README §"Get started") + +``` +specs/ → check_specification_completeness → run_planning → run_generation + (local skill, free) (local skill, free) (backend, 2–8 hrs, ~$400) +``` + +### SpecFlow 2.0 + +``` +specs/ → /specflow-analysis → /specflow-refine → /specflow-planning + (gap detection) (the product) (now trustworthy) + ↕ + you, resolving + ranked blockers +``` + +**Yes — one new user-facing verb.** That is the whole UX change. `run_generation` is replaced by `/specflow-refine`, and everything the backend used to do dissolves into subagents inside that one skill. + +### The one correction: planning moves *after* refine + +You had it as analysis → planning → refine. I'd swap the last two, for the reason that broke 1.0's measurement. + +**A plan is downstream of the spec.** If the spec is ambiguous, the plan is *one arbitrary resolution* of that ambiguity — and once it exists it anchors everything after it. That is precisely what `sync_plan_to_workspaces` (`workflow_steps.py:700`) did: one plan copied to all N workspaces, so the largest interpretation step ran exactly once and its arbitrariness became invisible to the statistics. + +Planning before refining reintroduces that defect: you'd be refining against a spec whose ambiguity has already been silently resolved by the planner. + +So planning takes on **two distinct roles**: + +| Role | When | Why | +|---|---|---| +| **Internal, per-lens** | Inside each refine round | Attempting to sequence work is a strong forcing function — you cannot phase what you don't understand. Divergent phase decomposition across lenses *is* a blocker signal. | +| **Final, user-facing** | After refinement converges | One plan, generated from a spec whose ambiguities are resolved. Now worth trusting. | + +This is also the better product story: *refine until the spec is unambiguous, then the plan you get is reliable.* Planning becomes the reward rather than a prerequisite. + +`/specflow-planning` still runs standalone whenever the user wants — nothing stops them. But the documented happy path puts it last. + +--- + +## 2. What `/specflow-refine` actually does + +One skill, owning a loop. Each round: + +**1. Fan out.** Spawn N lens subagents **in a single message** so they run concurrently. Each gets one adversarial lens and the spec. Blind to each other — no interpreter sees another's output, and there is no shared plan. Fresh context each. + +**2. Each lens produces a *total* artifact.** Not a prose blocker list — a filled structure: + +- the architectural dimensions (Parts A–D, every one "Pick exactly ONE") +- a state transition table +- a failure-mode matrix +- a phase decomposition (the internal planning role) +- blockers, each with a spec anchor + +**Totality is the forcing function that replaces building.** A prose list is partial by nature; a filled matrix is total by construction. An agent filling a state table *cannot skip* the cell for "payment succeeded + reservation expired." + +**3. Run the oracles** (scripts, not prose): schema conformance, totality check, contract validation. + +**4. Triage.** Cross-lens concordance, then rank by cost asymmetry. Concordance is *not* a score shown to the user — it decides what is worth your attention. If 5 of 6 lenses independently ask the same question, it's real. If 1 of 6 asks, it's probably pedantry. + +**5. Gate.** `AskUserQuestion` (native, supports multiSelect and previews). Prefer proposing over asking — "I'll assume X unless you object" clears most items at near-zero cost. Reserve blocking questions for consequential forks. + +**6. Write decisions back into the specs**, with traceability, and record them so later rounds don't re-ask. + +**7. Converge or loop.** Stop when a fresh round produces no new high-concordance blockers. Saturation, not a threshold — directly observable, no scoring, honest completion signal. + +### The lenses + +| Lens | Attacks | +|---|---| +| `concurrency` | simultaneous access, races, lock scope | +| `partial-failure` | half-completed operations, compensations, retries | +| `data-lifecycle` | migration, retention, deletion, backfill | +| `auth-boundaries` | who can do what to whose data | +| `idempotency` | replay, duplicate delivery, at-least-once | +| `ordering` | sequence assumptions, out-of-order arrival | + +These are the failure classes physical building surfaced and that naive "think about blockers" misses. **Lens count is the cost dial.** + +They ship as `lenses/*.md` assets, not as separate skills — nobody types "run the idempotency lens." Marketplace entries should be things a user would actually invoke. + +### On "all the parallelism by subagents?" — yes + +Spawned concurrently in one message, fresh context each, blind to each other. Two honest caveats: + +- **Subagents are Claude-only** (opus/sonnet/haiku/fable). No GPT-5.5, no GLM. The existing `recommended-models: openai/gpt-5.3-codex` frontmatter goes inert. Adversarial lenses replace vendor diversity — deliberate attack angles beat hoping three vendors have different blind spots — but it *is* a real reduction. +- **N is tunable, and practical concurrency has limits.** Treat lens count as the cost/coverage dial and measure actual behavior at P2 rather than assuming all six run truly simultaneously. + +--- + +## 3. Prose for orchestration, code for oracles + +The architecture in one line. + +**Orchestration is prose** — a skill spawns subagents, sequences rounds, decides when to ask you. Few steps, judgment calls, fine for a model. + +**Oracles are code.** An oracle's entire value is that it is *not* a language model. "Verify the state table is complete" as an instruction is advisory; a script that exits non-zero on a blank cell is a forcing function. + +**Code ships with the plugin.** A skill is a directory — `SKILL.md` plus assets and executables. Already proven in this repo: `.claude/skills/pr-loc-breakdown/` ships `count_py_loc.py` and the skill runs it via Bash. + +| Script | Job | +|---|---| +| `validate_artifact.py` | JSON Schema conformance — malformed fails loudly, not silently | +| `check_totality.py` | Every dimension filled, every matrix cell present. **The gate.** | +| `contracts_oracle.py` | Real SQL DDL / OpenAPI / type-def validators | +| `concordance.py` | Anchor-scoped cross-lens agreement | +| `rank_blockers.py` | Cost-asymmetry ordering, dedup against resolved | +| `saturation.py` | The stop rule | + +~1–2k LOC of pure functions over files. No server, no persistent state, no network — assertable in a test. + +### The asset we already have + +`specflow-analysis/SKILL.md` is 488 lines and already contains the total-artifact framework: + +- **Part A** — 6 universal dimensions, each "Pick exactly **ONE**" +- **Part B** — technology-specific dimensions by project type +- **Part C** — project-specific dimensions, headed *"Discover additional variance sources"* +- **Part D** — micro-level consistency locks, *"AGGRESSIVE ENFORCEMENT"*, "Must specify ALL" + +2.0 does not invent this. It (a) replicates the fill across independent lenses, (b) makes the fill machine-checkable, (c) diffs the filled values. **Divergence on a locked dimension is a named, localized spec ambiguity** — no scoring involved. + +--- + +## 4. Why no backend, no MCP server, no Agent SDK + +**The Agent SDK is Claude Code packaged as a library** — built-in tools, agent loop, context management, subagents, permissions. It supplies the **harness only; deployment is yours.** That is exactly what `backend/app/services/claude_code.py` + workspace pool + NFS + K8s exist to do: run the harness *somewhere other than the user's machine*. + +Once the product runs in the user's IDE, **their Claude Code session is the harness.** Nothing to host, so nothing the SDK provides is needed. Same for `mcp_server/` — it exists to precheck and call a backend that won't exist. + +Consequences worth stating plainly: + +- **COGS → ~0.** Runs on the user's own subscription. This shifts the business model from consumption to licensing — a bigger change than the 10x we started from. +- **Zero egress, no server to audit.** Strictly stronger than the compliance story that killed P10Y. +- **State = files in the user's repo.** Git-tracked, human-readable, human-editable. No Firestore, SQLite, or NFS. Better than an opaque database the user can't inspect. +- **HITL becomes possible at all.** 1.0's own constraint was "no opportunity to prompt the user" mid-run. The HITL pivot *requires* the local architecture. + +--- + +## 5. Skill inventory + +**7 published, 4 net new.** + +| Skill | Status | Role | +|---|---|---| +| `specflow-analysis` | extend | Gap detection. Add JSON output + totality gate. Drop Part F (`INTEGRATION_TESTS_READY` — it exists to tell the backend whether to run E2E). | +| `specflow-refine` | **new** | The orchestrator and entry point. §2. | +| `specflow-simulate` | **new** | Single-lens run, no loop. Cheap first touch, natural demo, immediate value. | +| `specflow-resolve` | **new** | Walk ranked blockers, write decisions into the spec files with traceability. | +| `specflow-contracts` | **new** | Emit data model + API contract as real schemas; validate with real validators. Keeps the compiler, drops the application. | +| `specflow-planning` | rework | Per-lens internally; final artifact after convergence. §1. | +| `specflow-report` | repurpose `specflow-compare-variants` (255 lines) | Current state: resolved, open, ranked. **Counts, never a score.** | + +Retired: `specflow-diagnose` (156 lines, reads backend failure state — nothing to salvage). +Internal only: `specflow-mutate` → `.claude/skills/`, not published. It's our QA harness for validating the loop, not a customer feature. + +**`specflow-resolve` is deliberately separate from finding blockers.** Applying decisions to spec files is an edit operation with its own hazards — don't clobber the user's prose, keep traceability, record resolutions for dedup. A loop that only *reports* blockers leaves all the work with the user, which isn't autonomous refinement. + +--- + +## 6. Plugin layout + +``` +plugins/specflow/ + .claude-plugin/plugin.json → v0.2.0, keywords += spec-refinement, blocker-detection + lib/ # shared oracles — ONE copy + schema/ + interpretation.schema.json + dimensions.schema.json # Parts A–D, machine-readable — the source of truth + blocker.schema.json + validate_artifact.py + check_totality.py + contracts_oracle.py + concordance.py + rank_blockers.py + saturation.py + skills/ + specflow-analysis/ SKILL.md + specflow-planning/ SKILL.md + specflow-refine/ SKILL.md lenses/*.md + specflow-simulate/ SKILL.md + specflow-resolve/ SKILL.md + specflow-contracts/ SKILL.md + specflow-report/ SKILL.md +``` + +**Shared `lib/`, not per-skill copies.** `concordance.py` is needed by two skills, `validate_artifact.py` by four. Copies drift — the single-source-of-truth rule in CLAUDE.md applies to shipped scripts too. + +Moving the dimensions framework into `lib/schema/dimensions.schema.json` does two things at once: shrinks the 488-line skill, and makes the framework machine-checkable. + +⚠️ **P0 open item.** I have not verified the supported mechanism for a skill to resolve a path *above* its own directory to reach `lib/`. Do not build on an assumed environment variable — check the plugin docs first. Fallback is a thin per-skill shim over one implementation. 15 minutes, and it shapes the layout. + +--- + +## 7. Build order + +Each phase leaves the plugin installable and prior phases working. + +| Phase | Work | Exit criterion | +|---|---|---| +| **P0** | Verify plugin-root path resolution. Move the four SKILL.md files from `mcp_server/services/skills/` into `plugins/specflow/skills/`. Drop the `<>` substitution layer — skills take arguments directly. | `/specflow-analysis` runs from the installed plugin with no MCP server | +| **P1** | `lib/schema/*.json` + `validate_artifact.py` + `check_totality.py`. Extend `specflow-analysis` to emit JSON and call the gate. | Totality check rejects a deliberately-blank dimension | +| **P2** | `specflow-simulate` + the six lens prompts. Single lens end-to-end on a real spec. | Artifact validates; blockers carry spec anchors; **measured cost and real concurrency confirmed** | +| **P3** | `contracts_oracle.py` + `specflow-contracts`. | Catches a planted contradiction as a schema impossibility | +| **P4** | `concordance.py` + `rank_blockers.py` + `specflow-refine` fan-out (no loop yet). | N lenses run concurrently; blockers ranked and deduped | +| **P5** | `specflow-resolve` + the `AskUserQuestion` gate. | A human resolves ranked blockers; specs updated with traceability | +| **P6** | `saturation.py` + the round loop. `specflow-report`. | Loop terminates on saturation, not a fixed count | +| **P7** | `specflow-mutate` (internal). | Injected ambiguity detected **and localized** to the mutated requirement | +| **P8** | Rework `specflow-planning` for per-lens + final roles. Retire `specflow-diagnose`. Bump to `0.2.0`, update README flow. | Marketplace install delivers the full 2.0 experience | +| **P9** | Delete `backend/`, `mcp_server/`, `server.py`, docker-compose, infra scripts. | No network I/O outside model calls, asserted in a test | + +**P2 and P7 are the gates.** P2 proves the economics and the concurrency assumption on a real spec. P7 proves the loop detects anything real. **Nothing is deleted until P7 is green** — the sequence front-loads cheap reversible work on purpose. + +--- + +## 8. What gets deleted (P9, not before) + +`backend/` (37.6k LOC app + 40.9k LOC tests), `mcp_server/`, `server.py`, `docker-compose.yml`, the K8s/NFS/Firestore/SQLite layer, `Dockerfile`, `scripts/init-mobile-sdk.sh`. + +Everything justified by *"run the harness on our infra"* or *"generated code takes hours and is irreplaceable"* — both premises are now false. Retry = rerun. Crash recovery = rerun. + +### Steel Commandments 2.0 (needs your ratification) + +The constitution rests on those same two premises. + +- **I–VI** (workspace sanctity, no-release-on-fail, archive-as-precondition, retry-reuses-workspace, no-background-touch) — **retire.** There are no workspaces. +- **VII–X** (state machine sole writer, forward-only checkpoints, transitions logged, invalid transitions raise) — **retire.** No state machine; state is files in the user's repo. +- **XI** — **retire**, superseded. + +Proposed replacements, each guarding a property 2.0 actually depends on: + +1. **No step performs network I/O beyond the model call.** Guards the compliance property that is now the product's main asset. +2. **No interpreter observes another interpreter's output.** Guards independence. +3. **Every verdict, count, and validation is produced by a script, never by a model.** Guards auditability — this is what makes output evidence rather than opinion. +4. **Artifacts are total or rejected.** The forcing function that replaces building. +5. **Samples are ephemeral and reproducible; never build machinery to preserve them.** The anti-pattern that produced ~9k LOC of preservation code. + +--- + +## 9. Risks + +| Risk | Severity | Handling | +|---|---|---| +| Simulated-build divergence may not track real spec defects | **High** | P7 mutation harness. This is the core product hypothesis and it is currently unproven. | +| Prose orchestration cannot guarantee a step ran | **High** | Artifact-passing (a stage's input is the prior stage's output file, so a skipped gate shows as a missing file) + non-zero-exit validators + hooks. **Cannot fully close** — this is the honest price of the architecture. | +| Self-reported blockers depend on agent introspection | **High** | An agent that silently assumes something won't report it. Divergence in the *total artifacts* is the objective backstop; do not build the report on self-reported blockers alone. | +| Integration-class defects only real building surfaces | **High** | §2's total artifacts + lenses + contract oracle recover much of it. Not all. Accepted, not solved. | +| Losing Cursor support | Medium | ⚠️ Skills and subagents are Claude Code only; `docs/IDE-SETUP.md` supports Cursor today. Either keep a thin shim or write parallel `.cursor/rules`. **Commercial decision — your call, not a technical blocker.** | +| No telemetry → slower iteration | Medium | For the compliance-sensitive buyer, not collecting telemetry is a feature. Substitute: artifacts live in the user's repo; ask design partners to share them. | +| Plugin-root path resolution for `lib/` | Medium | P0 verification. Per-skill shim as fallback. | +| Sales narrative weakens ("we build 3 prototypes") | Medium | Counter: runs on your own subscription, nothing leaves your machine, no third-party vendor, and per-requirement findings 1.0 could not produce. | + +--- + +## 10. Decisions needed from you + +1. **Ratify the planning/refine order swap** (§1) — or tell me to keep your original order and I'll note why it's weaker. +2. **Ratify Steel Commandments 2.0** (§8) — I proposed retiring all eleven and replacing them with five. That's your constitution; I won't edit it unilaterally. +3. **Cursor: keep or drop** (§9). diff --git a/plugins/specflow/.claude-plugin/plugin.json b/plugins/specflow/.claude-plugin/plugin.json index 0644ce6..a9f3044 100644 --- a/plugins/specflow/.claude-plugin/plugin.json +++ b/plugins/specflow/.claude-plugin/plugin.json @@ -1,12 +1,19 @@ { "name": "specflow", - "description": "Grid Dynamics SpecFlow Marketplace plugin", - "version": "0.1.0", + "description": "Refine specifications before you build. Independent subagents simulate building your spec under different adversarial lenses; deterministic checks rank what they disagree about; you resolve the decisions that matter and the answers are written back into the spec. Runs entirely locally — no backend, no third party, nothing leaves your machine.", + "version": "0.2.0", "author": { "name": "Grid Dynamics" }, "homepage": "https://github.com/griddynamics/specflow", "repository": "https://github.com/griddynamics/specflow", "license": "MIT", - "keywords": ["specflow", "spec-analysis", "implementation-planning"] + "keywords": [ + "specflow", + "spec-analysis", + "spec-refinement", + "blocker-detection", + "requirements", + "implementation-planning" + ] } diff --git a/plugins/specflow/lib/README.md b/plugins/specflow/lib/README.md new file mode 100644 index 0000000..222c71b --- /dev/null +++ b/plugins/specflow/lib/README.md @@ -0,0 +1,117 @@ +# SpecFlow oracles + +The deterministic half of the refinement loop. + +Orchestration is prose — skills spawning subagents, sequencing rounds, deciding +when to ask the user. Everything in this directory is code, because an oracle's +whole value is that it is **not** a language model. "Check the state table is +complete" as an instruction is advisory; a script that exits non-zero on an empty +cell is a forcing function. + +## Entry point + +One dispatcher, invoked by path from a skill: + +```bash +python3 "${CLAUDE_PLUGIN_ROOT}/lib/specflow_cli.py" [options] +``` + +`specflow_cli.py` sits outside the `specflow/` package deliberately: the package +stays a pure library, and the script owns the one fragile thing in a plugin — the +path bootstrap. It works from any working directory. + +| Command | Job | +|---|---| +| `new-round` | allocate the next round directory | +| `validate` | schema conformance + totality, nothing else | +| `round` | validate, merge, rank, decide whether to stop — the workhorse | +| `resolve` | record a decision so later rounds stop asking | +| `status` | current state, for reporting | +| `contracts` | model contradictions, and emitted SQL/API cross-checks | +| `mutate` | inject a known defect and verify it gets caught (internal QA) | + +Exit codes: `0` success, `1` checks failed, `2` bad usage. The non-zero on +failure is the point — a skill cannot quietly proceed past a gate that did not +pass. + +## Modules + +| Module | Responsibility | +|---|---| +| `jsonschema_mini.py` | JSON Schema validation, stdlib only | +| `artifacts.py` | on-disk layout and IO | +| `totality.py` | the forcing function — see below | +| `contracts.py` | structural contradictions in the model, and emitted-artifact checks | +| `concordance.py` | cross-lens agreement and located divergence | +| `rank.py` | cost-asymmetry ordering and ask/assume/note disposition | +| `saturation.py` | the stop rule | +| `mutate.py` | ambiguity injection and verification | +| `schema/` | the artifact contracts — source of truth | + +## Two policies worth knowing + +**Stdlib only.** This ships inside a marketplace plugin and runs on whatever +Python the user has. A `pip install` step turns a working skill into a support +ticket, so there is a hand-written JSON Schema validator instead of the +`jsonschema` package, and API contracts are validated as JSON rather than YAML. + +The validator's supported keyword set is closed: anything used in `schema/*.json` +is implemented, and anything unimplemented **raises** rather than passing +silently. A constraint that is quietly ignored is worse than no constraint. + +**Nothing here calls a model or the network.** Every count, ranking and verdict is +reproducible from the artifacts on disk. That is what lets the output be treated +as evidence rather than opinion, and it is what makes "nothing leaves the +machine" true of the measurement path and not just the storage. + +## Why totality matters most + +A real build compels decisions — you cannot run code past a point the spec left +undefined. Simulation has no such compulsion, so an agent asked "what would block +you?" produces a plausible list, not an exhaustive one: it finds the legible gaps +and skips the awkward ones. + +`totality.py` restores the compulsion structurally: + +1. Every dimension carries a real value, not an evasion (`TBD`, `unknown`, + `varies` are rejected). +2. Every state × event pair in a lifecycle has an outcome. +3. Every reference resolves to something that exists. +4. **Every escape hatch is paid for with a blocker.** + +(4) is the one that closes the loophole. An agent can always write +`inferred: true`, or `outcome: "undefined_in_spec"`, or +`spec_says: "nothing"` — those are legitimate answers, but only if the gap is +also *raised*. Without this check they become a silent way past the hard cells, +which is precisely the failure mode simulation is prone to. + +## Testing + +```bash +python3 plugins/specflow/lib/tests/test_oracles.py +``` + +Stdlib `unittest`, no pytest — same zero-dependency policy as the library. + +Each test corresponds to a defect the loop must keep catching, so a failure means +the product has become less able to find real specification gaps. The ones worth +knowing about: + +- the totality gate rejects a partial state matrix, an evasion value, an + unresolvable operation entity or foreign key, an unraised `inferred` anchor, and + a recommendation that is not one of the offered options; +- it *accepts* an admitted gap when a blocker was raised for it — the escape + hatch is legitimate, only skipping it silently is not; +- `contracts` catches a required-and-derived contradiction, a + mutual-required-reference cycle, an unguarded mutation, a missing table, and a + dangling `$ref`; +- `concordance` turns two lenses disagreeing on a Part A dimension into a ranked + blocker, and merges the same blocker found by two lenses into one with both + attributed; +- `rank` asks about blocking and irreversible decisions, assumes reversible ones, + and only notes a lone cosmetic finding; +- `saturation` converges on a dry round and treats a resolved blocker as seen; +- `mutate.verify` fails on detection without localization — a loop that + complained about everything must not pass; +- the schema validator raises on an unimplemented keyword rather than ignoring + it. diff --git a/plugins/specflow/lib/specflow/__init__.py b/plugins/specflow/lib/specflow/__init__.py new file mode 100644 index 0000000..3a8bf57 --- /dev/null +++ b/plugins/specflow/lib/specflow/__init__.py @@ -0,0 +1,36 @@ +"""SpecFlow oracles — the deterministic half of the refinement loop. + +Orchestration is prose (skills spawning subagents); everything in this package +is code. The split is deliberate: an oracle's whole value is that it is not a +language model. "Check the state table is complete" as an instruction is +advisory. A script that exits non-zero on an empty cell is a forcing function. + +Stdlib only, by policy. This ships inside a marketplace plugin and runs on +whatever Python the user has; a ``pip install`` step turns a working skill into +a support ticket. + +Entry point is ``../specflow_cli.py`` — deliberately outside this package, so +the package stays a pure library and the script owns the path bootstrap. +""" + +from . import jsonschema_mini +from . import artifacts +from . import totality +from . import contracts +from . import concordance +from . import rank +from . import saturation +from . import mutate + +__all__ = [ + "artifacts", + "concordance", + "contracts", + "jsonschema_mini", + "mutate", + "rank", + "saturation", + "totality", +] + +__version__ = "0.2.0" diff --git a/plugins/specflow/lib/specflow/artifacts.py b/plugins/specflow/lib/specflow/artifacts.py new file mode 100644 index 0000000..0f9967f --- /dev/null +++ b/plugins/specflow/lib/specflow/artifacts.py @@ -0,0 +1,140 @@ +"""On-disk layout for a refinement run. + +Everything lives in the user's own repo as readable JSON. That is deliberate: +the artifacts are the state, so there is no database to inspect, no server to +query, and the user can read, diff, and edit any of it with the tools they +already have. A refinement run is reviewable in a pull request. + + /refine/ + state.json round counter + saturation history + resolutions.json decisions made, cumulative + blockers.json current ranked list + round-01/ + interpretation.concurrency.json + interpretation.ordering.json + ... + contracts/ + schema.sql api.json types.json +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +REFINE_SUBDIR = "refine" +STATE_FILE = "state.json" +RESOLUTIONS_FILE = "resolutions.json" +BLOCKERS_FILE = "blockers.json" +CONTRACTS_SUBDIR = "contracts" +INTERPRETATION_PREFIX = "interpretation." + + +@dataclass(frozen=True) +class Layout: + """Resolved paths for one project's refinement run.""" + + outputs_dir: Path + + @property + def root(self) -> Path: + return self.outputs_dir / REFINE_SUBDIR + + @property + def state_path(self) -> Path: + return self.root / STATE_FILE + + @property + def resolutions_path(self) -> Path: + return self.root / RESOLUTIONS_FILE + + @property + def blockers_path(self) -> Path: + return self.root / BLOCKERS_FILE + + @property + def contracts_dir(self) -> Path: + return self.root / CONTRACTS_SUBDIR + + def round_dir(self, number: int) -> Path: + return self.root / f"round-{number:02d}" + + def interpretation_path(self, number: int, lens: str) -> Path: + return self.round_dir(number) / f"{INTERPRETATION_PREFIX}{lens}.json" + + def rounds(self) -> list[int]: + """Round numbers present on disk, ascending.""" + if not self.root.is_dir(): + return [] + numbers = [] + for entry in self.root.iterdir(): + if entry.is_dir() and entry.name.startswith("round-"): + suffix = entry.name.removeprefix("round-") + if suffix.isdigit(): + numbers.append(int(suffix)) + return sorted(numbers) + + def latest_round(self) -> int | None: + rounds = self.rounds() + return rounds[-1] if rounds else None + + def interpretations(self, number: int) -> list[Path]: + directory = self.round_dir(number) + if not directory.is_dir(): + return [] + return sorted(directory.glob(f"{INTERPRETATION_PREFIX}*.json")) + + +def layout_for(outputs_dir: str | Path) -> Layout: + return Layout(Path(outputs_dir)) + + +def read_json(path: Path) -> Any: + """Read JSON with a message that says which file is broken.""" + try: + return json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError: + raise FileNotFoundError(f"Not found: {path}") from None + except json.JSONDecodeError as exc: + raise ValueError(f"{path} is not valid JSON: {exc}") from None + + +def write_json(path: Path, payload: Any) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + ) + return path + + +def load_interpretations(layout: Layout, number: int) -> list[dict[str, Any]]: + """Load every lens artifact for a round, tagging each with its lens name.""" + loaded = [] + for path in layout.interpretations(number): + data = read_json(path) + if not isinstance(data, dict): + raise ValueError(f"{path} should contain an object, got {type(data).__name__}") + data.setdefault("lens", path.stem.removeprefix(INTERPRETATION_PREFIX)) + data["_path"] = str(path) + loaded.append(data) + return loaded + + +def load_state(layout: Layout) -> dict[str, Any]: + if not layout.state_path.exists(): + return {"rounds": [], "converged": False} + return read_json(layout.state_path) + + +def load_resolutions(layout: Layout) -> list[dict[str, Any]]: + """Decisions already made. Used to keep later rounds from re-asking.""" + if not layout.resolutions_path.exists(): + return [] + data = read_json(layout.resolutions_path) + return data.get("resolved", []) if isinstance(data, dict) else data + + +def resolved_ids(layout: Layout) -> set[str]: + return {r["blocker_id"] for r in load_resolutions(layout) if "blocker_id" in r} diff --git a/plugins/specflow/lib/specflow/concordance.py b/plugins/specflow/lib/specflow/concordance.py new file mode 100644 index 0000000..7220aa0 --- /dev/null +++ b/plugins/specflow/lib/specflow/concordance.py @@ -0,0 +1,275 @@ +"""Cross-lens agreement — the triage function. + +Multiplicity was never primarily about producing a number. Independent readings +give two things a single reading cannot: better recall (the union of what each +lens found) and a way to rank (agreement between lenses that could not see each +other's work). + +Human attention is the scarce resource in this design, so agreement is spent on +deciding *what to ask about*, not on scoring the spec. Nothing here is shown to +the user as a metric. + +Matching is anchored on the spec, not on names. Comparing entity or field names +globally would measure synonyms — one lens's ``User`` against another's +``Account``. Comparing within a requirement's scope reduces that to a small +local problem, which is why every artifact element carries a spec anchor. +""" + +from __future__ import annotations + +import re +from collections import defaultdict +from dataclasses import dataclass, field +from typing import Any + +# Dropped before comparing labels within an anchor's scope. +_STOPWORDS = frozenset({ + "a", "an", "the", "of", "for", "to", "in", "on", "at", "by", "with", + "is", "are", "be", "when", "if", "should", "must", "will", "does", +}) +_WORD = re.compile(r"[a-z0-9]+") + + +def normalize(text: str) -> frozenset[str]: + """Deterministic bag-of-words for within-scope comparison. + + Lowercase, split, drop stopwords, strip a trailing plural 's'. No embeddings + and no model call: the comparison has to be reproducible, and an LLM here + would inject judgment into the measurement. + """ + words = [] + for word in _WORD.findall(text.lower()): + if word in _STOPWORDS: + continue + words.append(word[:-1] if len(word) > 3 and word.endswith("s") else word) + return frozenset(words) + + +def _anchor_key(anchor: dict[str, Any] | None) -> str: + anchor = anchor or {} + parts = [str(anchor.get("file", "")), str(anchor.get("section", ""))] + return "::".join(p for p in parts if p) + + +@dataclass +class Divergence: + """One located disagreement between lenses.""" + + kind: str + where: str + detail: str + lenses: dict[str, str] = field(default_factory=dict) + + def as_dict(self) -> dict[str, Any]: + return { + "kind": self.kind, + "where": self.where, + "detail": self.detail, + "lenses": self.lenses, + } + + +@dataclass +class ConcordanceResult: + lens_count: int + blockers: list[dict[str, Any]] = field(default_factory=list) + divergences: list[Divergence] = field(default_factory=list) + coverage: dict[str, list[str]] = field(default_factory=dict) + + def as_dict(self) -> dict[str, Any]: + return { + "lens_count": self.lens_count, + "blockers": self.blockers, + "divergences": [d.as_dict() for d in self.divergences], + "coverage": self.coverage, + } + + +def _merge_blockers( + interpretations: list[dict[str, Any]] +) -> list[dict[str, Any]]: + """Union the blockers, recording which lenses independently raised each. + + Two passes: exact id collision first (ids are stable slugs, so the same + decision found twice should collide), then within-anchor label overlap to + catch the same gap described in different words. + """ + by_id: dict[str, dict[str, Any]] = {} + for interpretation in interpretations: + lens = interpretation.get("lens", "?") + for blocker in interpretation.get("blockers", []): + key = blocker.get("id") + if not key: + continue + existing = by_id.get(key) + if existing is None: + merged = dict(blocker) + merged["found_by"] = [lens] + by_id[key] = merged + elif lens not in existing["found_by"]: + existing["found_by"].append(lens) + # Keep the richest option set we have seen. + if len(blocker.get("options", [])) > len(existing.get("options", [])): + existing["options"] = blocker["options"] + existing["recommended"] = blocker.get("recommended", existing.get("recommended")) + + # Second pass: same anchor, overlapping wording, different id. + grouped: dict[str, list[dict[str, Any]]] = defaultdict(list) + for blocker in by_id.values(): + grouped[_anchor_key(blocker.get("spec_anchor"))].append(blocker) + + absorbed: set[str] = set() + for peers in grouped.values(): + for i, left in enumerate(peers): + if left["id"] in absorbed: + continue + left_words = normalize(left.get("title", "")) + for right in peers[i + 1:]: + if right["id"] in absorbed: + continue + right_words = normalize(right.get("title", "")) + if not left_words or not right_words: + continue + overlap = len(left_words & right_words) / len(left_words | right_words) + if overlap >= 0.6: + for lens in right["found_by"]: + if lens not in left["found_by"]: + left["found_by"].append(lens) + left.setdefault("also_known_as", []).append(right["id"]) + absorbed.add(right["id"]) + + return [b for b in by_id.values() if b["id"] not in absorbed] + + +def _dimension_divergences( + interpretations: list[dict[str, Any]] +) -> list[Divergence]: + """Different lenses locking the same dimension to different values. + + This is the strongest and cheapest signal available: Part A and Part D are + enums or short strings, so disagreement is unambiguous — no matching + heuristics, no judgment. A dimension the spec determined would not diverge. + """ + found: list[Divergence] = [] + picks: dict[str, dict[str, str]] = defaultdict(dict) + + for interpretation in interpretations: + lens = interpretation.get("lens", "?") + dimensions = interpretation.get("dimensions") or {} + + for name, entry in (dimensions.get("part_a") or {}).items(): + if isinstance(entry, dict): + value = entry.get("value") + if value is None and name == "scope_boundaries": + value = " | ".join(sorted(entry.get("in_scope", []))) + if value is not None: + picks[f"part_a.{name}"][lens] = str(value) + + part_d = dimensions.get("part_d") or {} + for group in ("naming", "patterns"): + for name, value in (part_d.get(group) or {}).items(): + if isinstance(value, str): + picks[f"part_d.{group}.{name}"][lens] = value + + for where, by_lens in sorted(picks.items()): + distinct = set(by_lens.values()) + if len(distinct) > 1: + found.append( + Divergence( + kind="dimension", + where=where, + detail=( + f"{len(distinct)} different values across {len(by_lens)} lenses — " + "the spec does not determine this" + ), + lenses=dict(sorted(by_lens.items())), + ) + ) + return found + + +def _phase_divergences(interpretations: list[dict[str, Any]]) -> list[Divergence]: + """Disagreement about how to sequence the work. + + Attempting to phase a build is its own forcing function — you cannot + sequence work you do not understand. Lenses that decompose the same spec + very differently are telling you the spec underdetermines the work. + """ + counts = { + i.get("lens", "?"): len(i.get("phases", [])) + for i in interpretations + if i.get("phases") + } + if len(counts) < 2: + return [] + low, high = min(counts.values()), max(counts.values()) + if low and high >= low * 2: + return [ + Divergence( + kind="decomposition", + where="phases", + detail=( + f"phase counts range {low}-{high} — lenses do not agree on how much " + "work this is" + ), + lenses={k: str(v) for k, v in sorted(counts.items())}, + ) + ] + return [] + + +def _coverage(interpretations: list[dict[str, Any]]) -> dict[str, list[str]]: + """Which lenses addressed each spec anchor. + + A requirement only some lenses engaged with is either unclear or hard to + find — both worth knowing. + """ + seen: dict[str, set[str]] = defaultdict(set) + for interpretation in interpretations: + lens = interpretation.get("lens", "?") + for collection in ("entities", "operations", "state_machines", "failure_modes", "blockers"): + for item in interpretation.get(collection, []): + key = _anchor_key(item.get("spec_anchor")) + if key: + seen[key].add(lens) + return {anchor: sorted(lenses) for anchor, lenses in sorted(seen.items())} + + +def compute(interpretations: list[dict[str, Any]]) -> ConcordanceResult: + """Merge a round's lens artifacts into one ranked, attributed view.""" + if not interpretations: + return ConcordanceResult(lens_count=0) + + result = ConcordanceResult(lens_count=len(interpretations)) + result.blockers = _merge_blockers(interpretations) + result.divergences = _dimension_divergences(interpretations) + _phase_divergences(interpretations) + result.coverage = _coverage(interpretations) + + # A diverged dimension is a concrete, located gap. Surface it as a blocker + # so it flows into the same ranking and resolution path as everything else. + for divergence in result.divergences: + if divergence.kind != "dimension": + continue + blocker_id = f"divergent-{divergence.where.replace('.', '-').replace('_', '-')}" + if any(b["id"] == blocker_id for b in result.blockers): + continue + options = [ + {"label": value, "consequence": f"chosen independently by: {lens}"} + for lens, value in sorted( + {v: k for k, v in divergence.lenses.items()}.items() + ) + ] + result.blockers.append({ + "id": blocker_id, + "title": f"Lenses disagree on {divergence.where}", + "spec_anchor": {"file": "", "section": divergence.where}, + "scenario": divergence.detail, + "question": f"Which value should {divergence.where} lock to?", + "options": options, + "recommended": options[0]["label"] if options else "", + "impact": "changes_architecture", + "reversible": False, + "found_by": sorted(divergence.lenses), + }) + + return result diff --git a/plugins/specflow/lib/specflow/contracts.py b/plugins/specflow/lib/specflow/contracts.py new file mode 100644 index 0000000..4d67b64 --- /dev/null +++ b/plugins/specflow/lib/specflow/contracts.py @@ -0,0 +1,259 @@ +"""The contracts oracle: keep the compiler, drop the application. + +A real build gives you an oracle that does not care what the agent believes — +the schema either loads or it does not. Dropping the build loses that, but not +all of it: you can still emit the data model and API contract as *real* +artifacts and check them mechanically, at no runtime cost. + +Two kinds of check: + + ``check_model`` contradictions derivable from the interpretation alone. + Always runnable, no emitted files needed. + ``check_emitted`` cross-checks emitted SQL DDL and API JSON against the + model, so the agent cannot describe one system and emit + another. + +The interesting finds are contradictions rather than omissions. Spec ambiguity +tends to surface as a structural impossibility — a field that must be both +supplied and computed, or two entities that each require the other to exist +first. Nobody writes those on purpose; they are what an underdetermined spec +looks like once you make it concrete. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from typing import Any + +_CREATE_TABLE = re.compile( + r"create\s+table\s+(?:if\s+not\s+exists\s+)?[\"`\[]?(\w+)[\"`\]]?\s*\((.*?)\)\s*;", + re.IGNORECASE | re.DOTALL, +) +_REFERENCES = re.compile(r"references\s+[\"`\[]?(\w+)[\"`\]]?", re.IGNORECASE) +_PRIMARY_KEY = re.compile(r"primary\s+key", re.IGNORECASE) + + +@dataclass +class ContractIssue: + kind: str + detail: str + + def __str__(self) -> str: + return f"[{self.kind}] {self.detail}" + + +@dataclass +class ContractReport: + issues: list[ContractIssue] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.issues + + def add(self, kind: str, detail: str) -> None: + self.issues.append(ContractIssue(kind, detail)) + + +def check_model(interpretation: dict[str, Any]) -> ContractReport: + """Structural contradictions in the model itself.""" + report = ContractReport() + entities = {e.get("name"): e for e in interpretation.get("entities", [])} + + for name, entity in entities.items(): + for field_def in entity.get("fields", []): + fname = f"{name}.{field_def.get('name')}" + + # Supplied by the caller AND computed by the system: impossible. + if field_def.get("required") and field_def.get("derived"): + report.add( + "contradiction", + f"{fname} is both required (caller supplies it) and derived " + "(system computes it) — the spec does not say which", + ) + + target = field_def.get("references") + if target and target not in entities: + report.add("dangling-reference", f"{fname} references undefined entity '{target}'") + + if not entity.get("identity"): + report.add("no-identity", f"{name} has no primary key — rows cannot be addressed") + + _check_circular_requirements(entities, report) + _check_operation_fields(interpretation, entities, report) + return report + + +def _check_circular_requirements( + entities: dict[str, Any], report: ContractReport +) -> None: + """Two entities that each require a reference to the other cannot be created. + + A real insert would deadlock on this. It is a common shape when a spec + describes a relationship without saying which side comes first. + """ + required_edges: dict[str, set[str]] = {} + for name, entity in entities.items(): + targets = { + f.get("references") + for f in entity.get("fields", []) + if f.get("references") and f.get("required") + } + required_edges[name] = {t for t in targets if t and t != name} + + seen_pairs = set() + for name, targets in required_edges.items(): + for target in targets: + if name in required_edges.get(target, set()): + pair = tuple(sorted((name, target))) + if pair not in seen_pairs: + seen_pairs.add(pair) + report.add( + "circular-requirement", + f"{pair[0]} and {pair[1]} each require a reference to the other — " + "neither can be created first", + ) + + +def _check_operation_fields( + interpretation: dict[str, Any], entities: dict[str, Any], report: ContractReport +) -> None: + """Operation inputs and outputs must name fields that exist on the entity.""" + for operation in interpretation.get("operations", []): + entity_name = operation.get("entity") + entity = entities.get(entity_name) + if not entity: + continue # totality.py reports the missing entity. + known = {f.get("name") for f in entity.get("fields", [])} + for direction in ("inputs", "outputs"): + for ref in operation.get(direction, []): + bare = ref.split(".")[-1] + if bare not in known: + report.add( + "unknown-field", + f"{operation.get('name')} {direction[:-1]} '{ref}' is not a field " + f"of {entity_name}", + ) + + if operation.get("kind") in ("create", "update", "delete", "command"): + if not operation.get("authorization"): + report.add( + "unguarded-mutation", + f"{operation.get('name')} changes data but the spec does not say who may call it", + ) + + +def parse_ddl(sql: str) -> dict[str, dict[str, Any]]: + """Extract table names, referenced tables and primary-key presence from DDL. + + Not a SQL parser — a contradiction detector. It needs to know what tables + exist, what they point at, and whether they are addressable. + """ + tables: dict[str, dict[str, Any]] = {} + for match in _CREATE_TABLE.finditer(sql): + name, body = match.group(1), match.group(2) + tables[name] = { + "references": {m.group(1) for m in _REFERENCES.finditer(body)}, + "has_primary_key": bool(_PRIMARY_KEY.search(body)), + "body": body, + } + return tables + + +def check_emitted( + interpretation: dict[str, Any], + *, + sql: str | None = None, + api: str | None = None, +) -> ContractReport: + """Cross-check emitted artifacts against the model they claim to implement.""" + report = ContractReport() + entities = {e.get("name") for e in interpretation.get("entities", [])} + + if sql is not None: + tables = parse_ddl(sql) + if not tables: + report.add("empty-ddl", "no CREATE TABLE statements found in the emitted SQL") + lowered = {t.lower() for t in tables} + for name in entities: + if name and name.lower() not in lowered and f"{name.lower()}s" not in lowered: + report.add("missing-table", f"entity '{name}' has no table in the emitted DDL") + for table, meta in tables.items(): + if not meta["has_primary_key"]: + report.add("no-primary-key", f"table '{table}' declares no primary key") + for target in meta["references"]: + if target.lower() not in lowered: + report.add( + "dangling-fk", + f"table '{table}' references '{target}', which is not created", + ) + + if api is not None: + try: + document = json.loads(api) + except json.JSONDecodeError as exc: + report.add("invalid-api-json", f"emitted API contract is not valid JSON: {exc}") + return report + _check_api(document, interpretation, report) + + return report + + +def _check_api( + document: Any, interpretation: dict[str, Any], report: ContractReport +) -> None: + """Validate an OpenAPI-shaped JSON document structurally.""" + if not isinstance(document, dict): + report.add("invalid-api-json", "API contract should be a JSON object") + return + + paths = document.get("paths") + if not isinstance(paths, dict) or not paths: + report.add("no-paths", "API contract declares no paths") + return + + schemas = (document.get("components") or {}).get("schemas") or {} + + # Every $ref must resolve — a dangling ref means the contract describes a + # shape it never defines. + def refs(node: Any): + if isinstance(node, dict): + for key, value in node.items(): + if key == "$ref" and isinstance(value, str): + yield value + else: + yield from refs(value) + elif isinstance(node, list): + for item in node: + yield from refs(item) + + for ref in set(refs(document)): + if ref.startswith("#/components/schemas/"): + if ref.rsplit("/", 1)[-1] not in schemas: + report.add("dangling-ref", f"{ref} does not resolve") + else: + report.add("external-ref", f"{ref} points outside the document") + + for name, schema in schemas.items(): + if isinstance(schema, dict): + for prop, definition in (schema.get("properties") or {}).items(): + if isinstance(definition, dict) and not ( + definition.get("type") or definition.get("$ref") or definition.get("allOf") + ): + report.add("untyped-property", f"{name}.{prop} has no type") + + operation_count = sum( + 1 + for methods in paths.values() + if isinstance(methods, dict) + for method in methods + if method.lower() in ("get", "post", "put", "patch", "delete") + ) + declared = len(interpretation.get("operations", [])) + if declared and operation_count < declared: + report.add( + "incomplete-api", + f"model declares {declared} operation(s) but the contract exposes " + f"{operation_count} — some are unreachable", + ) diff --git a/plugins/specflow/lib/specflow/jsonschema_mini.py b/plugins/specflow/lib/specflow/jsonschema_mini.py new file mode 100644 index 0000000..cf822eb --- /dev/null +++ b/plugins/specflow/lib/specflow/jsonschema_mini.py @@ -0,0 +1,220 @@ +"""A JSON Schema validator covering exactly the keywords SpecFlow's schemas use. + +Why not the ``jsonschema`` package: this library ships inside a Claude Code +marketplace plugin and runs on whatever Python the user happens to have. A +``pip install`` step turns a working skill into a support ticket, so the +validator is stdlib-only. + +The trade is deliberate: we own both the schemas and the validator, so the +supported keyword set is closed. Anything used in ``schema/*.json`` is +implemented here; anything not implemented raises rather than silently passing, +so a schema author cannot accidentally write a constraint that is never checked. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +SCHEMA_DIR = Path(__file__).parent / "schema" + +# Keywords this validator understands. A schema using anything else is a bug in +# the schema, not an input we should quietly accept. +_SUPPORTED = frozenset({ + "$schema", "$id", "$ref", "$defs", "title", "description", + "type", "required", "properties", "additionalProperties", + "items", "minItems", "maxItems", + "minLength", "maxLength", "pattern", + "enum", "const", "minimum", "maximum", + "anyOf", "oneOf", +}) + +_TYPE_MAP = { + "object": dict, + "array": list, + "string": str, + "boolean": bool, + "number": (int, float), + "integer": int, + "null": type(None), +} + + +@dataclass +class Problem: + """One validation failure, addressed by JSON path so it can be acted on.""" + + path: str + message: str + + def __str__(self) -> str: + where = self.path or "" + return f"{where}: {self.message}" + + +@dataclass +class Result: + problems: list[Problem] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.problems + + def add(self, path: str, message: str) -> None: + self.problems.append(Problem(path, message)) + + def merge(self, other: "Result") -> None: + self.problems.extend(other.problems) + + +class SchemaStore: + """Loads and resolves the bundled schemas by ``$id``.""" + + def __init__(self, schema_dir: Path = SCHEMA_DIR) -> None: + self._by_id: dict[str, dict[str, Any]] = {} + for path in sorted(schema_dir.glob("*.schema.json")): + schema = json.loads(path.read_text(encoding="utf-8")) + schema_id = schema.get("$id") + if not schema_id: + raise ValueError(f"{path.name} has no $id — cannot be referenced") + self._by_id[schema_id] = schema + + def get(self, schema_id: str) -> dict[str, Any]: + try: + return self._by_id[schema_id] + except KeyError: + raise KeyError( + f"Unknown schema '{schema_id}'. Known: {sorted(self._by_id)}" + ) from None + + def resolve(self, ref: str, current: dict[str, Any]) -> dict[str, Any]: + """Resolve a ``$ref``. Supports whole-schema ids and local ``#/$defs/x``.""" + if ref.startswith("#/"): + node: Any = current + for part in ref[2:].split("/"): + node = node[part] + return node + return self.get(ref) + + +def _type_name(value: Any) -> str: + for name, py in _TYPE_MAP.items(): + # bool is a subclass of int; check it before number/integer. + if name in ("number", "integer") and isinstance(value, bool): + continue + if isinstance(value, py): + return name + return type(value).__name__ + + +def _check_type(value: Any, expected: str) -> bool: + py = _TYPE_MAP.get(expected) + if py is None: + raise ValueError(f"Unsupported type keyword: {expected!r}") + if expected in ("number", "integer") and isinstance(value, bool): + return False + return isinstance(value, py) + + +def validate( + instance: Any, + schema: dict[str, Any], + *, + store: SchemaStore | None = None, + root: dict[str, Any] | None = None, + path: str = "", +) -> Result: + """Validate ``instance`` against ``schema``. Returns every problem found.""" + store = store or SchemaStore() + root = root if root is not None else schema + result = Result() + + unsupported = set(schema) - _SUPPORTED + if unsupported: + raise ValueError( + f"Schema at {path or ''} uses unimplemented keywords: " + f"{sorted(unsupported)}. Implement them in jsonschema_mini or " + f"remove them from the schema — a silently-ignored constraint is worse " + f"than no constraint." + ) + + if "$ref" in schema: + target = store.resolve(schema["$ref"], root) + # A cross-schema ref carries its own $defs, so it becomes the new root. + new_root = target if schema["$ref"].startswith("specflow/") else root + return validate(instance, target, store=store, root=new_root, path=path) + + if "anyOf" in schema or "oneOf" in schema: + branches = schema.get("anyOf") or schema.get("oneOf") or [] + for branch in branches: + if validate(instance, branch, store=store, root=root, path=path).ok: + return result + result.add(path, "matches none of the allowed shapes") + return result + + expected = schema.get("type") + if expected and not _check_type(instance, expected): + result.add(path, f"expected {expected}, got {_type_name(instance)}") + return result # Wrong type — downstream checks would be noise. + + if "const" in schema and instance != schema["const"]: + result.add(path, f"must be {schema['const']!r}") + + if "enum" in schema and instance not in schema["enum"]: + result.add(path, f"{instance!r} is not one of {schema['enum']}") + + if isinstance(instance, str): + if "minLength" in schema and len(instance) < schema["minLength"]: + result.add(path, f"must be at least {schema['minLength']} character(s)") + if "maxLength" in schema and len(instance) > schema["maxLength"]: + result.add(path, f"must be at most {schema['maxLength']} characters") + if "pattern" in schema and not re.search(schema["pattern"], instance): + result.add(path, f"{instance!r} does not match {schema['pattern']}") + + if isinstance(instance, (int, float)) and not isinstance(instance, bool): + if "minimum" in schema and instance < schema["minimum"]: + result.add(path, f"must be >= {schema['minimum']}") + if "maximum" in schema and instance > schema["maximum"]: + result.add(path, f"must be <= {schema['maximum']}") + + if isinstance(instance, list): + if "minItems" in schema and len(instance) < schema["minItems"]: + result.add(path, f"needs at least {schema['minItems']} item(s), has {len(instance)}") + if "maxItems" in schema and len(instance) > schema["maxItems"]: + result.add(path, f"allows at most {schema['maxItems']} item(s)") + item_schema = schema.get("items") + if item_schema: + for i, item in enumerate(instance): + result.merge( + validate(item, item_schema, store=store, root=root, path=f"{path}[{i}]") + ) + + if isinstance(instance, dict): + for key in schema.get("required", []): + if key not in instance: + result.add(path, f"missing required property '{key}'") + properties = schema.get("properties", {}) + for key, value in instance.items(): + child = f"{path}.{key}" if path else key + if key in properties: + result.merge( + validate(value, properties[key], store=store, root=root, path=child) + ) + continue + extra = schema.get("additionalProperties") + if extra is False: + result.add(path, f"unexpected property '{key}'") + elif isinstance(extra, dict): + result.merge(validate(value, extra, store=store, root=root, path=child)) + + return result + + +def validate_as(instance: Any, schema_id: str) -> Result: + """Validate against a bundled schema by ``$id``.""" + store = SchemaStore() + schema = store.get(schema_id) + return validate(instance, schema, store=store, root=schema) diff --git a/plugins/specflow/lib/specflow/mutate.py b/plugins/specflow/lib/specflow/mutate.py new file mode 100644 index 0000000..695e39b --- /dev/null +++ b/plugins/specflow/lib/specflow/mutate.py @@ -0,0 +1,228 @@ +"""Ambiguity mutation: manufacturing ground truth for an instrument that has none. + +The open question in this design is whether simulated-build divergence actually +tracks real spec defects. With no builds, there is nothing to check against. + +So we manufacture the ground truth. Take a spec, programmatically remove a +constraint or introduce a contradiction, and assert two things: the loop raises a +blocker, and the blocker lands on the requirement we damaged. Detection without +localization is not good enough — a loop that always complains about everything +would pass a detection-only test. + +This is also the regression suite for the whole pipeline. A mutation the loop +stops catching is a concrete bug with a reproducible input. + +Selection is index-based rather than random so a run is reproducible from its +manifest alone. +""" + +from __future__ import annotations + +import re +import shutil +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +# Sentences carrying a hard constraint — the ones whose removal creates a real gap. +_CONSTRAINT = re.compile( + r"\b(must not|must|shall not|shall|always|never|only|exactly|at least|at most|" + r"required to|no more than|no fewer than)\b", + re.IGNORECASE, +) +_ERROR_CASE = re.compile( + r"\b(if .* fails|on failure|on error|when .* is unavailable|timeout|invalid|" + r"rejected|error case|otherwise)\b", + re.IGNORECASE, +) +_QUANTITY = re.compile(r"\b(\d+)\s*(seconds?|minutes?|hours?|days?|items?|times?|retries|attempts?)\b", re.IGNORECASE) +_ENUM = re.compile(r"\b(?:one of|either)\s+([^.]+?)(?:\.|$)", re.IGNORECASE) + +_INVERSIONS = [ + ("must not", "must"), + ("must", "must not"), + ("shall not", "shall"), + ("shall", "shall not"), + ("always", "never"), + ("never", "always"), +] + +MUTATIONS = ("drop_constraint", "contradict", "vague_quantity", "drop_error_case", "blur_enum") + + +@dataclass +class Mutation: + """One deliberate defect, with everything needed to check for it later.""" + + kind: str + file: str + line: int + original: str + replacement: str + expect_anchor_file: str + + def as_dict(self) -> dict[str, Any]: + return { + "kind": self.kind, + "file": self.file, + "line": self.line, + "original": self.original, + "replacement": self.replacement, + "expect_anchor_file": self.expect_anchor_file, + } + + +@dataclass +class Manifest: + spec_dir: str + mutated_dir: str + mutations: list[Mutation] = field(default_factory=list) + + def as_dict(self) -> dict[str, Any]: + return { + "spec_dir": self.spec_dir, + "mutated_dir": self.mutated_dir, + "mutations": [m.as_dict() for m in self.mutations], + } + + +def _spec_files(spec_dir: Path) -> list[Path]: + return sorted( + p for p in spec_dir.rglob("*") + if p.is_file() and p.suffix.lower() in (".md", ".txt", ".markdown") + ) + + +def _sentences(text: str) -> list[tuple[int, str]]: + """(1-based line number, line) for non-trivial prose lines.""" + return [ + (i + 1, line) + for i, line in enumerate(text.splitlines()) + if len(line.strip()) > 25 and not line.lstrip().startswith(("#", "|", "```")) + ] + + +def _pick(candidates: list[tuple[int, str]], index: int) -> tuple[int, str] | None: + if not candidates: + return None + return candidates[index % len(candidates)] + + +def _mutate_line(kind: str, line: str) -> str | None: + """Apply one mutation to a line, or return None if it does not apply.""" + if kind == "drop_constraint": + return "" # Delete the constraint entirely. + + if kind == "contradict": + for needle, replacement in _INVERSIONS: + match = re.search(rf"\b{re.escape(needle)}\b", line, re.IGNORECASE) + if match: + return line[: match.start()] + replacement + line[match.end():] + return None + + if kind == "vague_quantity": + match = _QUANTITY.search(line) + if not match: + return None + return line[: match.start()] + f"several {match.group(2)}" + line[match.end():] + + if kind == "drop_error_case": + return "" + + if kind == "blur_enum": + match = _ENUM.search(line) + if not match: + return None + return line[: match.start(1)] + "an appropriate value" + line[match.end(1):] + + raise ValueError(f"Unknown mutation kind: {kind}") + + +def apply_mutation( + spec_dir: Path, + mutated_dir: Path, + *, + kind: str, + index: int = 0, +) -> Manifest: + """Copy the spec tree, apply one mutation of ``kind``, and return the manifest.""" + if kind not in MUTATIONS: + raise ValueError(f"Unknown mutation {kind!r}. Available: {', '.join(MUTATIONS)}") + + if mutated_dir.exists(): + shutil.rmtree(mutated_dir) + shutil.copytree(spec_dir, mutated_dir) + + manifest = Manifest(spec_dir=str(spec_dir), mutated_dir=str(mutated_dir)) + + pattern = { + "drop_constraint": _CONSTRAINT, + "contradict": _CONSTRAINT, + "vague_quantity": _QUANTITY, + "drop_error_case": _ERROR_CASE, + "blur_enum": _ENUM, + }[kind] + + for path in _spec_files(mutated_dir): + text = path.read_text(encoding="utf-8") + candidates = [(n, line) for n, line in _sentences(text) if pattern.search(line)] + chosen = _pick(candidates, index) + if chosen is None: + continue + line_number, original = chosen + replacement = _mutate_line(kind, original) + if replacement is None: + continue + + lines = text.splitlines() + lines[line_number - 1] = replacement + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + relative = str(path.relative_to(mutated_dir)) + manifest.mutations.append( + Mutation( + kind=kind, + file=relative, + line=line_number, + original=original.strip(), + replacement=replacement.strip(), + expect_anchor_file=relative, + ) + ) + return manifest + + raise RuntimeError( + f"No line in {spec_dir} was eligible for mutation {kind!r} — " + "the spec may be too short or lack hard constraints to remove." + ) + + +def verify(manifest: dict[str, Any], blockers: list[dict[str, Any]]) -> dict[str, Any]: + """Did the loop detect the injected defect, and did it land in the right place? + + Localization is checked separately from detection on purpose. A loop that + raises blockers everywhere would score well on detection alone and be + useless. + """ + results = [] + for mutation in manifest.get("mutations", []): + expected = mutation["expect_anchor_file"] + anchored = [ + b for b in blockers + if (b.get("spec_anchor") or {}).get("file", "").endswith(Path(expected).name) + ] + results.append({ + "kind": mutation["kind"], + "expected_file": expected, + "detected": bool(blockers), + "localized": bool(anchored), + "matching_blockers": [b.get("id") for b in anchored], + }) + + localized = sum(1 for r in results if r["localized"]) + return { + "mutations": len(results), + "localized": localized, + "passed": bool(results) and localized == len(results), + "results": results, + } diff --git a/plugins/specflow/lib/specflow/rank.py b/plugins/specflow/lib/specflow/rank.py new file mode 100644 index 0000000..b58da5d --- /dev/null +++ b/plugins/specflow/lib/specflow/rank.py @@ -0,0 +1,145 @@ +"""Ranking and disposition: deciding what is worth a human's attention. + +Every question has a cost, so the loop must not hand the user a flat list of +everything it found. Two inputs decide each blocker's fate: + + *Cost asymmetry* — how expensive is being wrong? A choice that is cheap to + reverse should be assumed and logged, not asked about. A choice that locks the + architecture should be asked about even if only one lens raised it. + + *Concordance* — how many independent lenses hit the same thing? Agreement is + evidence; a lone cosmetic nitpick is probably one agent being pedantic. + +The output is a three-way disposition rather than a queue, because "ask the +user" is only correct for a minority of findings: + + ask blocking or irreversible — the user decides + assume apply the recommendation, record it, move on + note recorded for the audit trail, not surfaced + +Nothing here is a score. The numbers order the list and then stop existing. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +# How far a wrong choice propagates. +IMPACT_WEIGHT = { + "blocks_build": 4, + "changes_architecture": 3, + "changes_behaviour": 2, + "cosmetic": 1, +} + +ASK = "ask" +ASSUME = "assume" +NOTE = "note" + + +@dataclass +class Ranked: + blocker: dict[str, Any] + score: float + disposition: str + rationale: str + + def as_dict(self) -> dict[str, Any]: + return { + **self.blocker, + "_score": round(self.score, 2), + "_disposition": self.disposition, + "_rationale": self.rationale, + } + + +def _disposition( + blocker: dict[str, Any], concordance: float +) -> tuple[str, str]: + impact = blocker.get("impact", "changes_behaviour") + reversible = bool(blocker.get("reversible", True)) + weight = IMPACT_WEIGHT.get(impact, 2) + + if impact == "blocks_build": + return ASK, "nothing can be built until this is decided" + + if not reversible and weight >= IMPACT_WEIGHT["changes_behaviour"]: + return ASK, "expensive to undo once chosen" + + if impact == "cosmetic" and concordance < 0.5: + return NOTE, "cosmetic, and most lenses did not raise it" + + if reversible: + return ( + ASSUME, + "cheap to change later — applying the recommendation and recording it", + ) + + return ASK, "irreversible" + + +def rank( + blockers: list[dict[str, Any]], + *, + lens_count: int, + already_resolved: set[str] | None = None, +) -> list[Ranked]: + """Order blockers and assign each a disposition. + + ``already_resolved`` drops blockers the user has decided in an earlier round, + which is what stops the loop re-asking the same question. Note this + deliberately dedups against *resolved* ids and not against everything ever + seen — a blocker that was noted rather than resolved should come back if a + later round finds it more strongly. + """ + already_resolved = already_resolved or set() + lens_count = max(lens_count, 1) + ranked: list[Ranked] = [] + + for blocker in blockers: + identifier = blocker.get("id", "") + aliases = set(blocker.get("also_known_as", [])) + if identifier in already_resolved or aliases & already_resolved: + continue + + found_by = blocker.get("found_by") or [] + concordance = len(found_by) / lens_count + weight = IMPACT_WEIGHT.get(blocker.get("impact", "changes_behaviour"), 2) + irreversibility = 2.0 if not blocker.get("reversible", True) else 1.0 + + score = weight * (1.0 + concordance) * irreversibility + disposition, rationale = _disposition(blocker, concordance) + + ranked.append( + Ranked( + blocker=blocker, + score=score, + disposition=disposition, + rationale=rationale, + ) + ) + + ranked.sort(key=lambda r: (-r.score, r.blocker.get("id", ""))) + return ranked + + +def summarize(ranked: list[Ranked]) -> dict[str, Any]: + """Counts by disposition. Counts, deliberately — not a readiness score.""" + counts = {ASK: 0, ASSUME: 0, NOTE: 0} + for item in ranked: + counts[item.disposition] = counts.get(item.disposition, 0) + 1 + return { + "total": len(ranked), + "ask": counts[ASK], + "assume": counts[ASSUME], + "note": counts[NOTE], + } + + +def partition(ranked: list[Ranked]) -> dict[str, list[dict[str, Any]]]: + """Split into the three lists the resolve step works from.""" + buckets: dict[str, list[dict[str, Any]]] = {ASK: [], ASSUME: [], NOTE: []} + for item in ranked: + buckets[item.disposition].append(item.as_dict()) + return buckets diff --git a/plugins/specflow/lib/specflow/saturation.py b/plugins/specflow/lib/specflow/saturation.py new file mode 100644 index 0000000..595fdbf --- /dev/null +++ b/plugins/specflow/lib/specflow/saturation.py @@ -0,0 +1,146 @@ +"""The stop rule. + +One legitimate job of a metric is telling you when to stop, and dropping the +score leaves that job open. Saturation fills it without reintroducing a number +to defend: stop when a fresh round of independent lenses finds nothing new worth +asking about. + +That is directly observable and needs no threshold to calibrate. It is also +honest about what it claims — not "the spec is now 94% complete" but "another +round of six independent readings surfaced nothing new." +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from . import rank + + +@dataclass +class RoundRecord: + number: int + lens_count: int + ask_ids: list[str] = field(default_factory=list) + new_ask_ids: list[str] = field(default_factory=list) + dry: bool = False + + def as_dict(self) -> dict[str, Any]: + return { + "number": self.number, + "lens_count": self.lens_count, + "ask_ids": self.ask_ids, + "new_ask_ids": self.new_ask_ids, + "dry": self.dry, + } + + +@dataclass +class SaturationVerdict: + converged: bool + dry_streak: int + required_streak: int + reason: str + record: RoundRecord + + def as_dict(self) -> dict[str, Any]: + return { + "converged": self.converged, + "dry_streak": self.dry_streak, + "required_streak": self.required_streak, + "reason": self.reason, + "round": self.record.as_dict(), + } + + +def evaluate( + state: dict[str, Any], + ranked: list[rank.Ranked], + *, + round_number: int, + lens_count: int, + resolved: set[str] | None = None, + required_streak: int = 1, +) -> SaturationVerdict: + """Decide whether this round converged, and return the updated record. + + ``required_streak`` is how many consecutive dry rounds end the loop. One is + the default because each extra round costs a full fan-out for diminishing + return; raise it to two when the spec is high-stakes and a missed gap is + expensive. + + Novelty is measured against every previously *asked* id plus everything + already resolved. Blockers that were assumed or merely noted are not counted + as seen, so a finding that shows up more strongly in a later round can still + reopen the loop. + """ + resolved = resolved or set() + history = [RoundRecord(**r) for r in state.get("rounds", [])] + seen: set[str] = set(resolved) + for record in history: + seen.update(record.ask_ids) + + ask_ids = [ + item.blocker.get("id", "") + for item in ranked + if item.disposition == rank.ASK and item.blocker.get("id") + ] + new_ask_ids = sorted(set(ask_ids) - seen) + + record = RoundRecord( + number=round_number, + lens_count=lens_count, + ask_ids=sorted(set(ask_ids)), + new_ask_ids=new_ask_ids, + dry=not new_ask_ids, + ) + + dry_streak = 0 + for previous in reversed(history): + if previous.dry: + dry_streak += 1 + else: + break + if record.dry: + dry_streak += 1 + else: + dry_streak = 0 + + converged = dry_streak >= required_streak + if converged: + reason = ( + f"{dry_streak} consecutive round(s) with no new blockers to ask about — " + "further rounds are unlikely to find more" + ) + elif record.dry: + reason = f"this round was dry but {required_streak} in a row are required" + else: + reason = ( + f"{len(new_ask_ids)} new blocker(s) need a decision: " + f"{', '.join(new_ask_ids[:4])}" + + (" ..." if len(new_ask_ids) > 4 else "") + ) + + return SaturationVerdict( + converged=converged, + dry_streak=dry_streak, + required_streak=required_streak, + reason=reason, + record=record, + ) + + +def updated_state( + state: dict[str, Any], verdict: SaturationVerdict +) -> dict[str, Any]: + """Append this round to the state, replacing any record with the same number.""" + rounds = [r for r in state.get("rounds", []) if r.get("number") != verdict.record.number] + rounds.append(verdict.record.as_dict()) + rounds.sort(key=lambda r: r["number"]) + return { + **state, + "rounds": rounds, + "converged": verdict.converged, + "dry_streak": verdict.dry_streak, + } diff --git a/plugins/specflow/lib/specflow/schema/blocker.schema.json b/plugins/specflow/lib/specflow/schema/blocker.schema.json new file mode 100644 index 0000000..78e5e6d --- /dev/null +++ b/plugins/specflow/lib/specflow/schema/blocker.schema.json @@ -0,0 +1,74 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "specflow/blocker", + "title": "Blocker", + "description": "One thing the spec does not determine, found by simulating the build. A blocker is not a complaint -- it is a decision the implementer would be forced to make, with the options they would be choosing between.", + "type": "object", + "required": ["id", "title", "spec_anchor", "question", "options", "recommended", "impact", "reversible"], + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$", + "description": "Stable slug derived from the requirement and the decision, so the same blocker found by two lenses collides on the same id." + }, + "title": { + "type": "string", + "minLength": 1, + "maxLength": 120, + "description": "One line, stated as the missing decision. Not 'the spec is unclear'." + }, + "spec_anchor": { + "type": "object", + "required": ["file"], + "properties": { + "file": { "type": "string", "minLength": 1 }, + "section": { "type": "string" }, + "line": { "type": "integer", "minimum": 1 }, + "requirement_id": { "type": "string" } + }, + "additionalProperties": false + }, + "scenario": { + "type": "string", + "description": "The concrete situation that forces the decision. This is what makes a blocker checkable rather than abstract." + }, + "question": { + "type": "string", + "minLength": 1, + "description": "Answerable in one line. If it needs a paragraph of setup, the lens has not finished its own work." + }, + "options": { + "type": "array", + "minItems": 2, + "items": { + "type": "object", + "required": ["label", "consequence"], + "properties": { + "label": { "type": "string", "minLength": 1 }, + "consequence": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + } + }, + "recommended": { + "type": "string", + "minLength": 1, + "description": "Must match one option label. The default applied if the user does not object." + }, + "impact": { + "type": "string", + "enum": ["blocks_build", "changes_architecture", "changes_behaviour", "cosmetic"], + "description": "How far the wrong choice propagates." + }, + "reversible": { + "type": "boolean", + "description": "Whether a wrong choice is cheap to undo later. Drives whether we ask or assume." + }, + "found_by": { + "type": "array", + "items": { "type": "string" }, + "description": "Lens names. Populated by concordance, not by the lens itself." + } + }, + "additionalProperties": false +} diff --git a/plugins/specflow/lib/specflow/schema/dimensions.schema.json b/plugins/specflow/lib/specflow/schema/dimensions.schema.json new file mode 100644 index 0000000..6c084eb --- /dev/null +++ b/plugins/specflow/lib/specflow/schema/dimensions.schema.json @@ -0,0 +1,192 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "specflow/dimensions", + "title": "Architectural dimensions (Parts A-D)", + "description": "Machine-readable form of the dimensions framework in specflow-analysis. Single source of truth: every dimension listed here must be filled with exactly one value. A blank or multi-valued dimension is a totality failure, not a warning.", + "type": "object", + "required": ["part_a", "part_d"], + "properties": { + "part_a": { + "description": "Universal dimensions. Mandatory for every project. Pick exactly ONE value each.", + "type": "object", + "required": [ + "persistence", + "infrastructure_complexity", + "scale_target", + "technology_stack", + "quality_level", + "scope_boundaries" + ], + "properties": { + "persistence": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { + "type": "string", + "enum": [ + "in_memory", + "embedded_file", + "relational", + "document", + "key_value", + "event_sourced", + "external_service_only" + ] + }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + }, + "infrastructure_complexity": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { + "type": "string", + "enum": ["single_process", "single_host_multi_service", "orchestrated", "serverless", "managed_paas"] + }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + }, + "scale_target": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { + "type": "string", + "enum": ["single_user", "small_team", "single_tenant_org", "multi_tenant", "public_internet_scale"] + }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + }, + "technology_stack": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { + "type": "string", + "minLength": 1, + "description": "Concrete stack, e.g. 'Python 3.12 / FastAPI / Postgres 16 / React 19'. Vague values such as 'a web stack' are a totality failure." + }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + }, + "quality_level": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { + "type": "string", + "enum": ["prototype", "internal_tool", "production", "regulated"] + }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + }, + "scope_boundaries": { + "type": "object", + "required": ["in_scope", "out_of_scope", "spec_anchor"], + "properties": { + "in_scope": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "out_of_scope": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + }, + + "part_b": { + "description": "Technology-specific dimensions. Only the sections matching this project's type are required, but a claimed section must be complete.", + "type": "object", + "additionalProperties": { + "type": "object", + "required": ["value", "spec_anchor"], + "properties": { + "value": { "type": "string", "minLength": 1 }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + } + }, + + "part_c": { + "description": "Project-specific dimensions the lens discovered. These are the variance sources the framework did not anticipate.", + "type": "array", + "items": { + "type": "object", + "required": ["id", "question", "value", "spec_anchor"], + "properties": { + "id": { "type": "string", "minLength": 1 }, + "question": { "type": "string", "minLength": 1 }, + "value": { "type": "string", "minLength": 1 }, + "spec_anchor": { "$ref": "#/$defs/anchor" }, + "note": { "type": "string" } + }, + "additionalProperties": false + } + }, + + "part_d": { + "description": "Micro-level consistency locks. All fields required. These are the values that silently diverge between independent implementations.", + "type": "object", + "required": ["naming", "patterns"], + "properties": { + "naming": { + "type": "object", + "required": ["files", "identifiers", "database", "api_paths"], + "properties": { + "files": { "type": "string", "minLength": 1 }, + "identifiers": { "type": "string", "minLength": 1 }, + "database": { "type": "string", "minLength": 1 }, + "api_paths": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + }, + "patterns": { + "type": "object", + "required": ["error_handling", "validation_boundary", "async_style", "config_source"], + "properties": { + "error_handling": { "type": "string", "minLength": 1 }, + "validation_boundary": { "type": "string", "minLength": 1 }, + "async_style": { "type": "string", "minLength": 1 }, + "config_source": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false, + + "$defs": { + "anchor": { + "description": "Where in the spec this value came from. 'inferred' means the spec did not determine it -- which is itself a blocker.", + "type": "object", + "required": ["file"], + "properties": { + "file": { "type": "string", "minLength": 1 }, + "section": { "type": "string" }, + "line": { "type": "integer", "minimum": 1 }, + "inferred": { + "type": "boolean", + "description": "True when no spec text determined this value. Every inferred dimension must have a matching blocker." + } + }, + "additionalProperties": false + } + } +} diff --git a/plugins/specflow/lib/specflow/schema/interpretation.schema.json b/plugins/specflow/lib/specflow/schema/interpretation.schema.json new file mode 100644 index 0000000..7e608cf --- /dev/null +++ b/plugins/specflow/lib/specflow/schema/interpretation.schema.json @@ -0,0 +1,188 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "specflow/interpretation", + "title": "Interpretation", + "description": "One lens's total reading of the spec. Total is the point: every structure here must be filled, because a filled structure cannot hide the gap a prose list would skip over. Divergence between two interpretations of the same spec is a located ambiguity.", + "type": "object", + "required": ["lens", "spec_root", "dimensions", "entities", "operations", "blockers"], + "properties": { + "lens": { + "type": "string", + "minLength": 1, + "description": "Which attack angle produced this reading." + }, + "spec_root": { "type": "string", "minLength": 1 }, + + "dimensions": { "$ref": "specflow/dimensions" }, + + "entities": { + "description": "The data model this lens would build. Every field needs a type; an untyped field is a totality failure.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": ["name", "fields"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "fields": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": ["name", "type", "required"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "type": { "type": "string", "minLength": 1 }, + "required": { "type": "boolean" }, + "derived": { + "type": "boolean", + "description": "Computed rather than supplied. Required AND derived is a contradiction the contracts oracle rejects." + }, + "references": { + "type": "string", + "description": "Target entity name for a foreign key. Must resolve." + } + }, + "additionalProperties": false + } + }, + "identity": { "type": "string", "description": "Which field is the primary key." }, + "spec_anchor": { "$ref": "#/$defs/anchor" } + }, + "additionalProperties": false + } + }, + + "operations": { + "description": "The transactions this lens would expose. Each must name an entity that exists.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": ["name", "kind", "entity"], + "properties": { + "name": { "type": "string", "minLength": 1 }, + "kind": { + "type": "string", + "enum": ["create", "read", "update", "delete", "list", "command", "query"] + }, + "entity": { "type": "string", "minLength": 1 }, + "inputs": { "type": "array", "items": { "type": "string" } }, + "outputs": { "type": "array", "items": { "type": "string" } }, + "idempotent": { "type": "boolean" }, + "authorization": { + "type": "string", + "description": "Who may call this. Absent authorization on a mutating operation is a blocker." + }, + "spec_anchor": { "$ref": "#/$defs/anchor" } + }, + "additionalProperties": false + } + }, + + "state_machines": { + "description": "Any entity with a lifecycle. The matrix is the forcing function: every state x event pair needs an outcome, so the awkward combination cannot be quietly skipped.", + "type": "array", + "items": { + "type": "object", + "required": ["entity", "states", "events", "matrix"], + "properties": { + "entity": { "type": "string", "minLength": 1 }, + "states": { "type": "array", "minItems": 2, "items": { "type": "string", "minLength": 1 } }, + "events": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "matrix": { + "description": "One row per state x event pair. check_totality rejects a partial matrix.", + "type": "array", + "items": { + "type": "object", + "required": ["state", "event", "outcome"], + "properties": { + "state": { "type": "string", "minLength": 1 }, + "event": { "type": "string", "minLength": 1 }, + "outcome": { + "type": "string", + "minLength": 1, + "description": "Target state, or 'reject', or 'undefined_in_spec'. The last one is a blocker, not an answer." + }, + "guard": { "type": "string" } + }, + "additionalProperties": false + } + }, + "spec_anchor": { "$ref": "#/$defs/anchor" } + }, + "additionalProperties": false + } + }, + + "failure_modes": { + "description": "What this lens found when it asked what goes wrong. These are the defects a real build surfaces at integration time.", + "type": "array", + "items": { + "type": "object", + "required": ["scenario", "spec_says"], + "properties": { + "scenario": { "type": "string", "minLength": 1 }, + "spec_says": { + "type": "string", + "minLength": 1, + "description": "What the spec prescribes, or 'nothing' -- which must have a matching blocker." + }, + "would_do": { "type": "string" }, + "spec_anchor": { "$ref": "#/$defs/anchor" } + }, + "additionalProperties": false + } + }, + + "phases": { + "description": "How this lens would sequence the build. Divergent decomposition across lenses is itself a signal the spec underdetermines the work.", + "type": "array", + "items": { + "type": "object", + "required": ["number", "name", "delivers"], + "properties": { + "number": { "type": "integer", "minimum": 1 }, + "name": { "type": "string", "minLength": 1 }, + "delivers": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "depends_on": { "type": "array", "items": { "type": "integer" } } + }, + "additionalProperties": false + } + }, + + "blockers": { + "type": "array", + "items": { "$ref": "specflow/blocker" } + }, + + "assumptions": { + "description": "Choices this lens made that the spec did not determine, but which were too small to raise as blockers. Recorded so they are auditable rather than invisible.", + "type": "array", + "items": { + "type": "object", + "required": ["statement", "spec_anchor"], + "properties": { + "statement": { "type": "string", "minLength": 1 }, + "spec_anchor": { "$ref": "#/$defs/anchor" } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false, + + "$defs": { + "anchor": { + "type": "object", + "required": ["file"], + "properties": { + "file": { "type": "string", "minLength": 1 }, + "section": { "type": "string" }, + "line": { "type": "integer", "minimum": 1 }, + "inferred": { "type": "boolean" } + }, + "additionalProperties": false + } + } +} diff --git a/plugins/specflow/lib/specflow/totality.py b/plugins/specflow/lib/specflow/totality.py new file mode 100644 index 0000000..99f5b15 --- /dev/null +++ b/plugins/specflow/lib/specflow/totality.py @@ -0,0 +1,228 @@ +"""Totality checks: the forcing function that replaces building. + +A physical build compels decisions — you cannot run code past a point the spec +left undefined. Simulation has no such compulsion, so an agent asked for +blockers produces a *plausible* list rather than an exhaustive one, and the +awkward cases go unmentioned. + +Totality restores the compulsion structurally. A prose list is partial by +nature; a filled matrix is total by construction. These checks enforce that: + + 1. Every dimension carries a real value, not an evasion. + 2. Every state x event pair in a lifecycle has an outcome. + 3. Every reference resolves to something that exists. + 4. Every escape hatch is paid for with a blocker. + +(4) is the one that matters most. An agent can always write ``inferred: true`` +or ``outcome: "undefined_in_spec"`` to get past a gap — those are legitimate +answers, but only if the gap is also *raised*. Without this check the escape +hatches become a silent way to skip the hard cells, which is exactly the failure +mode simulation is prone to. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from itertools import product +from typing import Any + +# Values that look filled but say nothing. An agent that cannot determine a +# value must raise a blocker, not shrug in the cell. +EVASIONS = frozenset({ + "", "-", "--", "?", "??", "n/a", "na", "none", "tbd", "todo", "tbc", + "unknown", "unclear", "unspecified", "not specified", "not defined", + "undefined", "any", "either", "varies", "depends", "flexible", + "to be determined", "to be decided", "open question", "see spec", +}) + + +@dataclass +class Finding: + path: str + message: str + + def __str__(self) -> str: + return f"{self.path}: {self.message}" + + +@dataclass +class TotalityReport: + lens: str + findings: list[Finding] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.findings + + def add(self, path: str, message: str) -> None: + self.findings.append(Finding(path, message)) + + +def _is_evasion(value: Any) -> bool: + return isinstance(value, str) and value.strip().lower() in EVASIONS + + +def _blocker_anchors(interpretation: dict[str, Any]) -> set[tuple[str, str]]: + """(file, section) pairs that have at least one blocker raised against them.""" + anchors = set() + for blocker in interpretation.get("blockers", []): + anchor = blocker.get("spec_anchor") or {} + anchors.add((anchor.get("file", ""), anchor.get("section", ""))) + return anchors + + +def _walk_anchors(node: Any, path: str = ""): + """Yield (path, anchor) for every spec_anchor in the tree.""" + if isinstance(node, dict): + for key, value in node.items(): + child = f"{path}.{key}" if path else key + if key == "spec_anchor" and isinstance(value, dict): + yield path or "", value + else: + yield from _walk_anchors(value, child) + elif isinstance(node, list): + for i, item in enumerate(node): + yield from _walk_anchors(item, f"{path}[{i}]") + + +def check(interpretation: dict[str, Any]) -> TotalityReport: + """Run every totality check against one lens artifact.""" + report = TotalityReport(lens=interpretation.get("lens", "")) + + _check_no_evasions(interpretation, report) + _check_state_matrices(interpretation, report) + _check_references(interpretation, report) + _check_escape_hatches(interpretation, report) + _check_blocker_shape(interpretation, report) + + return report + + +def _check_no_evasions(interpretation: dict[str, Any], report: TotalityReport) -> None: + """A filled-looking cell that says nothing is not filled.""" + + def walk(node: Any, path: str) -> None: + if isinstance(node, dict): + for key, value in node.items(): + walk(value, f"{path}.{key}" if path else key) + elif isinstance(node, list): + for i, item in enumerate(node): + walk(item, f"{path}[{i}]") + elif _is_evasion(node): + report.add(path, f"{node!r} is an evasion, not a decision — raise a blocker instead") + + # Blocker text is allowed to discuss uncertainty; the rest of the artifact is not. + for key, value in interpretation.items(): + if key in ("blockers", "_path"): + continue + walk(value, key) + + +def _check_state_matrices(interpretation: dict[str, Any], report: TotalityReport) -> None: + """Every state x event pair needs an outcome. This is the core forcing function.""" + for i, machine in enumerate(interpretation.get("state_machines", [])): + path = f"state_machines[{i}]" + states = machine.get("states") or [] + events = machine.get("events") or [] + covered = { + (row.get("state"), row.get("event")) + for row in machine.get("matrix", []) + } + missing = [pair for pair in product(states, events) if pair not in covered] + if missing: + shown = ", ".join(f"{s} x {e}" for s, e in missing[:6]) + more = f" (+{len(missing) - 6} more)" if len(missing) > 6 else "" + report.add( + path, + f"{machine.get('entity', '?')} matrix is partial — " + f"{len(missing)} uncovered pair(s): {shown}{more}", + ) + unknown = [ + (row.get("state"), row.get("event")) + for row in machine.get("matrix", []) + if row.get("state") not in states or row.get("event") not in events + ] + for state, event in unknown: + report.add(path, f"matrix row references undeclared state/event: {state} x {event}") + + +def _check_references(interpretation: dict[str, Any], report: TotalityReport) -> None: + """Operations and foreign keys must point at entities that exist.""" + entities = {e.get("name") for e in interpretation.get("entities", [])} + + for i, operation in enumerate(interpretation.get("operations", [])): + target = operation.get("entity") + if target and target not in entities: + report.add( + f"operations[{i}]", + f"'{operation.get('name')}' acts on unknown entity '{target}'", + ) + + for i, entity in enumerate(interpretation.get("entities", [])): + for j, field_def in enumerate(entity.get("fields", [])): + target = field_def.get("references") + if target and target not in entities: + report.add( + f"entities[{i}].fields[{j}]", + f"'{field_def.get('name')}' references unknown entity '{target}'", + ) + + for i, machine in enumerate(interpretation.get("state_machines", [])): + target = machine.get("entity") + if target and target not in entities: + report.add(f"state_machines[{i}]", f"lifecycle for unknown entity '{target}'") + + +def _check_escape_hatches(interpretation: dict[str, Any], report: TotalityReport) -> None: + """Every admitted gap must be raised as a blocker. + + Without this, ``inferred: true`` and ``outcome: "undefined_in_spec"`` become a + quiet way to skip the hard cells while still passing every other check. + """ + raised = _blocker_anchors(interpretation) + + def is_raised(anchor: dict[str, Any]) -> bool: + key = (anchor.get("file", ""), anchor.get("section", "")) + # Match on file+section, or fall back to file alone. + return key in raised or any(f == key[0] for f, _ in raised) + + for path, anchor in _walk_anchors(interpretation): + if anchor.get("inferred") and not is_raised(anchor): + report.add( + path, + "value is marked inferred but no blocker was raised against " + f"{anchor.get('file')} — an admitted gap must be surfaced", + ) + + for i, machine in enumerate(interpretation.get("state_machines", [])): + anchor = machine.get("spec_anchor") or {} + for row in machine.get("matrix", []): + if row.get("outcome") == "undefined_in_spec" and not is_raised(anchor): + report.add( + f"state_machines[{i}]", + f"{row.get('state')} x {row.get('event')} is undefined in the spec " + "but no blocker was raised for it", + ) + break + + for i, mode in enumerate(interpretation.get("failure_modes", [])): + anchor = mode.get("spec_anchor") or {} + if str(mode.get("spec_says", "")).strip().lower() in ("nothing", "silent", "unhandled"): + if not is_raised(anchor): + report.add( + f"failure_modes[{i}]", + f"'{mode.get('scenario')}' is unhandled by the spec but no blocker " + "was raised for it", + ) + + +def _check_blocker_shape(interpretation: dict[str, Any], report: TotalityReport) -> None: + """A recommendation that is not one of the options cannot be applied.""" + for i, blocker in enumerate(interpretation.get("blockers", [])): + labels = [o.get("label") for o in blocker.get("options", [])] + recommended = blocker.get("recommended") + if recommended and recommended not in labels: + report.add( + f"blockers[{i}]", + f"recommended {recommended!r} is not one of the options {labels}", + ) diff --git a/plugins/specflow/lib/specflow_cli.py b/plugins/specflow/lib/specflow_cli.py new file mode 100644 index 0000000..c94658b --- /dev/null +++ b/plugins/specflow/lib/specflow_cli.py @@ -0,0 +1,489 @@ +#!/usr/bin/env python3 +"""Single entry point for every SpecFlow oracle. + +One dispatcher rather than seven scripts, for a practical reason: a skill has to +name a path to invoke anything, and that path is the one fragile thing in a +plugin. Keeping it to one file per skill invocation minimises the surface, and +Python resolves its own siblings from __file__ regardless of how it was called. + +Commands the refinement loop actually uses: + + new-round allocate the next round directory + round validate, merge, rank, and decide whether to stop <- the workhorse + resolve record a decision so later rounds stop asking + status render current state + contracts check emitted SQL/API against the model + mutate inject a known defect and verify it gets caught (internal) + +Exit codes: 0 success, 1 checks failed, 2 bad usage. The non-zero on failure is +the point — a skill cannot quietly proceed past a gate that did not pass. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +# Skills invoke this by absolute path from any working directory, so make the +# sibling package importable before importing it. +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from specflow import ( # noqa: E402 (must follow the sys.path bootstrap) + artifacts, + concordance, + contracts, + mutate, + rank, + saturation, + totality, +) +from specflow.jsonschema_mini import validate_as # noqa: E402 + +EXIT_OK, EXIT_FAILED, EXIT_USAGE = 0, 1, 2 + + +def _emit(payload: dict[str, Any], as_json: bool, human: str) -> None: + if as_json: + print(json.dumps(payload, indent=2, ensure_ascii=False)) + else: + print(human) + + +# ---------------------------------------------------------------- new-round + +def cmd_new_round(args: argparse.Namespace) -> int: + layout = artifacts.layout_for(args.outputs) + latest = layout.latest_round() or 0 + number = latest + 1 + directory = layout.round_dir(number) + directory.mkdir(parents=True, exist_ok=True) + + payload = { + "round": number, + "dir": str(directory), + "write_to": [ + str(layout.interpretation_path(number, lens)) for lens in args.lens + ], + } + lines = [f"Round {number} -> {directory}"] + lines += [f" expects {Path(p).name}" for p in payload["write_to"]] + _emit(payload, args.json, "\n".join(lines)) + return EXIT_OK + + +# ---------------------------------------------------------------- validate + +def _validate_one(interpretation: dict[str, Any]) -> dict[str, Any]: + """Schema conformance then totality, for one lens artifact.""" + payload = {k: v for k, v in interpretation.items() if not k.startswith("_")} + schema_result = validate_as(payload, "specflow/interpretation") + problems = [str(p) for p in schema_result.problems] + + # Totality only means something on a structurally valid artifact. + gaps: list[str] = [] + if schema_result.ok: + gaps = [str(f) for f in totality.check(interpretation).findings] + + return { + "lens": interpretation.get("lens"), + "path": interpretation.get("_path"), + "schema_problems": problems, + "totality_gaps": gaps, + "ok": not problems and not gaps, + } + + +def _validate_round(layout: artifacts.Layout, number: int) -> dict[str, Any]: + loaded = artifacts.load_interpretations(layout, number) + if not loaded: + return {"round": number, "lenses": [], "ok": False, "error": "no interpretation files found"} + reports = [_validate_one(item) for item in loaded] + return { + "round": number, + "lenses": reports, + "ok": all(r["ok"] for r in reports), + } + + +def cmd_validate(args: argparse.Namespace) -> int: + layout = artifacts.layout_for(args.outputs) + number = args.round or layout.latest_round() + if number is None: + _emit({"error": "no rounds found"}, args.json, "No rounds found. Run new-round first.") + return EXIT_USAGE + + result = _validate_round(layout, number) + _emit(result, args.json, _render_validation(result)) + return EXIT_OK if result["ok"] else EXIT_FAILED + + +def _render_validation(result: dict[str, Any]) -> str: + if result.get("error"): + return f"Round {result['round']}: {result['error']}" + lines = [f"Round {result['round']} — {len(result['lenses'])} lens artifact(s)"] + for report in result["lenses"]: + mark = "ok" if report["ok"] else "FAIL" + lines.append(f" [{mark}] {report['lens']}") + for problem in report["schema_problems"]: + lines.append(f" schema: {problem}") + for gap in report["totality_gaps"]: + lines.append(f" totality: {gap}") + if not result["ok"]: + lines.append("") + lines.append("Artifacts are not total. Fix them and re-run — do not proceed.") + return "\n".join(lines) + + +# ---------------------------------------------------------------- round + +def cmd_round(args: argparse.Namespace) -> int: + """Validate, merge, rank, and decide whether to stop. One call per round.""" + layout = artifacts.layout_for(args.outputs) + number = args.round or layout.latest_round() + if number is None: + _emit({"error": "no rounds found"}, args.json, "No rounds found. Run new-round first.") + return EXIT_USAGE + + validation = _validate_round(layout, number) + if not validation["ok"]: + _emit( + {"stage": "validate", "validation": validation}, + args.json, + _render_validation(validation), + ) + return EXIT_FAILED + + interpretations = artifacts.load_interpretations(layout, number) + merged = concordance.compute(interpretations) + + model_issues: list[str] = [] + for interpretation in interpretations: + report = contracts.check_model(interpretation) + model_issues += [f"{interpretation.get('lens')}: {issue}" for issue in map(str, report.issues)] + + resolved = artifacts.resolved_ids(layout) + ranked = rank.rank(merged.blockers, lens_count=merged.lens_count, already_resolved=resolved) + buckets = rank.partition(ranked) + summary = rank.summarize(ranked) + + state = artifacts.load_state(layout) + verdict = saturation.evaluate( + state, + ranked, + round_number=number, + lens_count=merged.lens_count, + resolved=resolved, + required_streak=args.consecutive, + ) + + artifacts.write_json(layout.state_path, saturation.updated_state(state, verdict)) + artifacts.write_json( + layout.blockers_path, + { + "round": number, + "lens_count": merged.lens_count, + "summary": summary, + "ask": buckets[rank.ASK], + "assume": buckets[rank.ASSUME], + "note": buckets[rank.NOTE], + "divergences": [d.as_dict() for d in merged.divergences], + "contract_issues": model_issues, + }, + ) + + payload = { + "stage": "complete", + "round": number, + "lens_count": merged.lens_count, + "summary": summary, + "converged": verdict.converged, + "saturation": verdict.as_dict(), + "ask": buckets[rank.ASK], + "assume": buckets[rank.ASSUME], + "divergences": [d.as_dict() for d in merged.divergences], + "contract_issues": model_issues, + "blockers_path": str(layout.blockers_path), + } + _emit(payload, args.json, _render_round(payload)) + return EXIT_OK + + +def _render_round(payload: dict[str, Any]) -> str: + summary = payload["summary"] + lines = [ + f"Round {payload['round']} — {payload['lens_count']} lenses", + f" ask {summary['ask']} assume {summary['assume']} note {summary['note']}", + "", + ] + + if payload["divergences"]: + lines.append("Located disagreements:") + for divergence in payload["divergences"]: + lines.append(f" {divergence['where']}: {divergence['detail']}") + for lens, value in divergence["lenses"].items(): + lines.append(f" {lens}: {value}") + lines.append("") + + if payload["contract_issues"]: + lines.append("Contract issues:") + lines += [f" {issue}" for issue in payload["contract_issues"]] + lines.append("") + + if payload["ask"]: + lines.append("Needs a decision:") + for blocker in payload["ask"]: + found = ", ".join(blocker.get("found_by", [])) + lines.append(f" [{blocker['_score']}] {blocker['id']} — {blocker['title']}") + lines.append(f" raised by: {found or 'unknown'} ({blocker['_rationale']})") + lines.append(f" {blocker.get('question', '')}") + lines.append("") + + if payload["assume"]: + lines.append("Assuming (recorded, reversible):") + for blocker in payload["assume"]: + lines.append(f" {blocker['id']} -> {blocker.get('recommended')}") + lines.append("") + + lines.append( + "CONVERGED — " + payload["saturation"]["reason"] + if payload["converged"] + else "NOT CONVERGED — " + payload["saturation"]["reason"] + ) + return "\n".join(lines) + + +# ---------------------------------------------------------------- resolve + +def cmd_resolve(args: argparse.Namespace) -> int: + layout = artifacts.layout_for(args.outputs) + existing = artifacts.load_resolutions(layout) + if any(r.get("blocker_id") == args.id for r in existing): + _emit( + {"error": "already resolved", "blocker_id": args.id}, + args.json, + f"{args.id} is already resolved.", + ) + return EXIT_USAGE + + record = { + "blocker_id": args.id, + "choice": args.choice, + "applied_to_spec": args.applied_to or [], + "source": args.source, + } + existing.append(record) + artifacts.write_json(layout.resolutions_path, {"resolved": existing}) + + _emit( + {"recorded": record, "total_resolved": len(existing)}, + args.json, + f"Recorded {args.id} -> {args.choice} ({len(existing)} resolved in total)", + ) + return EXIT_OK + + +# ---------------------------------------------------------------- status + +def cmd_status(args: argparse.Namespace) -> int: + layout = artifacts.layout_for(args.outputs) + state = artifacts.load_state(layout) + resolutions = artifacts.load_resolutions(layout) + blockers: dict[str, Any] = {} + if layout.blockers_path.exists(): + blockers = artifacts.read_json(layout.blockers_path) + + payload = { + "rounds_run": len(state.get("rounds", [])), + "converged": state.get("converged", False), + "dry_streak": state.get("dry_streak", 0), + "resolved": len(resolutions), + "open_ask": len(blockers.get("ask", [])), + "assumed": len(blockers.get("assume", [])), + "noted": len(blockers.get("note", [])), + "resolutions": resolutions, + "ask": blockers.get("ask", []), + } + + lines = [ + f"Rounds run {payload['rounds_run']}", + f"Converged {'yes' if payload['converged'] else 'no'}", + f"Resolved {payload['resolved']}", + f"Open decisions {payload['open_ask']}", + f"Assumed {payload['assumed']}", + f"Noted {payload['noted']}", + ] + if payload["ask"]: + lines.append("") + lines.append("Still open:") + lines += [f" {b['id']} — {b['title']}" for b in payload["ask"]] + _emit(payload, args.json, "\n".join(lines)) + return EXIT_OK + + +# ---------------------------------------------------------------- contracts + +def cmd_contracts(args: argparse.Namespace) -> int: + layout = artifacts.layout_for(args.outputs) + number = args.round or layout.latest_round() + if number is None: + _emit({"error": "no rounds found"}, args.json, "No rounds found.") + return EXIT_USAGE + + interpretations = artifacts.load_interpretations(layout, number) + if not interpretations: + _emit({"error": "no interpretations"}, args.json, f"No lens artifacts in round {number}.") + return EXIT_USAGE + + sql = Path(args.sql).read_text(encoding="utf-8") if args.sql else None + api = Path(args.api).read_text(encoding="utf-8") if args.api else None + + findings = [] + for interpretation in interpretations: + report = contracts.check_model(interpretation) + if sql or api: + emitted = contracts.check_emitted(interpretation, sql=sql, api=api) + report.issues.extend(emitted.issues) + findings.append({ + "lens": interpretation.get("lens"), + "issues": [str(issue) for issue in report.issues], + "ok": report.ok, + }) + + ok = all(f["ok"] for f in findings) + lines = [] + for finding in findings: + lines.append(f"[{'ok' if finding['ok'] else 'FAIL'}] {finding['lens']}") + lines += [f" {issue}" for issue in finding["issues"]] + _emit({"round": number, "findings": findings, "ok": ok}, args.json, "\n".join(lines) or "No issues.") + return EXIT_OK if ok else EXIT_FAILED + + +# ---------------------------------------------------------------- mutate + +def cmd_mutate(args: argparse.Namespace) -> int: + if args.mutate_command == "apply": + manifest = mutate.apply_mutation( + Path(args.spec_dir), + Path(args.into), + kind=args.kind, + index=args.index, + ) + out = Path(args.into) / "mutation-manifest.json" + artifacts.write_json(out, manifest.as_dict()) + first = manifest.mutations[0] + _emit( + {"manifest": manifest.as_dict(), "manifest_path": str(out)}, + args.json, + f"Applied {first.kind} to {first.file}:{first.line}\n" + f" was: {first.original}\n" + f" now: {first.replacement or ''}\n" + f"Manifest: {out}", + ) + return EXIT_OK + + manifest = artifacts.read_json(Path(args.manifest)) + layout = artifacts.layout_for(args.outputs) + blockers_doc = artifacts.read_json(layout.blockers_path) if layout.blockers_path.exists() else {} + all_blockers = ( + blockers_doc.get("ask", []) + blockers_doc.get("assume", []) + blockers_doc.get("note", []) + ) + result = mutate.verify(manifest, all_blockers) + + lines = [f"Mutations: {result['mutations']} localized: {result['localized']}"] + for entry in result["results"]: + mark = "PASS" if entry["localized"] else "MISS" + lines.append(f" [{mark}] {entry['kind']} expected in {entry['expected_file']}") + if entry["matching_blockers"]: + lines.append(f" matched: {', '.join(entry['matching_blockers'])}") + lines.append("") + lines.append("PASS — the loop detects and localizes injected ambiguity." if result["passed"] + else "FAIL — an injected defect was not localized. This is a real bug.") + _emit(result, args.json, "\n".join(lines)) + return EXIT_OK if result["passed"] else EXIT_FAILED + + +# ---------------------------------------------------------------- parser + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="specflow", + description="Deterministic oracles for the SpecFlow refinement loop.", + ) + parser.add_argument("--json", action="store_true", help="machine-readable output") + subparsers = parser.add_subparsers(dest="command", required=True) + + def with_outputs(sub: argparse.ArgumentParser) -> argparse.ArgumentParser: + sub.add_argument("--outputs", default="docs", help="outputs dir (default: docs)") + return sub + + new_round = with_outputs(subparsers.add_parser("new-round", help="allocate the next round")) + new_round.add_argument("--lens", nargs="*", default=[], help="lens names expected this round") + new_round.set_defaults(func=cmd_new_round) + + validate = with_outputs(subparsers.add_parser("validate", help="schema + totality only")) + validate.add_argument("--round", type=int) + validate.set_defaults(func=cmd_validate) + + round_cmd = with_outputs(subparsers.add_parser("round", help="validate, merge, rank, decide")) + round_cmd.add_argument("--round", type=int) + round_cmd.add_argument( + "--consecutive", type=int, default=1, + help="consecutive dry rounds required to converge (default 1)", + ) + round_cmd.set_defaults(func=cmd_round) + + resolve = with_outputs(subparsers.add_parser("resolve", help="record a decision")) + resolve.add_argument("--id", required=True, help="blocker id") + resolve.add_argument("--choice", required=True, help="the chosen option label") + resolve.add_argument("--applied-to", nargs="*", help="spec files updated") + resolve.add_argument( + "--source", default="user", choices=["user", "assumed"], + help="whether the user decided or the default was applied", + ) + resolve.set_defaults(func=cmd_resolve) + + status = with_outputs(subparsers.add_parser("status", help="current refinement state")) + status.set_defaults(func=cmd_status) + + contracts_cmd = with_outputs(subparsers.add_parser("contracts", help="check the model and emitted artifacts")) + contracts_cmd.add_argument("--round", type=int) + contracts_cmd.add_argument("--sql", help="emitted DDL file") + contracts_cmd.add_argument("--api", help="emitted API contract (JSON)") + contracts_cmd.set_defaults(func=cmd_contracts) + + mutate_cmd = subparsers.add_parser("mutate", help="inject a defect and verify detection") + mutate_subs = mutate_cmd.add_subparsers(dest="mutate_command", required=True) + + apply_cmd = mutate_subs.add_parser("apply") + apply_cmd.add_argument("--spec-dir", required=True) + apply_cmd.add_argument("--into", required=True, help="destination for the mutated copy") + apply_cmd.add_argument("--kind", required=True, choices=list(mutate.MUTATIONS)) + apply_cmd.add_argument("--index", type=int, default=0, help="which eligible line (deterministic)") + apply_cmd.set_defaults(func=cmd_mutate) + + verify_cmd = with_outputs(mutate_subs.add_parser("verify")) + verify_cmd.add_argument("--manifest", required=True) + verify_cmd.set_defaults(func=cmd_mutate) + + return parser + + +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + try: + return args.func(args) + except (FileNotFoundError, ValueError, RuntimeError, KeyError) as exc: + message = str(exc).strip("'") + if args.json: + print(json.dumps({"error": message}, indent=2)) + else: + print(f"error: {message}", file=sys.stderr) + return EXIT_USAGE + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/specflow/lib/tests/test_oracles.py b/plugins/specflow/lib/tests/test_oracles.py new file mode 100644 index 0000000..5e5e142 --- /dev/null +++ b/plugins/specflow/lib/tests/test_oracles.py @@ -0,0 +1,450 @@ +#!/usr/bin/env python3 +"""Regression tests for the SpecFlow oracles. + +Run with the stdlib runner — no pytest, matching the plugin's zero-dependency +policy: + + python3 plugins/specflow/lib/tests/test_oracles.py + +Every test here corresponds to a defect the loop must keep catching. If one +starts failing, the loop has become less able to find real specification gaps — +which is the only thing this product does. +""" + +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from specflow import artifacts, concordance, contracts, mutate, rank, saturation, totality +from specflow.jsonschema_mini import validate_as + + +def anchor( + file: str = "specs/orders.md", section: str = "Checkout", inferred: bool = False +) -> dict[str, Any]: + value: dict[str, Any] = {"file": file, "section": section} + if inferred: + value["inferred"] = True + return value + + +def valid_interpretation(lens: str = "concurrency") -> dict[str, Any]: + """A minimal artifact that passes schema and totality. The baseline.""" + return { + "lens": lens, + "spec_root": "specs", + "dimensions": { + "part_a": { + "persistence": {"value": "relational", "spec_anchor": anchor()}, + "infrastructure_complexity": {"value": "single_process", "spec_anchor": anchor()}, + "scale_target": {"value": "small_team", "spec_anchor": anchor()}, + "technology_stack": {"value": "Python 3.12 / FastAPI / Postgres 16", "spec_anchor": anchor()}, + "quality_level": {"value": "production", "spec_anchor": anchor()}, + "scope_boundaries": { + "in_scope": ["checkout"], + "out_of_scope": ["refunds"], + "spec_anchor": anchor(), + }, + }, + "part_d": { + "naming": { + "files": "snake_case", + "identifiers": "snake_case", + "database": "plural snake_case", + "api_paths": "kebab-case", + }, + "patterns": { + "error_handling": "exceptions at boundary", + "validation_boundary": "request schema", + "async_style": "async/await", + "config_source": "env vars", + }, + }, + }, + "entities": [ + { + "name": "Order", + "identity": "id", + "spec_anchor": anchor(), + "fields": [ + {"name": "id", "type": "uuid", "required": True}, + {"name": "total", "type": "decimal", "required": False, "derived": True}, + ], + } + ], + "operations": [ + { + "name": "createOrder", + "kind": "create", + "entity": "Order", + "idempotent": False, + "authorization": "authenticated buyer", + "spec_anchor": anchor(), + } + ], + "state_machines": [ + { + "entity": "Order", + "states": ["pending", "paid"], + "events": ["pay", "expire"], + "spec_anchor": anchor(), + "matrix": [ + {"state": "pending", "event": "pay", "outcome": "paid"}, + {"state": "pending", "event": "expire", "outcome": "reject"}, + {"state": "paid", "event": "pay", "outcome": "reject"}, + {"state": "paid", "event": "expire", "outcome": "reject"}, + ], + } + ], + "failure_modes": [], + "phases": [{"number": 1, "name": "Checkout core", "delivers": ["createOrder"]}], + "blockers": [ + { + "id": "paid-order-expiry", + "title": "What happens when a paid order's reservation expires", + "spec_anchor": anchor(), + "scenario": "Payment settles after the hold lapses.", + "question": "Honour the order or refund it?", + "options": [ + {"label": "honour", "consequence": "may oversell"}, + {"label": "refund", "consequence": "buyer loses the item"}, + ], + "recommended": "refund", + "impact": "changes_behaviour", + "reversible": False, + } + ], + "assumptions": [], + } + + +class TestBaseline(unittest.TestCase): + """The fixture itself must be clean, or every other test is meaningless.""" + + def test_schema_valid(self): + result = validate_as(valid_interpretation(), "specflow/interpretation") + self.assertTrue(result.ok, [str(p) for p in result.problems]) + + def test_totality_clean(self): + report = totality.check(valid_interpretation()) + self.assertTrue(report.ok, [str(f) for f in report.findings]) + + +class TestTotalityGate(unittest.TestCase): + """The forcing function. Each of these is a way to skip real work.""" + + def _findings(self, mutate_fn) -> list[str]: + artifact = valid_interpretation() + mutate_fn(artifact) + return [str(f) for f in totality.check(artifact).findings] + + def test_rejects_partial_state_matrix(self): + def drop_rows(a): + a["state_machines"][0]["matrix"] = a["state_machines"][0]["matrix"][:2] + + findings = self._findings(drop_rows) + self.assertTrue(any("matrix is partial" in f for f in findings), findings) + + def test_rejects_evasion_value(self): + def evade(a): + a["dimensions"]["part_d"]["patterns"]["config_source"] = "TBD" + + findings = self._findings(evade) + self.assertTrue(any("evasion" in f for f in findings), findings) + + def test_rejects_unresolvable_operation_entity(self): + def dangle(a): + a["operations"].append({"name": "archiveInvoice", "kind": "command", "entity": "Invoice"}) + + findings = self._findings(dangle) + self.assertTrue(any("unknown entity 'Invoice'" in f for f in findings), findings) + + def test_rejects_unresolvable_foreign_key(self): + def dangle(a): + a["entities"][0]["fields"].append( + {"name": "customer_id", "type": "uuid", "required": True, "references": "Customer"} + ) + + findings = self._findings(dangle) + self.assertTrue(any("unknown entity 'Customer'" in f for f in findings), findings) + + def test_rejects_inferred_value_with_no_blocker(self): + """The loophole that matters most: admitting a gap without raising it.""" + + def infer_silently(a): + a["dimensions"]["part_a"]["scale_target"]["spec_anchor"] = anchor( + file="specs/unrelated.md", section="", inferred=True + ) + + findings = self._findings(infer_silently) + self.assertTrue(any("marked inferred" in f for f in findings), findings) + + def test_rejects_undefined_matrix_outcome_with_no_blocker(self): + def undefined_without_blocker(a): + a["state_machines"][0]["matrix"][3]["outcome"] = "undefined_in_spec" + a["state_machines"][0]["spec_anchor"] = anchor(file="specs/unrelated.md", section="") + + findings = self._findings(undefined_without_blocker) + self.assertTrue(any("undefined in the spec" in f for f in findings), findings) + + def test_rejects_recommendation_outside_options(self): + def bad_recommendation(a): + a["blockers"][0]["recommended"] = "something-else" + + findings = self._findings(bad_recommendation) + self.assertTrue(any("not one of the options" in f for f in findings), findings) + + def test_accepts_admitted_gap_when_raised(self): + """The escape hatch is legitimate — as long as the gap is surfaced.""" + artifact = valid_interpretation() + artifact["state_machines"][0]["matrix"][3]["outcome"] = "undefined_in_spec" + report = totality.check(artifact) + self.assertTrue(report.ok, [str(f) for f in report.findings]) + + +class TestContracts(unittest.TestCase): + def test_detects_required_and_derived_contradiction(self): + artifact = valid_interpretation() + artifact["entities"][0]["fields"][1]["required"] = True # total is already derived + issues = [str(i) for i in contracts.check_model(artifact).issues] + self.assertTrue(any("contradiction" in i for i in issues), issues) + + def test_detects_circular_required_reference(self): + artifact = valid_interpretation() + artifact["entities"][0]["fields"].append( + {"name": "reservation_id", "type": "uuid", "required": True, "references": "Reservation"} + ) + artifact["entities"].append({ + "name": "Reservation", + "identity": "id", + "fields": [ + {"name": "id", "type": "uuid", "required": True}, + {"name": "order_id", "type": "uuid", "required": True, "references": "Order"}, + ], + }) + issues = [str(i) for i in contracts.check_model(artifact).issues] + self.assertTrue(any("circular-requirement" in i for i in issues), issues) + + def test_detects_unguarded_mutation(self): + artifact = valid_interpretation() + del artifact["operations"][0]["authorization"] + issues = [str(i) for i in contracts.check_model(artifact).issues] + self.assertTrue(any("unguarded-mutation" in i for i in issues), issues) + + def test_detects_missing_table_in_emitted_ddl(self): + artifact = valid_interpretation() + report = contracts.check_emitted(artifact, sql="CREATE TABLE unrelated (id uuid PRIMARY KEY);") + issues = [str(i) for i in report.issues] + self.assertTrue(any("missing-table" in i for i in issues), issues) + + def test_detects_dangling_api_ref(self): + artifact = valid_interpretation() + api = json.dumps({ + "paths": {"/orders": {"post": {"responses": {"200": {"$ref": "#/components/schemas/Ghost"}}}}}, + "components": {"schemas": {}}, + }) + issues = [str(i) for i in contracts.check_emitted(artifact, api=api).issues] + self.assertTrue(any("dangling-ref" in i for i in issues), issues) + + +class TestConcordance(unittest.TestCase): + def test_dimension_disagreement_becomes_a_blocker(self): + first = valid_interpretation("concurrency") + second = valid_interpretation("ordering") + second["dimensions"]["part_a"]["persistence"]["value"] = "event_sourced" + + result = concordance.compute([first, second]) + divergences = [d.where for d in result.divergences] + self.assertIn("part_a.persistence", divergences) + self.assertTrue(any(b["id"].startswith("divergent-") for b in result.blockers)) + + def test_same_blocker_from_two_lenses_merges(self): + first = valid_interpretation("concurrency") + second = valid_interpretation("ordering") + result = concordance.compute([first, second]) + merged = [b for b in result.blockers if b["id"] == "paid-order-expiry"] + self.assertEqual(len(merged), 1) + self.assertEqual(sorted(merged[0]["found_by"]), ["concurrency", "ordering"]) + + def test_agreement_produces_no_divergence(self): + result = concordance.compute([valid_interpretation("a"), valid_interpretation("b")]) + self.assertEqual([d for d in result.divergences if d.kind == "dimension"], []) + + +class TestRanking(unittest.TestCase): + def _blocker(self, **overrides): + base = { + "id": "b1", + "title": "t", + "options": [{"label": "x", "consequence": "c"}, {"label": "y", "consequence": "c"}], + "recommended": "x", + "impact": "changes_behaviour", + "reversible": True, + "found_by": ["one"], + } + base.update(overrides) + return base + + def test_blocking_impact_is_always_asked(self): + ranked = rank.rank([self._blocker(impact="blocks_build")], lens_count=6) + self.assertEqual(ranked[0].disposition, rank.ASK) + + def test_reversible_low_impact_is_assumed(self): + ranked = rank.rank([self._blocker(reversible=True)], lens_count=6) + self.assertEqual(ranked[0].disposition, rank.ASSUME) + + def test_irreversible_is_asked(self): + ranked = rank.rank([self._blocker(reversible=False)], lens_count=6) + self.assertEqual(ranked[0].disposition, rank.ASK) + + def test_lone_cosmetic_finding_is_only_noted(self): + ranked = rank.rank([self._blocker(impact="cosmetic", found_by=["one"])], lens_count=6) + self.assertEqual(ranked[0].disposition, rank.NOTE) + + def test_resolved_blockers_are_dropped(self): + ranked = rank.rank([self._blocker()], lens_count=6, already_resolved={"b1"}) + self.assertEqual(ranked, []) + + def test_concordance_raises_rank(self): + many = self._blocker(id="many", found_by=["a", "b", "c", "d", "e", "f"]) + few = self._blocker(id="few", found_by=["a"]) + ranked = rank.rank([few, many], lens_count=6) + self.assertEqual(ranked[0].blocker["id"], "many") + + +class TestSaturation(unittest.TestCase): + def test_new_blockers_prevent_convergence(self): + ranked = rank.rank( + [{"id": "new-one", "impact": "blocks_build", "reversible": False, "found_by": ["a"]}], + lens_count=1, + ) + verdict = saturation.evaluate({}, ranked, round_number=1, lens_count=1) + self.assertFalse(verdict.converged) + + def test_dry_round_converges(self): + verdict = saturation.evaluate({}, [], round_number=1, lens_count=1) + self.assertTrue(verdict.converged) + + def test_resolved_blocker_counts_as_seen(self): + ranked = rank.rank( + [{"id": "known", "impact": "blocks_build", "reversible": False, "found_by": ["a"]}], + lens_count=1, + ) + verdict = saturation.evaluate( + {}, ranked, round_number=2, lens_count=1, resolved={"known"} + ) + self.assertTrue(verdict.converged) + + def test_two_consecutive_required(self): + first = saturation.evaluate({}, [], round_number=1, lens_count=1, required_streak=2) + self.assertFalse(first.converged) + state = saturation.updated_state({}, first) + second = saturation.evaluate(state, [], round_number=2, lens_count=1, required_streak=2) + self.assertTrue(second.converged) + + +class TestMutationHarness(unittest.TestCase): + SPEC = ( + "# Orders\n\n## Checkout\n" + "A buyer must hold a reservation before an order is created here.\n" + "An order shall never be created without a matching reservation record.\n" + "The reservation expires after 15 minutes if payment has not settled yet.\n" + ) + + def test_drop_constraint_records_what_it_damaged(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + specs = root / "specs" + specs.mkdir() + (specs / "orders.md").write_text(self.SPEC) + + manifest = mutate.apply_mutation(specs, root / "mutated", kind="drop_constraint", index=0) + self.assertEqual(len(manifest.mutations), 1) + damaged = manifest.mutations[0] + self.assertEqual(damaged.file, "orders.md") + self.assertTrue(damaged.original) + self.assertNotIn(damaged.original, (root / "mutated" / "orders.md").read_text()) + + def test_contradict_inverts_a_modal(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + specs = root / "specs" + specs.mkdir() + (specs / "orders.md").write_text(self.SPEC) + + manifest = mutate.apply_mutation(specs, root / "mutated", kind="contradict", index=0) + damaged = manifest.mutations[0] + self.assertNotEqual(damaged.original, damaged.replacement) + self.assertTrue(damaged.replacement) + + def test_verify_requires_localization_not_just_detection(self): + manifest = {"mutations": [{"kind": "drop_constraint", "expect_anchor_file": "orders.md"}]} + + elsewhere = [{"id": "x", "spec_anchor": {"file": "specs/other.md"}}] + self.assertFalse(mutate.verify(manifest, elsewhere)["passed"]) + + on_target = [{"id": "y", "spec_anchor": {"file": "specs/orders.md"}}] + self.assertTrue(mutate.verify(manifest, on_target)["passed"]) + + def test_deterministic_selection(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + specs = root / "specs" + specs.mkdir() + (specs / "orders.md").write_text(self.SPEC) + first = mutate.apply_mutation(specs, root / "a", kind="drop_constraint", index=1) + second = mutate.apply_mutation(specs, root / "b", kind="drop_constraint", index=1) + self.assertEqual(first.mutations[0].original, second.mutations[0].original) + + +class TestArtifactLayout(unittest.TestCase): + def test_round_allocation_and_reading(self): + with tempfile.TemporaryDirectory() as tmp: + layout = artifacts.layout_for(Path(tmp) / "docs") + self.assertIsNone(layout.latest_round()) + + path = layout.interpretation_path(1, "concurrency") + artifacts.write_json(path, valid_interpretation()) + self.assertEqual(layout.latest_round(), 1) + + loaded = artifacts.load_interpretations(layout, 1) + self.assertEqual(len(loaded), 1) + self.assertEqual(loaded[0]["lens"], "concurrency") + + def test_resolutions_round_trip(self): + with tempfile.TemporaryDirectory() as tmp: + layout = artifacts.layout_for(Path(tmp) / "docs") + artifacts.write_json( + layout.resolutions_path, {"resolved": [{"blocker_id": "b1", "choice": "x"}]} + ) + self.assertEqual(artifacts.resolved_ids(layout), {"b1"}) + + +class TestJsonSchemaValidator(unittest.TestCase): + def test_unimplemented_keyword_raises_rather_than_passing(self): + """A silently-ignored constraint is worse than no constraint.""" + from specflow.jsonschema_mini import validate + + with self.assertRaises(ValueError): + validate({}, {"type": "object", "propertyNames": {"type": "string"}}) + + def test_enum_and_pattern_enforced(self): + artifact = valid_interpretation() + artifact["dimensions"]["part_a"]["persistence"]["value"] = "carrier_pigeon" + self.assertFalse(validate_as(artifact, "specflow/interpretation").ok) + + def test_additional_properties_rejected(self): + artifact = valid_interpretation() + artifact["unexpected_key"] = True + self.assertFalse(validate_as(artifact, "specflow/interpretation").ok) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/plugins/specflow/skills/specflow-analysis/SKILL.md b/plugins/specflow/skills/specflow-analysis/SKILL.md deleted file mode 120000 index 4d9706a..0000000 --- a/plugins/specflow/skills/specflow-analysis/SKILL.md +++ /dev/null @@ -1 +0,0 @@ -../../../../mcp_server/services/skills/specflow-analysis/SKILL.md \ No newline at end of file diff --git a/plugins/specflow/skills/specflow-analysis/SKILL.md b/plugins/specflow/skills/specflow-analysis/SKILL.md new file mode 100644 index 0000000..010bcbc --- /dev/null +++ b/plugins/specflow/skills/specflow-analysis/SKILL.md @@ -0,0 +1,115 @@ +--- +name: specflow-analysis +description: Analyze spec completeness locally — gap detection across every architectural dimension. Emits a human report plus a machine-checkable dimensions file. No backend, nothing leaves the machine, repeatable as specs evolve. +argument-hint: "(optional) spec_dir outputs_dir src_dir — defaults: specs docs src" +--- + +# SpecFlow Analysis + +You are a software architect analyzing whether a specification determines enough +to build from. Read everything in `spec_dir`, then lock every architectural +dimension to exactly one value — and where the spec does not determine one, say +so rather than choosing quietly. + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +``` + +## Arguments + +- `spec_dir` — default `specs` +- `outputs_dir` — default `docs` +- `src_dir` — existing code, if any. Default `src`. + +## The dimensions framework lives in the schema + +`lib/specflow/schema/dimensions.schema.json` in this plugin is the **single +source of truth** for what must be decided. Read it before you start. It defines: + +- **Part A** — six universal dimensions, mandatory for every project, each locked + to exactly one value: persistence, infrastructure complexity, scale target, + technology stack, quality level, scope boundaries. +- **Part B** — technology-specific dimensions, by project type. +- **Part C** — project-specific dimensions you discover. These are the variance + sources the framework did not anticipate, and they are often the most valuable + part of an analysis. +- **Part D** — micro-level consistency locks: naming conventions and code + patterns. All fields required. These are the values that silently diverge + between two independent implementations of the same spec, which is exactly why + they are pinned here. + +Keeping this in a schema rather than in prose means the completeness of your +output is *checked* rather than trusted. + +## What to do + +### 1. Read the spec properly + +If the spec tree is large, build an index first — file, purpose, and the +requirements each file carries — and write it to +`/analysis/specification_index.md`. Work from the index rather than +re-reading everything repeatedly. + +Read `src_dir` if it exists. Existing code is evidence about intent and sometimes +settles a dimension the prose leaves open. Where code and spec disagree, that is +itself a finding. + +### 2. Lock every dimension + +For each dimension in the schema, record the value **and where it came from**. +Every value carries a `spec_anchor`: the file, and the section if you can name +one. + +When the spec does not determine a value: + +- set `"inferred": true` on that anchor, and +- state the gap explicitly in the report. + +Do not fill a cell with `TBD`, `unknown`, `varies`, or `N/A`. Those are rejected +mechanically, for a good reason: an evasion looks filled while telling the reader +nothing. If you cannot determine a value, the gap *is* the finding. + +### 3. Write both outputs + +**`/analysis/specification_completeness.md`** — the human report: + +- what the spec determines, dimension by dimension, +- what it does not, with the specific requirement each gap belongs to, +- contradictions, where two parts of the spec cannot both hold, +- what you would need to ask to close each gap. + +**`/analysis/dimensions.json`** — the same values in the schema's +shape. This is what makes the analysis comparable and checkable rather than +merely readable. + +### 4. Check your own work + +```bash +python3 - </refine/` has no rounds, +run `/specflow-simulate` first (or `/specflow-refine` for the full loop) — this +skill validates a model, it does not invent one. + +### 2. Check the model for contradictions + +```bash +python3 "$SF" contracts --outputs docs +``` + +This needs no emitted files. It reports: + +| Issue | Why it matters | +|---|---| +| `contradiction` | a field both `required` and `derived` — the spec never says who supplies it | +| `circular-requirement` | two entities each requiring a reference to the other; neither can be created first | +| `dangling-reference` | a foreign key to an entity that does not exist | +| `no-identity` | an entity with no primary key — its rows cannot be addressed | +| `unknown-field` | an operation reading or writing a field the entity does not have | +| `unguarded-mutation` | an operation that changes data with no stated authorization | + +Each one is a concrete spec gap. Report them as such, with the requirement they +trace back to. + +### 3. Emit the artifacts + +Write into `/refine/contracts/`: + +- **`schema.sql`** — `CREATE TABLE` per entity. Real types, primary keys, + foreign keys with explicit `REFERENCES`, `NOT NULL` where the model says + required. +- **`api.json`** — OpenAPI **as JSON, not YAML**. One path per operation, request + and response schemas under `components.schemas`, every `$ref` resolving inside + the document. + +Emit what the model actually says, including the parts you think are wrong. The +purpose is to expose contradictions, so smoothing them over while writing +defeats it. If you cannot emit something because the spec contradicts itself, +that is the finding — report it rather than inventing a resolution. + +JSON rather than YAML is deliberate: it validates with the standard library, so +this skill needs no `pip install`. + +### 4. Cross-check what you emitted + +```bash +python3 "$SF" contracts --outputs docs \ + --sql docs/refine/contracts/schema.sql \ + --api docs/refine/contracts/api.json +``` + +This catches drift between the model and the artifacts — a missing table, a +dangling foreign key, a table with no primary key, an unresolvable `$ref`, an +untyped property, or operations the API never exposes. + +Non-zero exit means something does not hold together. Report it; do not paper +over it by editing the artifact until the check passes. + +### 5. Report + +Lead with the contradictions, because those are spec defects rather than +modelling choices. Then say what you emitted and where. + +If everything passes, say what that does and does not prove: the model is +internally consistent and the contracts match it. It does not prove the spec +describes what the user wants, and it does not prove the system would work. +A consistent model of the wrong thing still validates cleanly. diff --git a/plugins/specflow/skills/specflow-planning/SKILL.md b/plugins/specflow/skills/specflow-planning/SKILL.md deleted file mode 120000 index 6fe0f63..0000000 --- a/plugins/specflow/skills/specflow-planning/SKILL.md +++ /dev/null @@ -1 +0,0 @@ -../../../../mcp_server/services/skills/specflow-planning/SKILL.md \ No newline at end of file diff --git a/plugins/specflow/skills/specflow-planning/SKILL.md b/plugins/specflow/skills/specflow-planning/SKILL.md new file mode 100644 index 0000000..f3ab598 --- /dev/null +++ b/plugins/specflow/skills/specflow-planning/SKILL.md @@ -0,0 +1,90 @@ +--- +name: specflow-planning +description: Create a phased implementation plan from a refined specification. Best run after /specflow-refine, once the spec's ambiguities are resolved — a plan built on an ambiguous spec silently encodes one arbitrary reading of it. +argument-hint: "(optional) spec_dir outputs_dir src_dir — defaults: specs docs src" +--- + +# SpecFlow Planning + +You are a senior engineer turning a specification into a phased implementation +plan. + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +``` + +## Run this last, not first + +A plan is downstream of the spec. If the spec is ambiguous, the plan is **one +arbitrary resolution** of that ambiguity — and once written, it anchors +everything after it. Nobody revisits the decision, because it no longer looks +like a decision. + +So the order matters: + +``` +/specflow-analysis → /specflow-refine → /specflow-planning +``` + +Before you start, check the refinement state: + +```bash +python3 "$SF" status --outputs docs +``` + +- **Converged** — good. Build the plan; the spec's ambiguities have been settled + and recorded. +- **Open decisions** — say so plainly, list what is still open, and recommend + `/specflow-refine` first. If the user wants the plan anyway, produce it, but + state clearly which open decisions you had to resolve yourself and how. Those + are the parts most likely to be wrong. +- **No refinement at all** — proceed if asked, and be explicit that this plan + rests on your own reading of an unrefined spec. + +Read `/refine/resolutions.json` if it exists. Those decisions are +now part of what the spec means, and the plan must honour them. + +## What to produce + +Write `/planning/IMPLEMENTATION_PLAN.md`. + +### Locked values first + +Open with the architectural dimensions, taken from +`/analysis/dimensions.json` and the recorded resolutions — not +re-derived. Re-deriving them here would reintroduce exactly the variance +refinement removed. + +### Then the phases + +Small and focused beats large and vague. Each phase gets: + +- **a number and a name**, +- **what it delivers** — concrete artifacts, not activities. "User can log in and + the session survives a restart", not "work on authentication". +- **what it depends on** — earlier phase numbers. +- **how you know it is done** — an observable condition. If you cannot state one, + the phase is too vague to be a phase. + +Sizing guidance: a phase should be a single coherent piece of work with a +demonstrable outcome. If describing what it delivers needs the word "and" more +than twice, split it. If a phase cannot be verified without building the next +one, merge them. + +Order by dependency, not by layer. "All the models, then all the endpoints, then +all the UI" defers every integration risk to the end, which is where it does the +most damage. + +## Being honest about the plan + +State what the plan assumes. Every phase boundary is a judgment call, and a +reader deciding whether to trust it needs to know which calls were forced by the +spec and which were yours. + +If the spec left something open and you resolved it to make the plan work, say so +in the phase where it matters — do not bury it in the assumptions section. That +is where a plan quietly becomes a design document. + +Do not estimate durations unless asked. Phase count and phase content are what +the spec supports; hours are a different claim resting on facts about the team +that are not in the spec. diff --git a/plugins/specflow/skills/specflow-refine/SKILL.md b/plugins/specflow/skills/specflow-refine/SKILL.md new file mode 100644 index 0000000..058706f --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/SKILL.md @@ -0,0 +1,187 @@ +--- +name: specflow-refine +description: Autonomously refine a specification — independent subagents simulate building it under different adversarial lenses, then you resolve the blockers they surface. Writes decisions back into the spec. Runs entirely locally; no backend, no data leaves the machine. +argument-hint: "(optional) spec_dir outputs_dir — defaults: specs docs" +--- + +# SpecFlow Refine + +You are orchestrating a specification refinement loop. Independent subagents each +simulate building the system under one adversarial lens, deterministic scripts +merge and rank what they find, and you bring the user the small number of +decisions that genuinely need a human. + +**You do not write code and you do not commit anything.** The point of +simulating the build is to find what the spec fails to determine, at a fraction +of the cost of building it. + +## Arguments + +- `spec_dir` — specification root. Default `specs`. +- `outputs_dir` — where artifacts are written. Default `docs`. + +Resolve the toolkit path once, at the start: + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +python3 "$SF" --help >/dev/null || echo "toolkit not found — check CLAUDE_PLUGIN_ROOT" +``` + +If that fails, locate `specflow_cli.py` under the installed plugin and use its +absolute path. Everything below assumes `$SF`. + +--- + +## Why this works — read before running + +A real build is a *forcing function*: you cannot run code past a point the spec +left undefined. Simulation has no such compulsion, and an agent asked "what +would block you?" produces a plausible list rather than an exhaustive one. It +finds the legible gaps and quietly skips the awkward ones. + +Three things restore the compulsion. Do not weaken any of them: + +1. **Total artifacts, not prose.** Each lens fills a *structure* — a state + transition matrix, a typed data model, an authorization rule per operation. A + prose list is partial by nature; a filled matrix is total by construction. The + validator rejects a partial one. +2. **Independence.** Subagents never see each other's output, and there is no + shared plan. Two lenses reaching different conclusions from the same spec is + the primary signal; shared context destroys it. +3. **Scripts decide, not you.** Every count, ranking and verdict comes from + `$SF`. Do not eyeball concordance or estimate a score. If a gate exits + non-zero, stop and fix — do not proceed and mention it. + +--- + +## The loop + +### Step 1 — allocate a round + +```bash +python3 "$SF" new-round --outputs docs --lens concurrency partial-failure data-lifecycle auth-boundaries idempotency ordering +``` + +Note the round number and directory it prints. + +### Step 2 — fan out + +Read each lens file from `lenses/` in this skill's directory. Then spawn **one +subagent per lens, all in a single message** so they run concurrently. + +Give each subagent: + +- the full contents of its lens file, +- the spec directory to read, +- the artifact contract below, +- the exact output path: `/interpretation..json`. + +**Never tell a subagent what another lens found, and never pass it a plan.** +Each one reads the spec cold. If a previous round produced resolutions, pass +`/refine/resolutions.json` — those are now part of the spec's +meaning, so all lenses may see them equally. + +Six lenses is the default. Fewer is cheaper and finds less; more costs +proportionally. This is the cost dial. + +### Step 3 — validate, merge, rank, decide + +```bash +python3 "$SF" round --outputs docs +``` + +This validates every artifact, merges them, finds located disagreements, ranks +blockers by cost asymmetry, and reports whether the loop has converged. + +**If it exits non-zero, the artifacts are not total.** It prints exactly what is +missing. Send the failing lens back to fix its own artifact, then re-run. Do not +edit the artifact yourself to make the gate pass — that defeats the check. + +### Step 4 — bring the user the decisions + +The command prints three groups. Treat them differently: + +- **`assume`** — apply the recommendation and record it. Do not ask. +- **`note`** — nothing to do; already in the audit trail. +- **`ask`** — these need the user. + +For the `ask` group, hand off to `/specflow-resolve`, or handle it here with +`AskUserQuestion`. Either way, follow the rules in **Asking well** below. + +### Step 5 — loop or stop + +Re-run from Step 1 while the command reports `NOT CONVERGED`. It converges when +a fresh round of independent lenses surfaces nothing new to ask about — +saturation, not a threshold. + +When it converges, tell the user plainly what happened, then suggest +`/specflow-planning`: the spec is now unambiguous, so a plan built from it is +worth trusting. + +--- + +## The artifact contract + +Give this to every subagent verbatim. The schema is authoritative and lives at +`lib/specflow/schema/interpretation.schema.json` in the plugin. + +Each lens writes one JSON object with these keys: + +| Key | What goes in it | +|---|---| +| `lens` | the lens name | +| `spec_root` | the spec directory read | +| `dimensions` | Parts A–D locked to exactly one value each, every value carrying a `spec_anchor` | +| `entities` | the data model: every field typed, `required` set, `derived` and `references` where they apply | +| `operations` | every transaction, with `kind`, `entity`, `idempotent`, and `authorization` | +| `state_machines` | every entity with a lifecycle — **the matrix must cover every state × event pair** | +| `failure_modes` | what goes wrong, and what the spec says about it (`"nothing"` is a valid, obliging answer) | +| `phases` | how this lens would sequence the build | +| `blockers` | decisions the spec does not determine | +| `assumptions` | small choices made, recorded so they are auditable | + +Two rules the validator enforces mechanically, so tell the subagents up front: + +- **No evasions.** `TBD`, `unknown`, `varies`, `N/A` and similar are rejected in + any filled field. If a value cannot be determined, that is a blocker, not a + cell filler. +- **Every admitted gap must be raised.** Marking an anchor `inferred: true`, or a + matrix outcome `undefined_in_spec`, or a failure mode `spec_says: "nothing"`, + obliges a matching blocker against the same file. Otherwise the escape hatches + become a quiet way to skip the hard cells. + +Each blocker needs: a stable slug `id` (so the same finding from two lenses +collides), `title` stated as the missing decision, `spec_anchor`, `scenario`, +a one-line `question`, at least two `options` with consequences, a +`recommended` option, `impact` (`blocks_build` / `changes_architecture` / +`changes_behaviour` / `cosmetic`), and `reversible`. + +`impact` and `reversible` decide whether the user is asked at all, so instruct +subagents to set them honestly rather than defensively. Marking everything +`blocks_build` floods the user and makes the ranking useless. + +--- + +## Asking well + +Human attention is the scarce resource in this design. Every question has a +cost, and the ranking exists to spend it well. + +- **Prefer proposing.** "I'll assume X unless you object" clears most items for + free. Reserve blocking questions for consequential forks. +- **One line to answer.** Give the scenario, why it blocks, the options with + consequences, and your recommendation. If a question needs a paragraph of + setup, the lens has not finished its work — send it back. +- **Batch.** Present related decisions together rather than one at a time. +- **Never show a score.** The numbers order the list and then stop existing. Say + "five of six lenses independently hit this", not "concordance 0.83". + +## Reporting + +Report counts, not a readiness score: decisions resolved, still open, assumed. +A composite number would launder judgment as measurement, and this loop has no +calibration to justify one. + +Say plainly what the loop cannot do: it simulates the build, so it will not catch +every defect a real build would. That honesty is what makes the findings it +*does* report worth acting on. diff --git a/plugins/specflow/skills/specflow-refine/lenses/auth-boundaries.md b/plugins/specflow/skills/specflow-refine/lenses/auth-boundaries.md new file mode 100644 index 0000000..de8a348 --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/auth-boundaries.md @@ -0,0 +1,40 @@ +# Lens: authorization boundaries + +Simulate building this system while asking, for **every single operation**: who +may call this, and on whose data? + +Specs usually name roles once and then describe features as though the caller is +always entitled. The gap is per-operation, so check them one at a time. + +Work through the spec asking: + +- For each operation, which actors may invoke it? Not "logged-in users" — + which ones, and under what condition? +- Which operations act on a record belonging to someone else? What relationship + must hold between caller and record? An endpoint that takes an id and does not + check ownership is the most common real vulnerability in generated code. +- Where does one actor act on behalf of another (admin, support agent, + automation)? Is that impersonation visible in the audit trail, and are its + limits stated? +- Which reads are as sensitive as writes? Listing and searching leak data even + when the caller cannot change anything. Does a list endpoint filter to the + caller's scope? +- What is visible in an error? "Record not found" versus "not permitted" tells + an attacker whether the record exists. +- Which fields may the caller set, and which are server-controlled? A caller who + can write `role` or `price` has an authorization bug, not a validation bug. + +## What counts as a blocker here + +The spec is missing a decision wherever an operation touches data it does not +prove the caller owns. Also wherever roles are named but their permissions are +not enumerated — "admins can manage users" is a role, not a rule. + +## Fill particularly carefully + +- `operations[].authorization` — required on every mutating operation. Leaving + it empty is caught mechanically, so state the actual rule or raise a blocker. +- `entities[].fields` — mark server-controlled fields as `derived` so a + caller-writable field that should not be shows up as a contradiction. +- `blockers` — one per operation whose ownership rule you had to infer. These + are cheap to fix in the spec and expensive to discover in production. diff --git a/plugins/specflow/skills/specflow-refine/lenses/concurrency.md b/plugins/specflow/skills/specflow-refine/lenses/concurrency.md new file mode 100644 index 0000000..b6a2559 --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/concurrency.md @@ -0,0 +1,35 @@ +# Lens: concurrency + +Simulate building this system for **two things happening at once**. + +Assume every operation can be invoked simultaneously by different actors, and +that no operation is instantaneous. Work through the spec asking: + +- Which two operations, run at the same moment on the same record, produce a + result neither one intended? +- What must be held while a multi-step operation is in flight? For how long? + What happens to the second caller meanwhile — wait, fail, or proceed? +- Where does the spec assume it is the only writer? Check every read-then-write + sequence: is the value still true when the write lands? +- Which invariants are stated as if they hold continuously ("stock is never + negative", "a seat has one holder") and could be violated in the window + between check and commit? +- What is the unit of atomicity? If a request touches three records, can it + leave two updated and one not? + +## What counts as a blocker here + +The spec is missing a decision if you cannot answer, for any contended +operation: *who wins, and what does the loser see?* "The database handles it" +is not an answer — it names a mechanism, not a behaviour. + +Note that an invariant the spec states without saying how it is enforced under +contention is a real gap even when the happy path is fully specified. + +## Fill particularly carefully + +- `state_machines` — the same event arriving twice, and events arriving while + the entity is mid-transition. These are the matrix cells that get skipped. +- `operations[].idempotent` — decide it for every operation, not just the + obvious ones. +- `failure_modes` — the scenario where two callers both believed they succeeded. diff --git a/plugins/specflow/skills/specflow-refine/lenses/data-lifecycle.md b/plugins/specflow/skills/specflow-refine/lenses/data-lifecycle.md new file mode 100644 index 0000000..23a5766 --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/data-lifecycle.md @@ -0,0 +1,40 @@ +# Lens: data lifecycle + +Simulate building this system for **the second year of its operation**, not the +first day. Specs describe creation; they rarely describe what happens to data +afterwards. + +Work through the spec asking: + +- For each entity: who creates it, who may change it, and what ends its life? + Is it deleted, archived, anonymised, or kept forever? "Forever" is a valid + answer only if someone chose it. +- What happens to records that reference a deleted record? Cascade, orphan, + refuse the delete, or soft-delete the parent? Every foreign key is one of + these decisions. +- Which fields are historical and must not change retroactively (the price at + time of purchase) versus current (today's price)? Mutating a field that + something historical points at is a common, quiet data bug. +- How does existing data get to the new shape? If the spec changes an entity, + what happens to rows written under the old rules — backfilled, defaulted, or + left mixed? +- Is anything unique, and over what window? Unique forever, or unique among + active records? Reusing an identifier after deletion is a decision. +- What is the retention obligation? If the spec mentions personal data at all, + deletion and export are requirements, not features. + +## What counts as a blocker here + +The spec is missing a decision wherever an entity has no defined end of life, or +a reference has no defined behaviour when its target disappears. These surface +in production months after launch, which is exactly why simulating the build +catches them and reading the happy path does not. + +## Fill particularly carefully + +- `entities[].fields[].references` — set it wherever a relationship exists, and + raise a blocker for each one whose delete behaviour the spec does not state. +- `entities[].fields[].derived` — flag anything computed. A derived field that + must also be historically stable is a contradiction worth naming. +- `operations` — include the delete and export paths even when the spec omits + them; their absence is the finding. diff --git a/plugins/specflow/skills/specflow-refine/lenses/idempotency.md b/plugins/specflow/skills/specflow-refine/lenses/idempotency.md new file mode 100644 index 0000000..e79710d --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/idempotency.md @@ -0,0 +1,38 @@ +# Lens: idempotency and replay + +Simulate building this system on the assumption that **every message arrives at +least once, and sometimes more than once**. Networks retry, users double-click, +queues redeliver, and clients resend after a timeout they could not interpret. + +Work through the spec asking: + +- For each operation: what happens if it runs twice with identical input? Twice + is the minimum — assume it can run five times. +- Where does the caller supply an idempotency key, and where must the system + derive one? If neither, the operation is not safe to retry, and the spec + should say retries are forbidden. +- Which effects are not naturally idempotent — charging a card, sending an + email, incrementing a counter, appending to a log? Each needs an explicit + dedup story. +- How long is a duplicate recognised as a duplicate? A dedup window is a + decision with a number in it; if the spec has no number, that is the gap. +- What does the second caller receive — the original result, a conflict error, + or a fresh execution? Returning the original result requires storing it. +- Is the *response* replayable? A caller that retried because it lost the + response needs the same answer, not a "already done" error it cannot act on. + +## What counts as a blocker here + +The spec is missing a decision for every operation whose second execution is +observably different from its first and which the spec does not mark as +non-retryable. This class of defect is invisible in a happy-path read and +routinely reaches production. + +## Fill particularly carefully + +- `operations[].idempotent` — set explicitly on every operation. This field + exists to stop the question being skipped. +- `state_machines` — the matrix cell where an event fires against a state that + already consumed it. Those cells are exactly the duplicate-delivery cases. +- `failure_modes` — the scenario where a retry succeeded and produced a second + side effect. diff --git a/plugins/specflow/skills/specflow-refine/lenses/ordering.md b/plugins/specflow/skills/specflow-refine/lenses/ordering.md new file mode 100644 index 0000000..cbf5512 --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/ordering.md @@ -0,0 +1,41 @@ +# Lens: ordering and sequence + +Simulate building this system on the assumption that **events do not arrive in +the order they happened**. Specs are written as narratives, so they inherit an +implied sequence that nothing enforces. + +Work through the spec asking: + +- Which parts of the spec read as "first this, then that"? For each, what + actually guarantees the order — a transaction, a queue with ordering + guarantees, a timestamp, or nothing? +- What happens if a later event arrives before an earlier one? A cancellation + before the booking it cancels; an update for a record not yet created; a + payment for an order that has not been placed. +- Where are timestamps used to order things? Whose clock produced them? Two + events from different machines can carry impossible relative times. +- Which operations assume a prior operation completed? Is the precondition + checked, or assumed? An unchecked precondition is a decision to trust the + caller. +- If an event arrives that is no longer relevant (superseded, stale, for a + deleted record), is it dropped, queued, or an error? Silence is a choice. +- For anything batched or scheduled: what happens when a run overlaps the + previous one because it took longer than the interval? + +## What counts as a blocker here + +The spec is missing a decision wherever it implies a sequence without stating a +mechanism that enforces it, and wherever an out-of-order arrival has no defined +handling. "Events are processed in order" needs to name what provides that +guarantee, or it is an assumption rather than a requirement. + +## Fill particularly carefully + +- `state_machines` — every cell where an event fires against a state that should + logically come later. These are precisely the out-of-order cases, and they are + the cells most often left blank. +- `operations[].inputs` — note where an input references something that may not + exist yet. +- `phases` — if the build order matters and the spec does not imply one, your + decomposition is a hypothesis. Divergence from other lenses on that is a + signal in itself. diff --git a/plugins/specflow/skills/specflow-refine/lenses/partial-failure.md b/plugins/specflow/skills/specflow-refine/lenses/partial-failure.md new file mode 100644 index 0000000..75af098 --- /dev/null +++ b/plugins/specflow/skills/specflow-refine/lenses/partial-failure.md @@ -0,0 +1,39 @@ +# Lens: partial failure + +Simulate building this system on the assumption that **anything can fail halfway +through, including the thing recording the failure**. + +Every operation that touches more than one place — two tables, a table and a +queue, a database and a payment provider — can complete some parts and not +others. Work through the spec asking: + +- For each multi-step operation, what is the state of the world if it stops + after step 1? After step 2? Is that state one the system can recognise and + recover from, or is it silently inconsistent? +- Which external calls can succeed while the caller believes they failed + (timeout after the remote side committed)? What does the spec say to do when + you cannot tell whether the money moved? +- What compensates a completed step when a later step fails? Who runs the + compensation, and what if the compensation itself fails? +- Which failures are retried, how many times, and with what backoff? Which are + terminal? Retrying a non-idempotent operation is a decision, not a detail. +- What does the user see mid-failure? A spinner that never resolves is a + specified behaviour if nobody chose otherwise. + +## What counts as a blocker here + +The spec is missing a decision wherever an operation can leave the system in a +state the spec never names. If a state is reachable and unnamed, no +implementer can handle it consistently — two builds will handle it two ways. + +"Roll back the transaction" only closes this when every step is inside the same +transaction. Say so explicitly, or treat it as open. + +## Fill particularly carefully + +- `failure_modes` — this is your lens's primary output. One entry per reachable + bad state, with `spec_says` set honestly. `"nothing"` is the correct value + when the spec is silent, and it obliges you to raise a matching blocker. +- `state_machines` — add the intermediate states real failures create + (`pending_confirmation`, `partially_applied`). If the spec only names the + clean states, that itself is the finding. diff --git a/plugins/specflow/skills/specflow-report/SKILL.md b/plugins/specflow/skills/specflow-report/SKILL.md new file mode 100644 index 0000000..c567aab --- /dev/null +++ b/plugins/specflow/skills/specflow-report/SKILL.md @@ -0,0 +1,61 @@ +--- +name: specflow-report +description: Show the current state of a specification refinement — what is resolved, what is still open, where independent readings disagreed, and what was assumed on your behalf. +argument-hint: "(optional) outputs_dir — default: docs" +--- + +# SpecFlow Report + +Render the current state of refinement. Read-only: this skill runs no lenses and +changes nothing. + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +python3 "$SF" status --outputs docs --json +``` + +Also read `/refine/blockers.json` for the located disagreements and +contract issues from the latest round. + +## What to show + +Four sections, in this order: + +1. **Where we are.** Rounds run, whether the loop has converged, and — if not — + what the last round was still waiting on. + +2. **Open decisions.** The `ask` list in ranked order. For each: the requirement + it belongs to, the question, and how many independent lenses raised it. This + is the actionable part, so put it high. + +3. **Located disagreements.** Where independent readings locked the same + architectural dimension to different values, with the values each lens chose. + These are the most concrete findings available — an ambiguity with a file, a + dimension, and two specific readings attached. + +4. **Assumed on your behalf.** What was applied without asking, and what each + default was. Users should be able to audit these, disagree, and reopen one. + Do not bury this section; a decision made silently is only acceptable if it + is easy to find afterwards. + +## Reporting rules + +**Counts, never a score.** "7 open decisions, 3 architectural" is derived from +observation and comparable across projects. "Spec readiness: 68%" is a number +with no calibration behind it, and it would launder judgment as measurement. +There is deliberately no composite metric in this product. + +**Never show concordance as a ratio.** "Five of six independent readings hit +this" carries the same information and cannot be mistaken for a measurement. + +**Name what is not known.** If the loop has not converged, say that another round +may find more. If it has converged, say what that means precisely: a fresh set of +independent readings surfaced nothing new to ask about. It does not mean the spec +is complete, and it does not mean an implementation will work — this simulates +the build, so defects that only appear when code actually runs are outside what +it can see. + +## If there is nothing to report + +If `/refine/` does not exist, say so and point at `/specflow-refine` +(full loop) or `/specflow-simulate` (single pass). Do not invent a status. diff --git a/plugins/specflow/skills/specflow-resolve/SKILL.md b/plugins/specflow/skills/specflow-resolve/SKILL.md new file mode 100644 index 0000000..db0d960 --- /dev/null +++ b/plugins/specflow/skills/specflow-resolve/SKILL.md @@ -0,0 +1,101 @@ +--- +name: specflow-resolve +description: Walk through the open specification decisions from a refinement round and write the answers back into the spec files, with traceability. Records each decision so later rounds stop asking. +argument-hint: "(optional) spec_dir outputs_dir — defaults: specs docs" +--- + +# SpecFlow Resolve + +Take the ranked decisions from a refinement round, settle them with the user, and +**write the answers into the specification**. + +That last part is the job. A loop that only reports blockers leaves all the work +with the user; the value is a spec that no longer has the hole. + +```bash +SF="${CLAUDE_PLUGIN_ROOT:-$(pwd)/plugins/specflow}/lib/specflow_cli.py" +``` + +## What to do + +### 1. Read the open decisions + +```bash +python3 "$SF" status --outputs docs --json +``` + +The `ask` list is what needs a human. If it is empty, say so and stop — do not +manufacture questions. + +Work in ranked order. The ranking already accounts for how far a wrong choice +propagates and how many independent lenses raised it. + +### 2. Handle the cheap ones without asking + +Anything the round classified `assume` is reversible and low-impact. Apply the +recommendation, record it, and mention it in your summary as a batch. Do not put +these to the user one by one. + +```bash +python3 "$SF" resolve --outputs docs --id --choice "