fix: redesign benchmark integrity foundation

.github/workflows/validate.yml

@@ -17,3 +17,4 @@ jobs:
       - run: python3 scripts/verify_no_copied_skills.py
         # Note: add --verify-content for full content integrity checks;
         # omitted by default to avoid CI dependency on GitHub raw content availability.
+      - run: python3 -m unittest discover

CLAUDE.md

@@ -20,8 +20,8 @@ python3 scripts/refresh_external_skills.py --approve     # apply pin + hash upda
 python3 scripts/refresh_external_skills.py --check       # CI: fail if updates needed
 python3 scripts/verify_no_copied_skills.py               # validate manifest structure
 python3 scripts/verify_no_copied_skills.py --verify-content  # fetch + hash verification
-python3 scripts/run_benchmark.py --all
-python3 scripts/aggregate_stability.py
+python3 scripts/run_benchmark.py --critic react --capture-commit <40-hex-commit> --run-id <uuid>
+python3 -m unittest discover
 ```
 
 ## Design Rules
@@ -57,5 +57,9 @@ but should be verified before adding a new owner to `TRUSTED_OWNERS`.
 
 - Fixtures: `research/benchmarks/fixtures/{react,next,react-native}/` (8 per critic)
 - Results: `research/benchmarks/results/`
-- Scoring: rubric-coverage evaluation (prompt checklist vs annotated fixture issues)
-- Seeds: 3 jackknife windows per critic; aggregate stability in `stability-report.md`
+- Corpus status: reviewed descriptive foundation only; no provider pilot or committed captures exist.
+- Controls: `react-06`, `next-02`, `rn-01`, `rn-02`, and `rn-07` are intentional-clean restraint controls.
+- Candidate treatment: only the committed inline `SKILL.md`; referenced rubrics, agents, router material, and external skills are excluded.
+- Future model scoring: exact rule-and-line response matching from self-attested repository-native capture blobs at an exact Git commit.
+- Windows: deterministic jackknife sensitivity references, never additional samples or provider/model seeds.
+- Boundaries: this corpus supports no performance, authenticity, severity-quality, full-skill, or statistical claim. Historical reports remain rejected.

README.md

@@ -56,10 +56,41 @@ python3 scripts/refresh_external_skills.py --approve     # apply pin + hash upda
 python3 scripts/refresh_external_skills.py --check       # CI: fail if updates needed
 python3 scripts/verify_no_copied_skills.py               # validate manifest structure
 python3 scripts/verify_no_copied_skills.py --verify-content  # fetch + hash verification
-python3 scripts/run_benchmark.py --all
-python3 scripts/aggregate_stability.py
+python3 scripts/run_benchmark.py --critic react --capture-commit <40-hex-commit> --run-id <uuid>
+python3 -m unittest discover
 ```
 
+## Reviewed benchmark corpus and future exact scorer
+
+`research/benchmarks/` is a reviewed descriptive foundation for a future exact
+structured defect-detection benchmark. It has eight synthetic fixtures per
+critic, including five intentional-clean restraint controls: `react-06`,
+`next-02`, `rn-01`, `rn-02`, and `rn-07`. Each fixture exposes the source
+context, assumptions, requirements, rule, span-level evidence, and official
+source used for adjudication. It does not represent provider execution or human
+expert certification.
+
+Candidate and baseline prompts have identical substantive instructions, rule
+definitions, source context, evidence-span contract, anti-speculation guard,
+and permission to return no findings. Candidate treatment adds only the
+committed inline `SKILL.md`; referenced rubrics, agents, router material, and
+external skills are excluded from both treatments. Each response is strict JSON
+containing only findings with a known `rule_id`, sorted unique in-range
+`evidence_lines`, and `assertion: "present"`.
+
+The future runner never calls a provider. It requires run-specific immutable
+captures before it can score any execution; provider/model/runtime/tool labels
+would remain self-attested repository-native custody rather than independent
+proof of authenticity. No provider pilot or capture set exists now. The corpus
+therefore supports no performance, full-skill, prose/remediation, severity,
+provider-authenticity, or statistical-superiority claim. Historical reports
+remain rejected.
+
+Publication reserves the exact bundle directory before writing and adds a final
+`COMPLETE` marker only after both report and manifest are in place. This is a
+fail-closed local publication guard; it does not claim crash-durable or
+independently witnessed publication.
+
 ## License
 
 Apache 2.0

research/benchmarks/DESIGN.md

@@ -0,0 +1,24 @@
+# Benchmark corpus design and adjudication
+
+This is a reviewed descriptive foundation for a future exact structured
+defect-detection benchmark. It contains 24 synthetic, independently reviewable
+fixtures: eight each for React, Next.js, and React Native. Every label names a
+catalog rule, cites public source lines, records the public assumptions needed
+to assess it, and cites an official primary source. This is corpus
+adjudication, not provider execution or a human-expert certification.
+
+The corpus has five intentional-clean restraint controls: `react-06`,
+`next-02`, `rn-01`, `rn-02`, and `rn-07`. Their `ADVERSARIAL` difficulty means
+they are designed to attract unsupported advice; it does not mean they contain
+a hidden defect.
+
+The candidate and baseline receive the same source context, rule catalog,
+response schema, evidence-span contract, and permission to return no findings.
+The candidate treatment adds only the committed inline `SKILL.md`. Referenced
+rubrics, agents, router material, and external skills are excluded from both
+prompts.
+
+No provider pilot, capture, model comparison, or aggregate run exists in this
+repository. The corpus therefore supports no performance, provider-authenticity,
+severity-quality, full-skill, or statistical claim. Historical benchmark
+reports remain rejected rather than evidence of the redesigned corpus.

research/benchmarks/fixtures/next/fixture-01.yaml

@@ -1,32 +1,56 @@
+schema_version: 2
 id: next-01
 category: rsc-boundary
-description: use client placed too high, blocking RSC optimization
-code: |
-  // app/dashboard/page.tsx
-  'use client';
-  import { getServerSession } from 'next-auth';
-  import { db } from '@/lib/db';
-  import RevenueChart from './RevenueChart'; // needs interactivity
-  import StatCard from './StatCard';          // pure display
+description: A Client Component directive conflicts with server-only work in the same module.
+intentional_clean: false
+public:
+  file_path: app/dashboard/page.tsx
+  language: TypeScript JSX
+  frameworks:
+    next: '15'
+    react: '19'
+  runtime: nodejs
+  assumptions:
+    - db is a server-only database client.
+    - RevenueChart is an interactive Client Component.
+  requirements:
+    - The dashboard page must load its session and database data on the server.
+  code: |
+    'use client';
 
-  export default async function DashboardPage() {
-    const session = await getServerSession();
-    const stats = await db.query.stats.findMany();
-    return (
-      <div>
-        <StatCard value={stats.revenue} />
-        <RevenueChart data={stats.chartData} />
-      </div>
-    );
-  }
+    import { getServerSession } from 'next-auth';
+    import { db } from '@/lib/db';
+    import RevenueChart from './RevenueChart';
+    import StatCard from './StatCard';
+
+    export default async function DashboardPage() {
+      const session = await getServerSession();
+      const stats = await db.query.stats.findMany();
+      return <RevenueChart data={stats.chartData} session={session} />;
+    }
+metadata:
+  provenance:
+    kind: synthetic
+    origin: Next.js Server and Client Components documentation
+    revision_reason: Narrowed the label to the direct client-boundary conflict; no browser-leak claim is made.
+  difficulty: HAS-BUGS
+  domains: [server-components, boundaries]
+  adjudication:
+    status: reviewed
+    method: Source-and-context rule adjudication
+    reviewer: codex-test-builder-review
+    reviewed_on: '2026-09-19'
+    sources:
+      - url: https://nextjs.org/docs/app/building-your-application/rendering/client-components
+        supports: The use client directive marks a Client Component entry point.
+    notes: The directive and server-only assumption make the incompatibility explicit.
+    excluded_claims: [Bundle contents and the placement of child component directives are not scored.]
 issues:
-  - severity: MAJOR
-    category: rsc-boundary
-    description: "'use client' on the page prevents server-side data fetching; getServerSession and db.query will fail or be called client-side, leaking db access to the browser bundle"
-    evidence: "'use client'; ... const session = await getServerSession();"
-    expected_checks: [rsc-boundary, server-actions-safety]
-  - severity: MAJOR
+  - id: next-01-client-server-conflict
+    rule_id: next.rsc-boundary
     category: rsc-boundary
-    description: "StatCard is a pure display component forced into the client bundle unnecessarily; only RevenueChart needs 'use client'"
-    evidence: "import StatCard from './StatCard'; // pure display"
-    expected_checks: [rsc-boundary, runtime-choice]
+    severity: MAJOR
+    description: The use client entry point conflicts with the stated server-only session and database work in the same module.
+    evidence_spans:
+      - lines: [1, 3, 4, 9, 10]
+        rationale: The module is marked as a Client Component while importing and awaiting server-only services.

research/benchmarks/fixtures/next/fixture-02.yaml

@@ -1,24 +1,41 @@
+schema_version: 2
 id: next-02
 category: cache-correctness
-description: overly aggressive no-store on every fetch disabling PPR
-code: |
-  // app/products/page.tsx
-  export default async function ProductsPage() {
-    const [products, categories, featuredBanner] = await Promise.all([
-      fetch('/api/products', { cache: 'no-store' }),
-      fetch('/api/categories', { cache: 'no-store' }),
-      fetch('/api/banner/featured', { cache: 'no-store' }),
-    ]);
-    // ...render
-  }
-issues:
-  - severity: MAJOR
-    category: cache-correctness
-    description: "cache: 'no-store' on all fetches disables PPR; categories and banner are stable data and should use revalidate or force-cache; every page visit re-fetches all three even if data hasn't changed"
-    evidence: "fetch('/api/categories', { cache: 'no-store' })"
-    expected_checks: [cache-correctness, ppr-correctness]
-  - severity: MINOR
-    category: cache-correctness
-    description: "no granular revalidate tags; cache invalidation requires a full page re-build rather than targeted revalidatePath/revalidateTag"
-    evidence: "fetch('/api/products', { cache: 'no-store' })"
-    expected_checks: [cache-correctness]
+description: Product data is intentionally uncached because it must reflect immediate writes.
+intentional_clean: true
+public:
+  file_path: app/products/page.tsx
+  language: TypeScript JSX
+  frameworks:
+    next: '15'
+    react: '19'
+  runtime: nodejs
+  assumptions:
+    - Inventory and prices can change immediately after a completed purchase.
+    - This route is the canonical confirmation view following a purchase.
+  requirements:
+    - The confirmation view must not serve a cached inventory or price response.
+  code: |
+    export default async function ProductsPage() {
+      const response = await fetch('https://catalog.example.test/products', { cache: 'no-store' });
+      const products = await response.json();
+      return <ProductList products={products} />;
+    }
+metadata:
+  provenance:
+    kind: synthetic
+    origin: Next.js data fetching documentation
+    revision_reason: Replaced a generic no-store performance preference with a clean freshness control.
+  difficulty: ADVERSARIAL
+  domains: [caching, freshness]
+  adjudication:
+    status: reviewed
+    method: Source-and-context rule adjudication
+    reviewer: codex-test-builder-review
+    reviewed_on: '2026-09-19'
+    sources:
+      - url: https://nextjs.org/docs/app/building-your-application/data-fetching/fetching
+        supports: Next.js supports cache configuration for fetch requests.
+    notes: The stated requirement supports no-store for this route.
+    excluded_claims: [PPR, tags, and broader catalog caching policy are not scored.]
+issues: []

research/benchmarks/fixtures/next/fixture-03.yaml

@@ -1,33 +1,63 @@
+schema_version: 2
 id: next-03
 category: server-actions-safety
-description: server action with no input validation or CSRF protection
-code: |
-  // app/actions/updateProfile.ts
-  'use server';
+description: A Server Action accepts identity and privilege from form data.
+intentional_clean: false
+public:
+  file_path: app/actions/updateProfile.ts
+  language: TypeScript
+  frameworks:
+    next: '15'
+  runtime: nodejs
+  assumptions:
+    - The authenticated session identifies the only profile the caller may update.
+    - Only an administrator may change a user's role.
+  requirements:
+    - The action must derive the target profile and authorization from the authenticated session.
+  code: |
+    'use server';
 
-  export async function updateProfile(formData: FormData) {
-    const userId = formData.get('userId') as string;
-    const bio = formData.get('bio') as string;
-    const role = formData.get('role') as string; // should never come from user
+    export async function updateProfile(formData: FormData) {
+      const userId = formData.get('userId') as string;
+      const bio = formData.get('bio') as string;
+      const role = formData.get('role') as string;
 
-    await db.user.update({
-      where: { id: userId },
-      data: { bio, role }, // role written directly from form input
-    });
-  }
+      await db.user.update({
+        where: { id: userId },
+        data: { bio, role },
+      });
+    }
+metadata:
+  provenance:
+    kind: synthetic
+    origin: Next.js Server Actions documentation
+    revision_reason: Limited the fixture to explicit authorization authority; removed CSRF and stored-XSS speculation.
+  difficulty: FUNDAMENTALLY-FLAWED
+  domains: [authorization, server-actions]
+  adjudication:
+    status: reviewed
+    method: Source-and-context rule adjudication
+    reviewer: codex-test-builder-review
+    reviewed_on: '2026-09-19'
+    sources:
+      - url: https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions-and-mutations#authentication-and-authorization
+        supports: Server Actions should verify authentication and authorization.
+    notes: Both authority violations are established by the supplied session requirements.
+    excluded_claims: [CSRF, validation-library choice, and HTML rendering behavior are not scored.]
 issues:
-  - severity: CRITICAL
+  - id: next-03-form-target-identity
+    rule_id: next.server-actions-safety
     category: server-actions-safety
-    description: "role is written directly from form input with no authorization check; any user can elevate their own role to admin"
-    evidence: "const role = formData.get('role') as string; ... data: { bio, role }"
-    expected_checks: [server-actions-safety, security]
-  - severity: CRITICAL
+    severity: CRITICAL
+    description: The update target is taken from user-controlled form data instead of the authenticated session.
+    evidence_spans:
+      - lines: [4, 9]
+        rationale: userId comes from formData and becomes the database where clause.
+  - id: next-03-form-role
+    rule_id: next.server-actions-safety
     category: server-actions-safety
-    description: "no session/auth check: any unauthenticated caller can update any user's profile by supplying an arbitrary userId"
-    evidence: "const userId = formData.get('userId') as string;"
-    expected_checks: [server-actions-safety, security]
-  - severity: MAJOR
-    category: server-actions-safety
-    description: "no input validation or sanitization on bio; stored XSS if bio is rendered with dangerouslySetInnerHTML elsewhere"
-    evidence: "const bio = formData.get('bio') as string;"
-    expected_checks: [server-actions-safety, security]
+    severity: CRITICAL
+    description: The privileged role field is taken from user-controlled form data without the required administrator authorization.
+    evidence_spans:
+      - lines: [6, 10]
+        rationale: role comes from formData and is written directly to the database.