diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 9bbdc29..18470dc 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -17,3 +17,4 @@ jobs: - run: python3 scripts/verify_no_copied_skills.py # Note: add --verify-content for full content integrity checks; # omitted by default to avoid CI dependency on GitHub raw content availability. + - run: python3 -m unittest discover diff --git a/CLAUDE.md b/CLAUDE.md index 4358103..9fd5c82 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,8 +20,8 @@ python3 scripts/refresh_external_skills.py --approve # apply pin + hash upda python3 scripts/refresh_external_skills.py --check # CI: fail if updates needed python3 scripts/verify_no_copied_skills.py # validate manifest structure python3 scripts/verify_no_copied_skills.py --verify-content # fetch + hash verification -python3 scripts/run_benchmark.py --all -python3 scripts/aggregate_stability.py +python3 scripts/run_benchmark.py --critic react --capture-commit <40-hex-commit> --run-id +python3 -m unittest discover ``` ## Design Rules @@ -57,5 +57,9 @@ but should be verified before adding a new owner to `TRUSTED_OWNERS`. - Fixtures: `research/benchmarks/fixtures/{react,next,react-native}/` (8 per critic) - Results: `research/benchmarks/results/` -- Scoring: rubric-coverage evaluation (prompt checklist vs annotated fixture issues) -- Seeds: 3 jackknife windows per critic; aggregate stability in `stability-report.md` +- Corpus status: reviewed descriptive foundation only; no provider pilot or committed captures exist. +- Controls: `react-06`, `next-02`, `rn-01`, `rn-02`, and `rn-07` are intentional-clean restraint controls. +- Candidate treatment: only the committed inline `SKILL.md`; referenced rubrics, agents, router material, and external skills are excluded. +- Future model scoring: exact rule-and-line response matching from self-attested repository-native capture blobs at an exact Git commit. +- Windows: deterministic jackknife sensitivity references, never additional samples or provider/model seeds. +- Boundaries: this corpus supports no performance, authenticity, severity-quality, full-skill, or statistical claim. Historical reports remain rejected. diff --git a/README.md b/README.md index 930e532..0a3510f 100644 --- a/README.md +++ b/README.md @@ -56,10 +56,41 @@ python3 scripts/refresh_external_skills.py --approve # apply pin + hash upda python3 scripts/refresh_external_skills.py --check # CI: fail if updates needed python3 scripts/verify_no_copied_skills.py # validate manifest structure python3 scripts/verify_no_copied_skills.py --verify-content # fetch + hash verification -python3 scripts/run_benchmark.py --all -python3 scripts/aggregate_stability.py +python3 scripts/run_benchmark.py --critic react --capture-commit <40-hex-commit> --run-id +python3 -m unittest discover ``` +## Reviewed benchmark corpus and future exact scorer + +`research/benchmarks/` is a reviewed descriptive foundation for a future exact +structured defect-detection benchmark. It has eight synthetic fixtures per +critic, including five intentional-clean restraint controls: `react-06`, +`next-02`, `rn-01`, `rn-02`, and `rn-07`. Each fixture exposes the source +context, assumptions, requirements, rule, span-level evidence, and official +source used for adjudication. It does not represent provider execution or human +expert certification. + +Candidate and baseline prompts have identical substantive instructions, rule +definitions, source context, evidence-span contract, anti-speculation guard, +and permission to return no findings. Candidate treatment adds only the +committed inline `SKILL.md`; referenced rubrics, agents, router material, and +external skills are excluded from both treatments. Each response is strict JSON +containing only findings with a known `rule_id`, sorted unique in-range +`evidence_lines`, and `assertion: "present"`. + +The future runner never calls a provider. It requires run-specific immutable +captures before it can score any execution; provider/model/runtime/tool labels +would remain self-attested repository-native custody rather than independent +proof of authenticity. No provider pilot or capture set exists now. The corpus +therefore supports no performance, full-skill, prose/remediation, severity, +provider-authenticity, or statistical-superiority claim. Historical reports +remain rejected. + +Publication reserves the exact bundle directory before writing and adds a final +`COMPLETE` marker only after both report and manifest are in place. This is a +fail-closed local publication guard; it does not claim crash-durable or +independently witnessed publication. + ## License Apache 2.0 diff --git a/research/benchmarks/DESIGN.md b/research/benchmarks/DESIGN.md new file mode 100644 index 0000000..addb4a5 --- /dev/null +++ b/research/benchmarks/DESIGN.md @@ -0,0 +1,24 @@ +# Benchmark corpus design and adjudication + +This is a reviewed descriptive foundation for a future exact structured +defect-detection benchmark. It contains 24 synthetic, independently reviewable +fixtures: eight each for React, Next.js, and React Native. Every label names a +catalog rule, cites public source lines, records the public assumptions needed +to assess it, and cites an official primary source. This is corpus +adjudication, not provider execution or a human-expert certification. + +The corpus has five intentional-clean restraint controls: `react-06`, +`next-02`, `rn-01`, `rn-02`, and `rn-07`. Their `ADVERSARIAL` difficulty means +they are designed to attract unsupported advice; it does not mean they contain +a hidden defect. + +The candidate and baseline receive the same source context, rule catalog, +response schema, evidence-span contract, and permission to return no findings. +The candidate treatment adds only the committed inline `SKILL.md`. Referenced +rubrics, agents, router material, and external skills are excluded from both +prompts. + +No provider pilot, capture, model comparison, or aggregate run exists in this +repository. The corpus therefore supports no performance, provider-authenticity, +severity-quality, full-skill, or statistical claim. Historical benchmark +reports remain rejected rather than evidence of the redesigned corpus. diff --git a/research/benchmarks/fixtures/next/fixture-01.yaml b/research/benchmarks/fixtures/next/fixture-01.yaml index d2c820f..38330e5 100644 --- a/research/benchmarks/fixtures/next/fixture-01.yaml +++ b/research/benchmarks/fixtures/next/fixture-01.yaml @@ -1,32 +1,56 @@ +schema_version: 2 id: next-01 category: rsc-boundary -description: use client placed too high, blocking RSC optimization -code: | - // app/dashboard/page.tsx - 'use client'; - import { getServerSession } from 'next-auth'; - import { db } from '@/lib/db'; - import RevenueChart from './RevenueChart'; // needs interactivity - import StatCard from './StatCard'; // pure display +description: A Client Component directive conflicts with server-only work in the same module. +intentional_clean: false +public: + file_path: app/dashboard/page.tsx + language: TypeScript JSX + frameworks: + next: '15' + react: '19' + runtime: nodejs + assumptions: + - db is a server-only database client. + - RevenueChart is an interactive Client Component. + requirements: + - The dashboard page must load its session and database data on the server. + code: | + 'use client'; - export default async function DashboardPage() { - const session = await getServerSession(); - const stats = await db.query.stats.findMany(); - return ( -
- - -
- ); - } + import { getServerSession } from 'next-auth'; + import { db } from '@/lib/db'; + import RevenueChart from './RevenueChart'; + import StatCard from './StatCard'; + + export default async function DashboardPage() { + const session = await getServerSession(); + const stats = await db.query.stats.findMany(); + return ; + } +metadata: + provenance: + kind: synthetic + origin: Next.js Server and Client Components documentation + revision_reason: Narrowed the label to the direct client-boundary conflict; no browser-leak claim is made. + difficulty: HAS-BUGS + domains: [server-components, boundaries] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/building-your-application/rendering/client-components + supports: The use client directive marks a Client Component entry point. + notes: The directive and server-only assumption make the incompatibility explicit. + excluded_claims: [Bundle contents and the placement of child component directives are not scored.] issues: - - severity: MAJOR - category: rsc-boundary - description: "'use client' on the page prevents server-side data fetching; getServerSession and db.query will fail or be called client-side, leaking db access to the browser bundle" - evidence: "'use client'; ... const session = await getServerSession();" - expected_checks: [rsc-boundary, server-actions-safety] - - severity: MAJOR + - id: next-01-client-server-conflict + rule_id: next.rsc-boundary category: rsc-boundary - description: "StatCard is a pure display component forced into the client bundle unnecessarily; only RevenueChart needs 'use client'" - evidence: "import StatCard from './StatCard'; // pure display" - expected_checks: [rsc-boundary, runtime-choice] + severity: MAJOR + description: The use client entry point conflicts with the stated server-only session and database work in the same module. + evidence_spans: + - lines: [1, 3, 4, 9, 10] + rationale: The module is marked as a Client Component while importing and awaiting server-only services. diff --git a/research/benchmarks/fixtures/next/fixture-02.yaml b/research/benchmarks/fixtures/next/fixture-02.yaml index 8433797..5b54895 100644 --- a/research/benchmarks/fixtures/next/fixture-02.yaml +++ b/research/benchmarks/fixtures/next/fixture-02.yaml @@ -1,24 +1,41 @@ +schema_version: 2 id: next-02 category: cache-correctness -description: overly aggressive no-store on every fetch disabling PPR -code: | - // app/products/page.tsx - export default async function ProductsPage() { - const [products, categories, featuredBanner] = await Promise.all([ - fetch('/api/products', { cache: 'no-store' }), - fetch('/api/categories', { cache: 'no-store' }), - fetch('/api/banner/featured', { cache: 'no-store' }), - ]); - // ...render - } -issues: - - severity: MAJOR - category: cache-correctness - description: "cache: 'no-store' on all fetches disables PPR; categories and banner are stable data and should use revalidate or force-cache; every page visit re-fetches all three even if data hasn't changed" - evidence: "fetch('/api/categories', { cache: 'no-store' })" - expected_checks: [cache-correctness, ppr-correctness] - - severity: MINOR - category: cache-correctness - description: "no granular revalidate tags; cache invalidation requires a full page re-build rather than targeted revalidatePath/revalidateTag" - evidence: "fetch('/api/products', { cache: 'no-store' })" - expected_checks: [cache-correctness] +description: Product data is intentionally uncached because it must reflect immediate writes. +intentional_clean: true +public: + file_path: app/products/page.tsx + language: TypeScript JSX + frameworks: + next: '15' + react: '19' + runtime: nodejs + assumptions: + - Inventory and prices can change immediately after a completed purchase. + - This route is the canonical confirmation view following a purchase. + requirements: + - The confirmation view must not serve a cached inventory or price response. + code: | + export default async function ProductsPage() { + const response = await fetch('https://catalog.example.test/products', { cache: 'no-store' }); + const products = await response.json(); + return ; + } +metadata: + provenance: + kind: synthetic + origin: Next.js data fetching documentation + revision_reason: Replaced a generic no-store performance preference with a clean freshness control. + difficulty: ADVERSARIAL + domains: [caching, freshness] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/building-your-application/data-fetching/fetching + supports: Next.js supports cache configuration for fetch requests. + notes: The stated requirement supports no-store for this route. + excluded_claims: [PPR, tags, and broader catalog caching policy are not scored.] +issues: [] diff --git a/research/benchmarks/fixtures/next/fixture-03.yaml b/research/benchmarks/fixtures/next/fixture-03.yaml index 2e085ee..08dd549 100644 --- a/research/benchmarks/fixtures/next/fixture-03.yaml +++ b/research/benchmarks/fixtures/next/fixture-03.yaml @@ -1,33 +1,63 @@ +schema_version: 2 id: next-03 category: server-actions-safety -description: server action with no input validation or CSRF protection -code: | - // app/actions/updateProfile.ts - 'use server'; +description: A Server Action accepts identity and privilege from form data. +intentional_clean: false +public: + file_path: app/actions/updateProfile.ts + language: TypeScript + frameworks: + next: '15' + runtime: nodejs + assumptions: + - The authenticated session identifies the only profile the caller may update. + - Only an administrator may change a user's role. + requirements: + - The action must derive the target profile and authorization from the authenticated session. + code: | + 'use server'; - export async function updateProfile(formData: FormData) { - const userId = formData.get('userId') as string; - const bio = formData.get('bio') as string; - const role = formData.get('role') as string; // should never come from user + export async function updateProfile(formData: FormData) { + const userId = formData.get('userId') as string; + const bio = formData.get('bio') as string; + const role = formData.get('role') as string; - await db.user.update({ - where: { id: userId }, - data: { bio, role }, // role written directly from form input - }); - } + await db.user.update({ + where: { id: userId }, + data: { bio, role }, + }); + } +metadata: + provenance: + kind: synthetic + origin: Next.js Server Actions documentation + revision_reason: Limited the fixture to explicit authorization authority; removed CSRF and stored-XSS speculation. + difficulty: FUNDAMENTALLY-FLAWED + domains: [authorization, server-actions] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions-and-mutations#authentication-and-authorization + supports: Server Actions should verify authentication and authorization. + notes: Both authority violations are established by the supplied session requirements. + excluded_claims: [CSRF, validation-library choice, and HTML rendering behavior are not scored.] issues: - - severity: CRITICAL + - id: next-03-form-target-identity + rule_id: next.server-actions-safety category: server-actions-safety - description: "role is written directly from form input with no authorization check; any user can elevate their own role to admin" - evidence: "const role = formData.get('role') as string; ... data: { bio, role }" - expected_checks: [server-actions-safety, security] - - severity: CRITICAL + severity: CRITICAL + description: The update target is taken from user-controlled form data instead of the authenticated session. + evidence_spans: + - lines: [4, 9] + rationale: userId comes from formData and becomes the database where clause. + - id: next-03-form-role + rule_id: next.server-actions-safety category: server-actions-safety - description: "no session/auth check: any unauthenticated caller can update any user's profile by supplying an arbitrary userId" - evidence: "const userId = formData.get('userId') as string;" - expected_checks: [server-actions-safety, security] - - severity: MAJOR - category: server-actions-safety - description: "no input validation or sanitization on bio; stored XSS if bio is rendered with dangerouslySetInnerHTML elsewhere" - evidence: "const bio = formData.get('bio') as string;" - expected_checks: [server-actions-safety, security] + severity: CRITICAL + description: The privileged role field is taken from user-controlled form data without the required administrator authorization. + evidence_spans: + - lines: [6, 10] + rationale: role comes from formData and is written directly to the database. diff --git a/research/benchmarks/fixtures/next/fixture-04.yaml b/research/benchmarks/fixtures/next/fixture-04.yaml index c2ad994..c0dccea 100644 --- a/research/benchmarks/fixtures/next/fixture-04.yaml +++ b/research/benchmarks/fixtures/next/fixture-04.yaml @@ -1,32 +1,54 @@ +schema_version: 2 id: next-04 category: route-handler-safety -description: route handler leaking internal error details and missing rate limiting -code: | - // app/api/auth/reset-password/route.ts - export async function POST(request: Request) { - const { email } = await request.json(); - try { - const user = await db.user.findUnique({ where: { email } }); - if (!user) throw new Error(`No user found with email: ${email}`); - await sendResetEmail(user); - return Response.json({ success: true }); - } catch (e) { - return Response.json({ error: e.message }, { status: 500 }); +description: A password-reset route returns a request-derived account-existence error. +intentional_clean: false +public: + file_path: app/api/auth/reset-password/route.ts + language: TypeScript + frameworks: + next: '15' + runtime: nodejs + assumptions: + - The route accepts unauthenticated password-reset requests. + - Email addresses are personal account information. + requirements: + - The route must not reveal whether the supplied email has an account. + code: | + export async function POST(request: Request) { + const { email } = await request.json(); + try { + const user = await db.user.findUnique({ where: { email } }); + if (!user) throw new Error(`No user found with email: ${email}`); + await sendResetEmail(user); + return Response.json({ success: true }); + } catch (error) { + return Response.json({ error: error.message }, { status: 500 }); + } } - } +metadata: + provenance: + kind: synthetic + origin: Next.js Route Handlers documentation + revision_reason: Removed the unsupported rate-limit claim and retained the direct response disclosure. + difficulty: HAS-BUGS + domains: [privacy, route-handlers] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/building-your-application/routing/route-handlers + supports: Route Handlers construct HTTP responses. + notes: The request input is interpolated into the caught error and returned in the response. + excluded_claims: [Rate limiting, email delivery policy, and stack-trace exposure are not scored.] issues: - - severity: CRITICAL + - id: next-04-account-existence-disclosure + rule_id: next.route-handler-safety category: route-handler-safety - description: "error message echoes back whether a user email exists; enables user enumeration: attackers can discover registered emails by probing error responses" - evidence: "throw new Error(`No user found with email: ${email}`);" - expected_checks: [route-handler-safety, security] - - severity: MAJOR - category: route-handler-safety - description: "no rate limiting: endpoint accepts unlimited reset requests, enabling email spam and account lockout attacks" - evidence: "export async function POST(request: Request) {" - expected_checks: [route-handler-safety, security] - - severity: MAJOR - category: route-handler-safety - description: "e.message returned verbatim to client may expose internal stack traces or database error details" - evidence: "return Response.json({ error: e.message }, { status: 500 });" - expected_checks: [route-handler-safety, security] + severity: MAJOR + description: The response returns an error stating that the request-supplied email has no account, contrary to the non-disclosure requirement. + evidence_spans: + - lines: [5, 9] + rationale: The email is interpolated into an error that is returned to the caller. diff --git a/research/benchmarks/fixtures/next/fixture-05.yaml b/research/benchmarks/fixtures/next/fixture-05.yaml index abffcc8..4aee1d9 100644 --- a/research/benchmarks/fixtures/next/fixture-05.yaml +++ b/research/benchmarks/fixtures/next/fixture-05.yaml @@ -1,26 +1,51 @@ +schema_version: 2 id: next-05 category: runtime-choice -description: edge runtime using Node.js-only APIs -code: | - // app/api/pdf/route.ts - export const runtime = 'edge'; +description: An Edge route uses Node.js filesystem APIs. +intentional_clean: false +public: + file_path: app/api/pdf/route.ts + language: TypeScript + frameworks: + next: '15' + runtime: edge + assumptions: + - The route must run in the configured Edge runtime. + requirements: + - The route must not use Node.js-only filesystem APIs. + code: | + export const runtime = 'edge'; - import { readFileSync } from 'fs'; // Node.js only - import path from 'path'; + import { readFileSync } from 'fs'; + import path from 'path'; - export async function GET() { - const template = readFileSync(path.join(process.cwd(), 'templates/invoice.pdf')); - const pdf = await generatePDF(template); - return new Response(pdf, { headers: { 'Content-Type': 'application/pdf' } }); - } + export async function GET() { + const template = readFileSync(path.join(process.cwd(), 'templates/invoice.pdf')); + return new Response(template, { headers: { 'Content-Type': 'application/pdf' } }); + } +metadata: + provenance: + kind: synthetic + origin: Next.js Edge runtime documentation + revision_reason: Merged duplicate filesystem and process claims into one runtime incompatibility. + difficulty: HAS-BUGS + domains: [runtime, deployment] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/pages/api-reference/edge + supports: The Edge runtime does not support all Node.js APIs. + notes: The declared runtime and filesystem imports are both shown in the source. + excluded_claims: [A replacement PDF implementation is not scored.] issues: - - severity: CRITICAL + - id: next-05-edge-filesystem-api + rule_id: next.runtime-choice category: runtime-choice - description: "runtime='edge' is incompatible with fs.readFileSync and path.join; these are Node.js APIs unavailable in the Edge Runtime; route will fail at deploy time" - evidence: "export const runtime = 'edge'; import { readFileSync } from 'fs';" - expected_checks: [runtime-choice] - - severity: MINOR - category: runtime-choice - description: "process.cwd() is also unavailable in Edge Runtime; file access patterns require Node.js runtime or static imports" - evidence: "path.join(process.cwd(), 'templates/invoice.pdf')" - expected_checks: [runtime-choice] + severity: CRITICAL + description: The explicitly Edge-configured route imports and calls Node.js filesystem APIs. + evidence_spans: + - lines: [1, 3, 4, 7] + rationale: The route declares edge while importing fs and path and calling readFileSync. diff --git a/research/benchmarks/fixtures/next/fixture-06.yaml b/research/benchmarks/fixtures/next/fixture-06.yaml index 0360e90..c9874ab 100644 --- a/research/benchmarks/fixtures/next/fixture-06.yaml +++ b/research/benchmarks/fixtures/next/fixture-06.yaml @@ -1,31 +1,57 @@ +schema_version: 2 id: next-06 category: suspense-boundary -description: missing Suspense boundary around async RSC in streaming layout -code: | - // app/feed/page.tsx - export default async function FeedPage() { - return ( -
-
- {/* async RSC, no Suspense */} - {/* async RSC, no Suspense */} -
-
- ); - } +description: A feed route must stream its shell before a slow Server Component resolves. +intentional_clean: false +public: + file_path: app/feed/page.tsx + language: TypeScript JSX + frameworks: + next: '15' + react: '19' + runtime: nodejs + assumptions: + - AsyncFeedItems awaits a slow database query. + requirements: + - Header and footer must stream before the feed items resolve. + code: | + export default function FeedPage() { + return ( +
+
+ +
+
+ ); + } - async function AsyncFeedItems() { - const items = await db.feed.findMany({ take: 50 }); // slow query - return items.map(i => ); - } + async function AsyncFeedItems() { + const items = await db.feed.findMany({ take: 50 }); + return items.map((item) => ); + } +metadata: + provenance: + kind: synthetic + origin: Next.js loading UI and streaming documentation + revision_reason: Added an explicit streaming requirement and removed unsupported cache advice. + difficulty: HAS-BUGS + domains: [streaming, server-components] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/building-your-application/routing/loading-ui-and-streaming + supports: Suspense boundaries support streaming UI. + notes: The slow await is shown and the streaming ordering is a public requirement. + excluded_claims: [Caching and the performance of a particular query are not scored.] issues: - - severity: MAJOR + - id: next-06-missing-streaming-boundary + rule_id: next.suspense-boundary category: suspense-boundary - description: "no Suspense boundary around AsyncFeedItems; the slow db query blocks streaming of the entire page including Header and Footer" - evidence: " {/* async RSC, no Suspense */}" - expected_checks: [suspense-boundary, cache-correctness] - - severity: MINOR - category: cache-correctness - description: "db.feed.findMany with no cache directive; each request re-queries even for the same user; revalidate or React cache() should be applied" - evidence: "const items = await db.feed.findMany({ take: 50 });" - expected_checks: [cache-correctness] + severity: MAJOR + description: AsyncFeedItems has no Suspense boundary even though the route must stream its shell before the slow component resolves. + evidence_spans: + - lines: [5, 11] + rationale: The awaited component is rendered directly between Header and Footer. diff --git a/research/benchmarks/fixtures/next/fixture-07.yaml b/research/benchmarks/fixtures/next/fixture-07.yaml index 6b277c3..d74333f 100644 --- a/research/benchmarks/fixtures/next/fixture-07.yaml +++ b/research/benchmarks/fixtures/next/fixture-07.yaml @@ -1,34 +1,63 @@ +schema_version: 2 id: next-07 category: upgrade-safety -description: Pages Router patterns used in App Router without migration -code: | - // app/shop/[id]/page.tsx (App Router file) - import { useRouter } from 'next/router'; // Pages Router import +description: An App Router page uses Pages Router APIs. +intentional_clean: false +public: + file_path: app/shop/[id]/page.tsx + language: TypeScript JSX + frameworks: + next: '15' + react: '19' + runtime: nodejs + assumptions: + - The file is an App Router page. + requirements: + - The route must use App Router-compatible routing and data APIs. + code: | + 'use client'; - export default function ShopPage({ params }) { - const router = useRouter(); + import { useRouter } from 'next/router'; - const handleBack = () => router.back(); + export default function ShopPage({ params }) { + const router = useRouter(); + return ; + } - return ( -
- - -
- ); - } - - export async function getServerSideProps(ctx) { // Pages Router API - return { props: { id: ctx.params.id } }; - } + export async function getServerSideProps(context) { + return { props: { id: context.params.id } }; + } +metadata: + provenance: + kind: synthetic + origin: Next.js App Router migration documentation + revision_reason: Retained only the explicit Pages Router API incompatibilities. + difficulty: FUNDAMENTALLY-FLAWED + domains: [routing, upgrade] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/guides/migrating/app-router-migration + supports: App Router migration replaces next/router and getServerSideProps patterns. + notes: The routing mode is a supplied assumption, not inferred from the filename. + excluded_claims: [The fixture does not score component design or params typing.] issues: - - severity: CRITICAL + - id: next-07-pages-router-hook + rule_id: next.upgrade-safety category: upgrade-safety - description: "next/router is the Pages Router hook; App Router requires next/navigation (useRouter from next/navigation); will throw at runtime" - evidence: "import { useRouter } from 'next/router';" - expected_checks: [upgrade-safety, app-router-conventions] - - severity: CRITICAL + severity: MAJOR + description: The App Router page imports useRouter from next/router instead of the App Router API. + evidence_spans: + - lines: [3, 6] + rationale: The Pages Router hook import is invoked by the App Router page. + - id: next-07-get-server-side-props + rule_id: next.upgrade-safety category: upgrade-safety - description: "getServerSideProps is a Pages Router API; App Router ignores it silently; params are already available via the page's props in App Router" - evidence: "export async function getServerSideProps(ctx) {" - expected_checks: [upgrade-safety, app-router-conventions] + severity: MAJOR + description: The App Router page exports the Pages Router getServerSideProps API. + evidence_spans: + - lines: [10] + rationale: getServerSideProps is declared directly in the App Router module. diff --git a/research/benchmarks/fixtures/next/fixture-08.yaml b/research/benchmarks/fixtures/next/fixture-08.yaml index cb73849..6935401 100644 --- a/research/benchmarks/fixtures/next/fixture-08.yaml +++ b/research/benchmarks/fixtures/next/fixture-08.yaml @@ -1,34 +1,47 @@ +schema_version: 2 id: next-08 category: app-router-conventions -description: incorrect metadata API and missing generateStaticParams safety bounds -code: | - // app/blog/[slug]/page.tsx - export const metadata = { // static, not dynamic - title: 'Blog Post', - }; - - export async function generateStaticParams() { - const posts = await db.post.findMany(); // no limit - return posts.map(p => ({ slug: p.slug })); - } - - export default async function BlogPost({ params }) { - const post = await db.post.findUnique({ where: { slug: params.slug } }); - return
{post.content}
; // no null check - } +description: A blog route does not meet its required not-found behavior. +intentional_clean: false +public: + file_path: app/blog/[slug]/page.tsx + language: TypeScript JSX + frameworks: + next: '15' + react: '19' + runtime: nodejs + assumptions: + - db.post.findUnique returns null when no post has the requested slug. + requirements: + - An unknown slug must produce the route's not-found response. + code: | + export default async function BlogPost({ params }) { + const post = await db.post.findUnique({ where: { slug: params.slug } }); + return
{post.content}
; + } +metadata: + provenance: + kind: synthetic + origin: Next.js notFound documentation + revision_reason: Removed valid static metadata and unbounded-query preferences; retained the stated route behavior. + difficulty: HAS-BUGS + domains: [routing, operability] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://nextjs.org/docs/app/api-reference/functions/not-found + supports: notFound renders the not-found segment and returns a 404 response. + notes: The null result and required not-found behavior are public assumptions. + excluded_claims: [Dynamic metadata and static-parameter sizing are not scored.] issues: - - severity: MAJOR + - id: next-08-missing-not-found + rule_id: next.app-router-conventions category: app-router-conventions - description: "metadata exported as a plain object instead of generateMetadata(); post-specific title/og tags (title, description, og:image) cannot be set dynamically" - evidence: "export const metadata = { title: 'Blog Post' };" - expected_checks: [app-router-conventions] - - severity: MAJOR - category: app-router-conventions - description: "generateStaticParams fetches all posts with no limit; at scale this causes build-time OOM and multi-hour static generation" - evidence: "const posts = await db.post.findMany(); // no limit" - expected_checks: [app-router-conventions, operability] - - severity: MINOR - category: operability - description: "no null check on post before rendering; if slug doesn't exist, post.content throws TypeError; should call notFound()" - evidence: "return
{post.content}
; // no null check" - expected_checks: [operability] + severity: MAJOR + description: The route dereferences a possibly null post instead of producing the required not-found response. + evidence_spans: + - lines: [2, 3] + rationale: findUnique can return null and post.content is rendered without a not-found branch. diff --git a/research/benchmarks/fixtures/react-native/fixture-01.yaml b/research/benchmarks/fixtures/react-native/fixture-01.yaml index d208aba..930761e 100644 --- a/research/benchmarks/fixtures/react-native/fixture-01.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-01.yaml @@ -1,36 +1,44 @@ +schema_version: 2 id: rn-01 category: render-performance -description: inline style objects and anonymous functions recreated on every render -code: | - function ProductCard({ product, onAddToCart }) { - return ( - - - onAddToCart(product.id)} - style={{ backgroundColor: '#007AFF', padding: 12 }} - > - Add to Cart - - - ); - } -issues: - - severity: MAJOR - category: render-performance - description: "inline style objects create new object references on every render; FlatList/memo optimization is fully defeated because style props never shallowEqual" - evidence: "style={{ flex: 1, padding: 16, backgroundColor: '#fff' }}" - expected_checks: [render-performance, native-styling] - - severity: MAJOR - category: render-performance - description: "anonymous arrow function in onPress recreates on every render; if ProductCard is used in a FlatList, every item re-renders on any parent state change" - evidence: "onPress={() => onAddToCart(product.id)}" - expected_checks: [render-performance] - - severity: MINOR - category: render-performance - description: "Image has no resizeMode; default 'cover' may not match container; no loading/error placeholder" - evidence: "style={{ width: 200, height: 200 }}" - expected_checks: [native-styling] +description: A simple product card uses ordinary inline values without a stated performance defect. +intentional_clean: true +public: + file_path: src/ProductCard.tsx + language: TypeScript JSX + frameworks: + react-native: '0.76' + runtime: hermes + assumptions: + - ProductCard is not rendered by a virtualized list. + - Its parent does not rely on referential equality for its style or handler props. + requirements: [] + code: | + function ProductCard({ product, onAddToCart }) { + return ( + + + onAddToCart(product.id)}> + Add to Cart + + + ); + } +metadata: + provenance: + kind: synthetic + origin: React Native performance documentation + revision_reason: Replaced generic inline-object and callback claims with a clean restraint control. + difficulty: ADVERSARIAL + domains: [rendering, performance] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://reactnative.dev/docs/performance + supports: React Native performance work is context-dependent. + notes: No catalog rule is established by the shown ordinary component. + excluded_claims: [Memoization, StyleSheet extraction, image placeholders, and FlatList behavior are not scored.] +issues: [] diff --git a/research/benchmarks/fixtures/react-native/fixture-02.yaml b/research/benchmarks/fixtures/react-native/fixture-02.yaml index 71ece7c..0cd1caf 100644 --- a/research/benchmarks/fixtures/react-native/fixture-02.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-02.yaml @@ -1,30 +1,45 @@ +schema_version: 2 id: rn-02 category: list-performance -description: FlatList without keyExtractor, removeClippedSubviews, or windowing config -code: | - function NotificationList({ notifications }) { - return ( - } - onEndReached={loadMore} - onEndReachedThreshold={0.5} - /> - ); - } -issues: - - severity: MAJOR - category: list-performance - description: "no keyExtractor: FlatList falls back to index-based keys causing broken diffing on list mutations (insert/delete reorders)" - evidence: "renderItem={({ item }) => }" - expected_checks: [list-performance, render-performance] - - severity: MAJOR - category: list-performance - description: "no removeClippedSubviews or windowSize config; on large lists all items stay mounted, causing memory pressure on low-end Android devices" - evidence: " item.id} + renderItem={({ item }) => } + onEndReached={loadMore} + onEndReachedThreshold={0.5} + /> + ); + } +metadata: + provenance: + kind: synthetic + origin: React Native FlatList documentation + revision_reason: Replaced unsupported tuning-prop defaults with a clean list-identity control. + difficulty: ADVERSARIAL + domains: [lists, performance] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://reactnative.dev/docs/flatlist + supports: FlatList accepts a keyExtractor and is a performant interface for flat lists. + notes: The stable key requirement is met and no tuning requirement is supplied. + excluded_claims: [removeClippedSubviews, windowSize, and getItemLayout are not scored.] +issues: [] diff --git a/research/benchmarks/fixtures/react-native/fixture-03.yaml b/research/benchmarks/fixtures/react-native/fixture-03.yaml index f11fab6..9c4a03d 100644 --- a/research/benchmarks/fixtures/react-native/fixture-03.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-03.yaml @@ -1,32 +1,66 @@ +schema_version: 2 id: rn-03 category: memory-leak -description: setInterval without cleanup and subscription without teardown -code: | - function LivePriceTicker({ symbol }) { - const [price, setPrice] = useState(null); +description: A ticker effect creates resources without cleanup. +intentional_clean: false +public: + file_path: src/LivePriceTicker.tsx + language: TypeScript JSX + frameworks: + react-native: '0.76' + runtime: hermes + assumptions: + - priceSocket.subscribe returns a subscription that requires unsubscribe. + - The component can unmount or receive a new symbol. + requirements: + - Effects must release timers and subscriptions that they create. + code: | + function LivePriceTicker({ symbol }) { + const [price, setPrice] = useState(null); - useEffect(() => { - const id = setInterval(() => { - fetchPrice(symbol).then(setPrice); - }, 1000); - // no cleanup return - }, [symbol]); + useEffect(() => { + const timer = setInterval(() => { + fetchPrice(symbol).then(setPrice); + }, 1000); + }, [symbol]); - useEffect(() => { - const sub = priceSocket.subscribe(symbol, setPrice); - // no unsubscribe - }, [symbol]); + useEffect(() => { + const subscription = priceSocket.subscribe(symbol, setPrice); + }, [symbol]); - return {price}; - } + return {price}; + } +metadata: + provenance: + kind: synthetic + origin: React effect cleanup documentation + revision_reason: Replaced answer comments with ownership assumptions and specific cleanup labels. + difficulty: FUNDAMENTALLY-FLAWED + domains: [effects, resource-lifecycle] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://react.dev/reference/react/useEffect + supports: Effects may return cleanup functions for external systems. + notes: Both external resources are created by their effects and neither effect returns cleanup. + excluded_claims: [Exact memory growth and unmounted-state warnings are not scored.] issues: - - severity: CRITICAL + - id: rn-03-interval-cleanup + rule_id: react-native.memory-leak category: memory-leak - description: "setInterval not cleared on unmount or symbol change; interval fires indefinitely after navigation away, calling setState on unmounted component" - evidence: "const id = setInterval(() => { ... // no cleanup return" - expected_checks: [memory-leak, hook-correctness] - - severity: CRITICAL + severity: MAJOR + description: The effect creates a repeating timer but returns no cleanup to clear it. + evidence_spans: + - lines: [5, 6, 7] + rationale: setInterval is called inside an effect whose body has no returned cleanup. + - id: rn-03-subscription-cleanup + rule_id: react-native.memory-leak category: memory-leak - description: "priceSocket subscription not unsubscribed on unmount; multiple subscriptions stack on each symbol change" - evidence: "const sub = priceSocket.subscribe(symbol, setPrice); // no unsubscribe" - expected_checks: [memory-leak, hook-correctness] + severity: MAJOR + description: The effect creates a socket subscription but returns no cleanup to unsubscribe it. + evidence_spans: + - lines: [10, 11] + rationale: subscribe is called inside an effect whose body has no returned cleanup. diff --git a/research/benchmarks/fixtures/react-native/fixture-04.yaml b/research/benchmarks/fixtures/react-native/fixture-04.yaml index dd4d8be..4a35b51 100644 --- a/research/benchmarks/fixtures/react-native/fixture-04.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-04.yaml @@ -1,31 +1,67 @@ +schema_version: 2 id: rn-04 category: native-bridge -description: synchronous native module call blocking JS thread -code: | - function useDeviceInfo() { - const [info, setInfo] = useState({}); +description: A synchronous optional native module call runs on startup without a guard. +intentional_clean: false +public: + file_path: src/useDeviceInfo.ts + language: TypeScript JSX + frameworks: + react-native: '0.76' + runtime: hermes + assumptions: + - DeviceInfo.getInfo is a synchronous native method and may block while it gathers device data. + - DeviceInfo is optional and is unavailable in the project's web preview target. + requirements: + - The startup header must not synchronously block on optional native work. + - The web preview must show a fallback when DeviceInfo is unavailable. + code: | + function useDeviceInfo() { + const [info, setInfo] = useState({}); - useEffect(() => { - // NativeModules.DeviceInfo.getInfo() is synchronous - const deviceInfo = NativeModules.DeviceInfo.getInfo(); - setInfo(deviceInfo); - }, []); + useEffect(() => { + const deviceInfo = NativeModules.DeviceInfo.getInfo(); + setInfo(deviceInfo); + }, []); - return info; - } + return info; + } - function AppHeader() { - const info = useDeviceInfo(); - return {info.model}; - } + function AppHeader() { + const info = useDeviceInfo(); + return {info.model}; + } +metadata: + provenance: + kind: synthetic + origin: React Native NativeModules documentation + revision_reason: Made synchrony and availability public; split bridge performance from bridge failure handling. + difficulty: HAS-BUGS + domains: [native-modules, startup] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://reactnative.dev/docs/next/legacy/native-modules-ios + supports: Native modules expose native functionality to JavaScript. + notes: The source does not infer the module's behavior; the public assumptions define the benchmark conditions. + excluded_claims: [ANR, a specific fallback implementation, and native-module architecture changes are not scored.] issues: - - severity: MAJOR + - id: rn-04-blocking-device-info + rule_id: react-native.native-bridge category: native-bridge - description: "NativeModules synchronous call blocks the JS thread; on startup this can cause frame drops and ANR on Android if the native implementation is slow" - evidence: "const deviceInfo = NativeModules.DeviceInfo.getInfo();" - expected_checks: [native-bridge, render-performance] - - severity: MINOR - category: native-bridge - description: "no error handling around NativeModules call; if the native module is unavailable (web target, storybook), throws uncaught TypeError" - evidence: "NativeModules.DeviceInfo.getInfo();" - expected_checks: [native-bridge, operability] + severity: MAJOR + description: The startup effect synchronously calls the known blocking DeviceInfo method, contrary to the startup requirement. + evidence_spans: + - lines: [5] + rationale: The synchronous native call is executed directly by the startup effect. + - id: rn-04-unguarded-device-info + rule_id: react-native.native-bridge.error-handling + category: native-bridge.error-handling + severity: MAJOR + description: The optional DeviceInfo call is unguarded even though the web preview target can lack that module. + evidence_spans: + - lines: [5] + rationale: The optional native module is accessed directly with no availability branch or fallback. diff --git a/research/benchmarks/fixtures/react-native/fixture-05.yaml b/research/benchmarks/fixtures/react-native/fixture-05.yaml index 6a02aab..48c83aa 100644 --- a/research/benchmarks/fixtures/react-native/fixture-05.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-05.yaml @@ -1,29 +1,56 @@ +schema_version: 2 id: rn-05 category: expo-workflow -description: bare-workflow native module used in managed Expo project -code: | - // app.json shows "expo": { "workflow": "managed" } - import { NativeModules } from 'react-native'; - import RNBluetoothClassic from 'react-native-bluetooth-classic'; // native module, no Expo config plugin +description: An Expo Go target imports a module requiring custom native code. +intentional_clean: false +public: + file_path: src/useBluetooth.ts + language: TypeScript JSX + frameworks: + expo: '52' + react-native: '0.76' + runtime: hermes + assumptions: + - The application is launched in Expo Go, not a development build or custom native client. + - RNBluetoothClassic requires custom native code that Expo Go does not include. + requirements: + - The Bluetooth feature must run in the stated Expo Go deployment target. + code: | + import RNBluetoothClassic from 'react-native-bluetooth-classic'; - export function useBluetooth() { - const [devices, setDevices] = useState([]); + export function useBluetooth() { + const [devices, setDevices] = useState([]); - useEffect(() => { - RNBluetoothClassic.startDiscovery().then(setDevices); - return () => RNBluetoothClassic.cancelDiscovery(); - }, []); + useEffect(() => { + RNBluetoothClassic.startDiscovery().then(setDevices); + return () => RNBluetoothClassic.cancelDiscovery(); + }, []); - return devices; - } + return devices; + } +metadata: + provenance: + kind: synthetic + origin: Expo Go and development-build documentation + revision_reason: Replaced unverified config-plugin and permission claims with explicit Expo Go native-code context. + difficulty: HAS-BUGS + domains: [expo, native-modules] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://docs.expo.dev/develop/development-builds/introduction/ + supports: Development builds allow custom native code not available in Expo Go. + notes: The requirement and assumptions make the deployment incompatibility explicit. + excluded_claims: [Config-plugin availability, permission declarations, EAS Build behavior, and store review are not scored.] issues: - - severity: CRITICAL + - id: rn-05-expo-go-native-module + rule_id: react-native.expo-workflow category: expo-workflow - description: "react-native-bluetooth-classic has no Expo config plugin; managed workflow EAS Build will fail because the native module requires bare workflow or a custom plugin to modify native files" - evidence: "import RNBluetoothClassic from 'react-native-bluetooth-classic'; // native module, no Expo config plugin" - expected_checks: [expo-workflow, upgrade-safety] - - severity: MAJOR - category: expo-workflow - description: "no Bluetooth permission declaration in app.json (ios.infoPlist, android.permissions); app will crash or be rejected at store review" - evidence: "RNBluetoothClassic.startDiscovery().then(setDevices);" - expected_checks: [expo-workflow, security] + severity: CRITICAL + description: The stated Expo Go target cannot load the imported module because it requires custom native code absent from Expo Go. + evidence_spans: + - lines: [1, 7] + rationale: The custom native module is imported and called by the required feature. diff --git a/research/benchmarks/fixtures/react-native/fixture-06.yaml b/research/benchmarks/fixtures/react-native/fixture-06.yaml index 92a8864..de13c05 100644 --- a/research/benchmarks/fixtures/react-native/fixture-06.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-06.yaml @@ -1,30 +1,57 @@ +schema_version: 2 id: rn-06 category: gesture-handling -description: Android back gesture not handled in custom navigation flow -code: | - function CheckoutFlow({ onComplete, onAbort }) { - const [step, setStep] = useState(1); +description: A custom checkout flow has no Android hardware-back handling. +intentional_clean: false +public: + file_path: src/CheckoutFlow.tsx + language: TypeScript JSX + frameworks: + react-native: '0.76' + runtime: hermes + assumptions: + - CheckoutFlow owns the custom multi-step screen state. + - Android hardware back reaches this component before navigation exits the screen. + requirements: + - Android hardware back at steps two and three must return to the preceding checkout step. + code: | + function CheckoutFlow({ onComplete }) { + const [step, setStep] = useState(1); + const nextStep = () => setStep((current) => Math.min(current + 1, 3)); + const prevStep = () => setStep((current) => Math.max(current - 1, 1)); - const nextStep = () => setStep(s => Math.min(s + 1, 3)); - const prevStep = () => setStep(s => Math.max(s - 1, 1)); - - return ( - - - {step === 1 && } - {step === 2 && } - {step === 3 && } - - ); - } + return ( + + + {step === 1 && } + {step === 2 && } + {step === 3 && } + + ); + } +metadata: + provenance: + kind: synthetic + origin: React Native BackHandler documentation + revision_reason: Added explicit navigation ownership and required Android behavior. + difficulty: HAS-BUGS + domains: [navigation, gestures] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://reactnative.dev/docs/backhandler + supports: BackHandler can handle Android hardware back presses. + notes: The source supplies only an in-flow back button and no Android hardware-back handling. + excluded_claims: [Confirmation dialogs, iOS gestures, and global navigator behavior are not scored.] issues: - - severity: MAJOR + - id: rn-06-missing-hardware-back + rule_id: react-native.gesture-handling category: gesture-handling - description: "Android hardware back button is not handled; pressing back during checkout will pop the entire checkout screen instead of going to the previous step, losing form data" - evidence: "function CheckoutFlow({ onComplete, onAbort }) {" - expected_checks: [gesture-handling, operability] - - severity: MINOR - category: operability - description: "no confirmation dialog on abort; user can accidentally discard cart state by pressing back" - evidence: "const prevStep = () => setStep(s => Math.max(s - 1, 1));" - expected_checks: [operability] + severity: MAJOR + description: The custom flow has no handling for the required Android hardware-back transition to the previous step. + evidence_spans: + - lines: [1, 4, 10, 11] + rationale: prevStep is only wired to PaymentForm and no hardware-back handler is shown. diff --git a/research/benchmarks/fixtures/react-native/fixture-07.yaml b/research/benchmarks/fixtures/react-native/fixture-07.yaml index ed9214d..ce4b691 100644 --- a/research/benchmarks/fixtures/react-native/fixture-07.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-07.yaml @@ -1,27 +1,50 @@ +schema_version: 2 id: rn-07 category: hermes-compatibility -description: optional chaining and nullish coalescing used without Hermes version guard -code: | - // package.json: "react-native": "0.66.0", "hermes-engine": "0.9.0" - function UserAvatar({ user }) { - const displayName = user?.profile?.displayName ?? user?.email ?? 'Guest'; - const avatarUri = user?.avatar?.['200x200']?.url; +description: Modern optional chaining is used with a current Hermes runtime. +intentional_clean: true +public: + file_path: src/UserAvatar.tsx + language: TypeScript JSX + frameworks: + react-native: '0.66' + hermes: '0.9.0' + runtime: hermes + assumptions: + - React Native 0.66 ships with Hermes 0.9.0. + - user may be null. + requirements: [] + code: | + function UserAvatar({ user }) { + const displayName = user?.profile?.displayName ?? user?.email ?? 'Guest'; + const avatarUri = user?.avatar?.['200x200']?.url; - return ( - - - {displayName} - - ); - } -issues: - - severity: MAJOR - category: hermes-compatibility - description: "Hermes 0.9.0 (bundled with RN 0.66) has partial optional chaining support; chained bracket notation user?.avatar?.['200x200']?.url may throw on Android Hermes but pass on iOS JSC" - evidence: "const avatarUri = user?.avatar?.['200x200']?.url;" - expected_checks: [hermes-compatibility, upgrade-safety] - - severity: MINOR - category: operability - description: "Image with undefined uri source renders broken image icon instead of placeholder; no fallback source or error handler" - evidence: "" - expected_checks: [operability] + return ( + + + {displayName} + + ); + } +metadata: + provenance: + kind: synthetic + origin: Hermes documentation + revision_reason: Replaced the unsupported old-version compatibility assertion with a clean current-runtime control. + difficulty: ADVERSARIAL + domains: [runtime, language] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://reactnative.dev/blog/2021/10/01/version-066 + supports: React Native 0.66 includes Hermes 0.9.0. + - url: https://raw.githubusercontent.com/facebook/hermes/v0.9.0/test/hermes/optional-chaining.js + supports: The Hermes 0.9.0 tag includes executable optional-chaining coverage. + - url: https://raw.githubusercontent.com/facebook/hermes/v0.9.0/test/hermes/nullish-coalescing.js + supports: The Hermes 0.9.0 tag includes executable nullish-coalescing coverage. + notes: The supplied runtime pairing directly supports the syntax used by this clean control. + excluded_claims: [Image fallback behavior and older Hermes versions are not scored.] +issues: [] diff --git a/research/benchmarks/fixtures/react-native/fixture-08.yaml b/research/benchmarks/fixtures/react-native/fixture-08.yaml index 6676cdf..4322945 100644 --- a/research/benchmarks/fixtures/react-native/fixture-08.yaml +++ b/research/benchmarks/fixtures/react-native/fixture-08.yaml @@ -1,33 +1,54 @@ +schema_version: 2 id: rn-08 category: release-safety -description: over-the-air update with no version gate or rollback strategy -code: | - // App.tsx - import * as Updates from 'expo-updates'; - - useEffect(() => { - async function checkUpdate() { - const update = await Updates.checkForUpdateAsync(); - if (update.isAvailable) { - await Updates.fetchUpdateAsync(); - await Updates.reloadAsync(); // immediate reload, no user prompt +description: An update check reloads immediately despite a user-initiated activation requirement. +intentional_clean: false +public: + file_path: App.tsx + language: TypeScript JSX + frameworks: + expo: '52' + expo-updates: '0.26' + runtime: hermes + assumptions: + - expo-updates is configured for this application. + - checkForUpdateAsync can report an available update. + requirements: + - The application must tell the user about an available update and let the user initiate any reload. + code: | + useEffect(() => { + async function checkUpdate() { + const update = await Updates.checkForUpdateAsync(); + if (update.isAvailable) { + await Updates.fetchUpdateAsync(); + await Updates.reloadAsync(); + } } - } - checkUpdate(); - }, []); + checkUpdate(); + }, []); +metadata: + provenance: + kind: synthetic + origin: Expo Updates documentation + revision_reason: Removed unsupported native-version and rollback claims; retained only the explicit activation-choice requirement. + difficulty: HAS-BUGS + domains: [updates, release] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://docs.expo.dev/versions/latest/sdk/updates/ + supports: expo-updates provides update checking, fetching, and reload APIs. + notes: The source shows an immediate reload and the requirement establishes the forbidden behavior. + excluded_claims: [Native version gates, rollback, release channels, and network-error handling are not scored.] issues: - - severity: MAJOR + - id: rn-08-immediate-reload + rule_id: react-native.release-safety category: release-safety - description: "OTA update reloads the app immediately without user consent; if update introduces a crash, users are forced into a broken state with no rollback path" - evidence: "await Updates.reloadAsync(); // immediate reload, no user prompt" - expected_checks: [release-safety, operability] - - severity: MAJOR - category: release-safety - description: "no minimum app version check before applying OTA update; update might target a different native binary version and cause incompatibility crashes" - evidence: "const update = await Updates.checkForUpdateAsync();" - expected_checks: [release-safety, upgrade-safety] - - severity: MINOR - category: operability - description: "no error handling on checkForUpdateAsync/fetchUpdateAsync; network errors during update check throw and crash the app on startup" - evidence: "const update = await Updates.checkForUpdateAsync();" - expected_checks: [operability] + severity: MAJOR + description: The available-update branch fetches and reloads immediately instead of waiting for the required user-initiated activation. + evidence_spans: + - lines: [3, 4, 5, 6] + rationale: An available update directly leads to fetchUpdateAsync followed by reloadAsync without a user action. diff --git a/research/benchmarks/fixtures/react/fixture-01.yaml b/research/benchmarks/fixtures/react/fixture-01.yaml index 57032d5..062cd9e 100644 --- a/research/benchmarks/fixtures/react/fixture-01.yaml +++ b/research/benchmarks/fixtures/react/fixture-01.yaml @@ -1,32 +1,66 @@ +schema_version: 2 id: react-01 category: hook-correctness -description: stale closure in useEffect with missing dependency array -code: | - function SearchResults({ query }) { - const [results, setResults] = useState([]); - const [page, setPage] = useState(1); +description: Search results must follow the current query and page. +intentional_clean: false +public: + file_path: src/SearchResults.tsx + language: TypeScript JSX + frameworks: + react: '18.3' + runtime: browser + assumptions: + - Changing page updates the page prop without remounting the component. + - A response for an earlier query or page may resolve after a later request. + requirements: + - The visible results must correspond to the current query and page. + code: | + function SearchResults({ query, page }) { + const [results, setResults] = useState([]); - useEffect(() => { - fetch(`/api/search?q=${query}&page=${page}`) - .then(r => r.json()) - .then(data => setResults(prev => [...prev, ...data])); - }, [query]); // page is missing from deps + useEffect(() => { + fetch(`/api/search?q=${query}&page=${page}`) + .then((response) => response.json()) + .then((data) => setResults(data)); + }, [query]); - return
{results.map(r =>
{r.title}
)}
; - } + return
{results.map((result) =>
{result.title}
)}
; + } +metadata: + provenance: + kind: synthetic + origin: React effect dependency documentation + revision_reason: Re-adjudicated to expose the changing page and response-order assumptions publicly. + difficulty: HAS-BUGS + domains: [hooks, async] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://react.dev/reference/react/useEffect + supports: Effects must declare reactive values they read. + - url: https://react.dev/reference/react/useEffect#fetching-data-with-effects + supports: Effect cleanup can prevent a later response from being overwritten by an earlier request. + notes: The missing page dependency and stale-result path are independently reviewable from the public assumptions and shown effect. + excluded_claims: [No particular stale-result guard or cancellation mechanism is required.] issues: - - severity: MAJOR + - id: react-01-missing-page-dependency + rule_id: react.hook-correctness category: hook-correctness - description: "page missing from useEffect dependency array causes stale closure; changing page silently fetches with wrong value" - evidence: "}, [query]); // page is missing from deps" - expected_checks: [hook-correctness, async-correctness] - - severity: MAJOR + severity: MAJOR + description: The effect reads page but its dependency list excludes page, so a page change does not start the required request. + evidence_spans: + - lines: [5, 8] + rationale: The request interpolates page and the dependency list contains only query. + - id: react-01-stale-response + rule_id: react.async-correctness category: async-correctness - description: "no cancellation/teardown: concurrent fetches race and stale responses overwrite newer ones" - evidence: "fetch(`/api/search?...`).then(r => r.json()).then(data => setResults" - expected_checks: [async-correctness] - - severity: MINOR - category: render-behavior - description: "results array grows unboundedly on re-render without reset on query change" - evidence: "setResults(prev => [...prev, ...data])" - expected_checks: [state-ownership, render-behavior] + severity: MAJOR + description: An earlier request can resolve last and publish results that do not correspond to the current query or page. + evidence_spans: + - lines: [5, 7] + rationale: Each effect invocation starts a request whose completion writes results without checking that it is still current. + - lines: [4, 5, 7] + rationale: The effect starts the request and unconditionally publishes its eventual result. diff --git a/research/benchmarks/fixtures/react/fixture-02.yaml b/research/benchmarks/fixtures/react/fixture-02.yaml index c1aa1e2..135a5fb 100644 --- a/research/benchmarks/fixtures/react/fixture-02.yaml +++ b/research/benchmarks/fixtures/react/fixture-02.yaml @@ -1,38 +1,58 @@ +schema_version: 2 id: react-02 category: render-behavior -description: expensive computation in render path without memoization -code: | - function DataGrid({ rows, filters }) { - const visibleRows = rows.filter(r => { - return filters.every(f => { - const val = getNestedValue(r, f.path); // O(depth) traversal - return matchFilter(val, f); // regex compile on every call - }); - }); +description: A large editable grid uses unstable row keys. +intentional_clean: false +public: + file_path: src/DataGrid.tsx + language: TypeScript JSX + frameworks: + react: '18.3' + runtime: browser + assumptions: + - rows can be reordered after a filter changes. + - Each row has a stable id. + requirements: + - Editing state must remain associated with the same row after reordering. + code: | + function DataGrid({ rows, filters }) { + const visibleRows = rows.filter((row) => filters.every((filter) => matchFilter(row[filter.key], filter))); - return ( - - {visibleRows.map((row, i) => ( - - {row.cells.map(c => )} - - ))} -
{c}
- ); - } + return ( + + + {visibleRows.map((row, index) => ( + + {row.cells.map((cell) => )} + + ))} + +
{cell.value}
+ ); + } +metadata: + provenance: + kind: synthetic + origin: React list rendering documentation + revision_reason: Replaced generic memoization and virtualization preferences with a stated identity requirement. + difficulty: HAS-BUGS + domains: [rendering, list-identity] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://react.dev/learn/rendering-lists + supports: Keys identify list items when their order changes. + notes: The row id assumption makes the keyed identity defect assessable. + excluded_claims: [Filtering cost and virtualization are not scored without a performance requirement.] issues: - - severity: MAJOR + - id: react-02-index-key + rule_id: react.render-behavior category: render-behavior - description: "filter + nested traversal + regex compile runs synchronously on every render; O(rows × filters × depth) in the render path" - evidence: "const visibleRows = rows.filter(r => { return filters.every(f => {" - expected_checks: [render-behavior, hook-correctness] - - severity: MAJOR - category: render-behavior - description: "key=index on list items: row reorder causes full re-render and lost input state" - evidence: "" - expected_checks: [render-behavior] - - severity: MINOR - category: render-behavior - description: "no virtualization for potentially large row sets" - evidence: "{visibleRows.map((row, i) =>" - expected_checks: [render-behavior] + severity: MAJOR + description: The index key changes row identity when the filtered order changes, contrary to the stated editing-state requirement. + evidence_spans: + - lines: [8] + rationale: The mapped rows have a stable id but use index as their key. diff --git a/research/benchmarks/fixtures/react/fixture-03.yaml b/research/benchmarks/fixtures/react/fixture-03.yaml index 0d7a307..b5e12e1 100644 --- a/research/benchmarks/fixtures/react/fixture-03.yaml +++ b/research/benchmarks/fixtures/react/fixture-03.yaml @@ -1,31 +1,69 @@ +schema_version: 2 id: react-03 category: state-ownership -description: direct state mutation and unsafe reducer pattern -code: | - function useCart() { - const [items, setItems] = useState([]); +description: Cart state is mutated before React receives an update. +intentional_clean: false +public: + file_path: src/useCart.ts + language: TypeScript + frameworks: + react: '18.3' + runtime: browser + assumptions: + - items is state owned by this hook. + - Consumers may memoize individual cart items by object identity. + requirements: + - Cart updates must use immutable state transitions. + code: | + function useCart() { + const [items, setItems] = useState([]); - const addItem = (item) => { - items.push(item); // direct mutation - setItems(items); - }; + const addItem = (item) => { + items.push(item); + setItems(items); + }; - const updateQty = (id, qty) => { - const found = items.find(i => i.id === id); - if (found) found.qty = qty; // mutates object in place - setItems([...items]); - }; + const updateQty = (id, qty) => { + const found = items.find((item) => item.id === id); + if (found) found.qty = qty; + setItems([...items]); + }; - return { items, addItem, updateQty }; - } + return { items, addItem, updateQty }; + } +metadata: + provenance: + kind: synthetic + origin: React state update documentation + revision_reason: Converted answer comments to public ownership assumptions and span-level evidence. + difficulty: FUNDAMENTALLY-FLAWED + domains: [state, rendering] + adjudication: + status: reviewed + method: Source-and-context rule adjudication + reviewer: codex-test-builder-review + reviewed_on: '2026-09-19' + sources: + - url: https://react.dev/learn/updating-arrays-in-state + supports: Arrays held in state should be treated as immutable. + - url: https://react.dev/learn/updating-objects-in-state + supports: Objects held in state should be treated as immutable. + notes: Both mutations occur before the setter call in the displayed state owner. + excluded_claims: [Exact rendering consequences for every consumer are not scored.] issues: - - severity: CRITICAL + - id: react-03-array-mutation + rule_id: react.state-ownership category: state-ownership - description: "items.push(item) mutates state array in place; React bailout means re-renders are skipped unpredictably" - evidence: "items.push(item); // direct mutation" - expected_checks: [state-ownership] - - severity: MAJOR + severity: MAJOR + description: addItem mutates the items state array before supplying it to setItems. + evidence_spans: + - lines: [5, 6] + rationale: items.push changes the state array and the same array reference is passed to the setter. + - id: react-03-object-mutation + rule_id: react.state-ownership category: state-ownership - description: "found.qty = qty mutates an object inside state; spread setItems([...items]) creates new array but keeps same object references; memoized children won't re-render" - evidence: "if (found) found.qty = qty; // mutates object in place" - expected_checks: [state-ownership, render-behavior] + severity: MAJOR + description: updateQty mutates an object already held in items before copying only the array. + evidence_spans: + - lines: [11, 12] + rationale: found references an item in state and its qty property is assigned directly. diff --git a/research/benchmarks/fixtures/react/fixture-04.yaml b/research/benchmarks/fixtures/react/fixture-04.yaml index 22fde83..7851874 100644 --- a/research/benchmarks/fixtures/react/fixture-04.yaml +++ b/research/benchmarks/fixtures/react/fixture-04.yaml @@ -1,26 +1,51 @@ +schema_version: 2 id: react-04 category: security -description: XSS via dangerouslySetInnerHTML with unescaped user input -code: | - function CommentBody({ comment }) { - // "safe" because we strip