feat(v2): workspace projection endpoint + extraction-table data layer#233
Merged
Conversation
…ve grid, review-page retirement
…table-value constraint
…architectural details - Enhanced the architecture section to detail the use of DuckDB for in-memory denormalization, replacing previous methods. - Clarified the tech stack to include DuckDB as a denormalization engine. - Updated Phase 1 description to specify the addition of DuckDB as a new dependency. - Added new integration tests for the workspace projection functionality.
…ema id - toPerspectiveData: add a reverse-alphabetical fixture asserting Object.keys() equals [reference, ...manifest order] so a stray alphabetical sort regresses the test, not just the comment. - Cover empty-rows for toPerspectiveData/bandParity, a cells key absent from the manifest, and a reference recurring after a gap ([0,1,0]). - buildAnnotationUrl: encodeURIComponent the schemaId path segment too (not just the reference), plus tests for &/# and a slashed schema id. - Correct the toPerspectiveData comment: the integer-like-key hazard is avoided today because column names always contain a dot, not because of anything the function does — and note Phase 2 should pass projection.columns as Perspective's explicit column config rather than rely on inferred key order.
Pins the two hazards the paging fix relies on but that were previously unasserted: (1) columns are captured once from the first page, in server order, never sorted or duplicated across pages; (2) an empty page with a huge totalReferences stops after one call instead of hammering the endpoint. Also captures totalReferences from the first page only (matching columns), and documents the server spine-CTE invariant the zero-progress guard depends on.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… review gaps Addresses medium-severity roborev findings on the Phase 1 branch. Behavior fix: - V2SuggestionValidator._validate_score now takes the question type and skips the list-cardinality rules for table questions. Those rules assume a list value is a list of answer choices (multi_label_selection, ranking); a table value's list is N rows (spec 3.4), and a suggestion's score applies to the suggestion as a whole. Before this, a multi-row table suggestion with one confidence score -- the exact shape Task 3's fan-out consumes -- was rejected by upsert_suggestion. Hardening: - Serialize DuckDB inputs with ensure_ascii=False. The projection joins Python strings against keys DuckDB parses from JSON text; emitting non-ASCII names as \uXXXX made that join depend on DuckDB decoding them back. It does decode them correctly today, so this removes a dependency rather than fixing a live break. Test gaps closed: - table suggestions: single score over multi-row value, mismatched score list, and score list over a bare dict; plus non-table types still enforcing cardinality - multi-question response envelope: pins the positional key/value zip-unnest that every other test exercised with only one key - non-ASCII schema/question/sub-column names resolve end to end - response-over-suggestion cell: seed agent/score on the losing suggestion so the `is None` assertions prove suppression instead of passing vacuously - repository fixture: suggestion-sourced table sub-column with agent/score and a list-valued score, so a transposition or dropped sub_column now fails - use-case: first-page totalReferences capture (growing total mid-sweep) and the empty-workspace path; manifest test pages now return distinct manifests Server 138 passed; frontend 871 passed, typecheck/lint/gen:api drift all clean.
…ean up SQL denormalization logic - Added extraPaths for cursorpyright analysis to match Python analysis settings. - Simplified the SQL denormalization logic by replacing the detailed SQL statement with a placeholder, indicating that the exact SQL is to be defined by the implementer.
…ests Addresses roborev job #228 (auto-review of 2a1ac44). Medium — the table score exemption was total, accepting arbitrary list scores. That contradicts its own rationale (a table suggestion's score is whole-suggestion confidence) and is not inert: the fan-out repeats the whole list onto every cell, surfacing a multi-value score no consumer can read. Narrowed to `score is None or isinstance(score, (int, float))`; a per-row score is a distinct future feature needing indexed fan-out. Flipped the two list-accepting tests to assert rejection; mutation-verified they fail without the guard. Low — the two non-table negative tests paired type=multi_label_selection with LABEL_SETTINGS (settings type label_selection), a combination the API cannot persist that only worked because both settings models expose `.options`. Added "no" to MULTI_LABEL_SETTINGS and use it, so the type/settings pair matches what QuestionSettingsValidator enforces. Low — test_non_ascii_names_and_bindings_resolve keyed its response envelope on ASCII "pays", so the responses-side ensure_ascii=False was untested. Renamed the scalar question to "país" and key the envelope with it, exercising the response-envelope join (rc.question_name = q.question_name) against a non-ASCII key. Server 138 passed; ruff clean.
… phrase) The v2 OpenAPI drift gate dumps the schema under CI's Python (<=3.12), where HTTPStatus(422).phrase is 'Unprocessable Entity'. The committed snapshot was generated under Python 3.13, which renamed the phrase to 'Unprocessable Content', so the full-file diff -u failed across all 28 occurrences. Regenerated openapi.json + v2-api.ts under Python 3.12; no schema/type changes beyond the phrase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 of the extraction-table build (plan:
docs/superpowers/plans/2026-07-20-extraction-table.md, spec §3–§4). Everything here is additive and independently shippable: server contract + workspace projection endpoint, plus the frontend data layer up to DI registration. No UI, no deletions, no new frontend dependencies.Phase 2 (Perspective grid,
/extractionspage, review-page retirement) is a separate stacked PR.What's included
Server
list[dict]table values accepted additively — baredictremains the 1-row case (spec §3.4)ProjectionCellenriched withrecord_id/agent/score; optional, so SDK feat(sdk): v2 SDK vertical slice — generated DTOs, async transport, resources, agentic CLI #231 stays backward-compatiblebuild_workspace_view— workspace-wide denormalization: effective-record dedup, submitted-response??suggestion coalesce, table fan-out with independent stacking and scalar repetitionGET /api/v2/projection?workspace_id=&offset=&limit=(limit∈ [1,100] default 50)Frontend (data layer only)
openapi.json+v2-api.ts); repository types againstcomponents["schemas"]WorkspaceProjectiondomain types +ProjectionRepository.getWorkspaceProjectionGetWorkspaceProjectionUseCase+ExtractionsStorage+ DI wiringArchitecture note — DuckDB
The denormalization runs as one SQL statement in an in-process, in-memory DuckDB connection, fed by batched Postgres slices from the existing
AsyncSessionand offloaded viato_thread. DuckDB neverATTACHes Postgres (integration tests run in a rolled-back transaction a second connection can't see). This keeps the transform declarative over arbitrary schema-defined JSON and pre-builds the Arrow path for later streaming.duckdb>=1.5.4is the only dependency added anywhere in this PR.Verification
tests/unit/validators/v2,tests/integration/contexts/v2,tests/integration/api/v2)npm run gen:api && git diff --exit-code v2/infrastructure/api— cleanpackage.json/lockfile/nuxt.config.ts/vitest.config.tschanges, no deletions, nopages//components//e2e/changesKnown flaky suite (not introduced here)
components/features/dataset-creation/configuration/DatasetConfiguration.spec.jsfails in the full run with a@nuxt/test-utilssetupNuxthook timeout (10s) — a timing symptom, not an assertion failure. It passes 21/21 in isolation. It's a v1 dataset-creation component with zero references to v2 DI, extractions, or projection, and both the new DI registration and the new Pinia store are lazy.I was unable to run a
developbaseline to prove it pre-existing (the command was blocked), so this is strong evidence but not proof. Worth a second opinion from someone who knows whether this suite is already known-flaky in CI.Review notes
Two findings surfaced and were fixed during review, both worth knowing about:
QuestionCreate.columnsconstrains list length, not element content, so a"or""was storable via the public API and made DuckDB abort the entire statement — 500ing the whole grid for every reference, not just one cell. Fixed structurally: no data-derived text reaches a JSON path anymore (join onqc.sub_column = e.entry_key). Hostile names (sub"col,back\slash,"",*) are covered by tests.authorize()leaks the full workspace.Recorded decision: response coalescing selects the latest submitted response envelope per record (any user), so a question absent from that envelope falls back to its suggestion even if an earlier submitted response contained it. Record-level per spec §3.2 — documented in the code, deliberately not changed.
Column ordering (server definition order, not alphabetical) is now pinned by tests at both the repository and adapter layers.
🤖 Generated with Claude Code