Skip to content

cms: bulk import of statement edits, with a preview and a stale-edit guard - #26

Merged
Lindeneg merged 5 commits into
masterfrom
LTET-26-bulk-import
Nov 7, 2026
Merged

Lindeneg merged 5 commits into
masterfrom
LTET-26-bulk-import

Conversation

@Lindeneg

@Lindeneg Lindeneg commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Closes LTET-26

What changed

  • The plan. src/lib/statement-import.ts parses the file (parseImport: 2 MB of UTF-8 at most, valid JSON, an edits list of 1 to 500 entries, each with an id text and a non-empty set whose every path has own from and to), then plans it against the live statements (planImport). A path walks, by recursion, own keys of objects and existing indices of arrays only. __proto__, constructor and prototype in any segment, an empty segment, length, or a missing key is an unknown path, so no key is ever created and the prototype is never reached. Each statement is planned on a clone of its body. Skips: unknown id, an id that appears more than once in the file (all its edits), the path id, an unknown path, overlapping paths in one set (scores.doom with scores.doom.score), a stale from (deep-equal against the live value), and a result that fails statementErrors (its errors listed). One skipped edit leaves the whole statement out. An edit whose to equals from is no change; a statement with only those is listed as unchanged and not written.
  • The write. StatementService.planImport reads every statement once and plans. applyImport plans again from the same JSON and hands every ready statement to StatementRepository.replaceMany in one transaction. Each row is first locked with updateMany where {id, updatedAt} (a write first, so the transaction waits on a busy writer instead of failing), and a row changed since the plan read it is left out and reported as "the statement changed during the import". Each written row gets a history entry with the note bulk import, and its published flag is untouched. The transaction has maxWait 10 s and timeout 60 s, room for 500 statements. A database error writes nothing. onChange runs once at the end, and not at all when nothing was written.
  • The page. /admin/import (GET the form, POST /admin/import/preview and /admin/import/apply), under the Statements tab, linked as "Import edits" from the statements list. The preview lists each statement that will change with a Path / Before / After table (values as JSON), then the skipped statements with their reasons, then the unchanged ones, with an Apply button carrying the exact previewed JSON and the editable form below. Apply renders the result on the POST. A second Apply finds every applied edit stale and writes nothing.
  • Form size. The admin forms keep the 256 KB parser. Paths under /admin/import/ skip it and the import router parses its two POST routes with a 7 MB limit (2 MB of JSON can triple when url-encoded), so that body is parsed only after the login and the same-origin check. Over 7 MB is the existing 413 answer.
  • CSS. td code wraps anywhere, and the import form is full width (.stack.import). No side scroll at 390 px.

New user-facing texts

  • Link and heading: Import edits
  • Form: Paste a JSON file of edits. Preview shows every change and every skipped edit, and nothing is written until you apply., the format line, A path is a dotted path into the statement, like venue, backlash.summary or scores.tone-deafness.score. It must exist already, and from must match the live value. A statement with any skipped edit is left as it is. At most 500 edits and 2 MB., label Edits (JSON), button Preview
  • Preview: <n> statement(s) will change, <n> skipped, <n> unchanged., Apply to <n> statement(s), Nothing to apply., headings Will change, Skipped, Unchanged, Edit and preview again, notes These statements are left as they are. and Every edit already matches., table headings Path, Before, After
  • Result: Applied to <n> statement(s)., heading Changed
  • Skip reasons: no statement has this id, the id appears more than once in the file, no such path in the statement, the id cannot change, overlaps another path in the same edit, from does not match the live value, it was edited since, the statement changed during the import, or the statementErrors messages
  • Form errors: The file is larger than 2 MB., The file is not valid JSON., The file needs an "edits" list., The edits list is empty., At most 500 edits per import., edits[i] needs an "id" text., edits[i] needs a "set" with at least one path., edits[i].set["<path>"] needs "from" and "to".
  • History note: bulk import

Tested

The edge-case list on the issue, by number:

1, 2, 3, 4: parseImport rejects invalid JSON / wrong shape (9 cases) / an empty list / an empty set; limits the number of edits, at and over; limits the bytes, at and over (and a 2-byte character over); e2e form error over 2 MB, a file over 256 KB accepted, 413 over 7 MB.
5: plan skips an unknown id. 6, 8: plan skips unknown paths and never creates a key; locate refuses 19 paths. 7: plan skips __proto__, prototype and constructor as a top, nested or leaf segment and writes nothing, and never touches the prototype. 9: plan skips the id path. 10: plan skips a stale edit. 11: plan skips an invalid result with the rule's errors. 12: plan treats a no-op as no change; lists a statement whose every edit is a no-op as unchanged. 13: plan skips a repeated id as a whole. 14: plan skips overlapping paths. 15: plan and applyImport leave a statement with one bad edit untouched and apply the others; e2e the same. 16: e2e the written statement stays unpublished. 17: plan finds every edit stale once applied; e2e a second apply writes nothing. 18: e2e replaceMany skips a statement changed since it was read and writes the rest; applyImport reports it. 19: applyImport calls onChange once, not at all when nothing is ready, changed or on a database error. 20: e2e GET and an anonymous POST redirect to the login and write nothing, a large anonymous POST too, a cross-origin POST is 403. 21: e2e history entry with bulk import and the old body; service passes the note.

Also: router unit tests for the form, preview, no apply when nothing changes, apply, form errors keeping the text, a missing field, and a 500. Plans through an array index. The real prisma/seed/imports/tone-deafness.json (95 edits, 35 KB) parses, and against the statements as they were before LTET-16 all 95 are ready with nothing skipped. Screenshots of the form and a preview at 390 and 1280 px, no side scroll, no console errors.

npm run build, npm run test:unit (787), npm run test:e2e (1048), npm run format:check, all green on 506a565.

Look at first

src/lib/statement-import.ts, then StatementRepository.replaceMany.

@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown

review: ac13a9e
no findings

@claude

claude Bot commented Nov 7, 2026

Copy link
Copy Markdown

review: 506a565
no findings

@Lindeneg
Lindeneg merged commit 70c94ea into master Nov 7, 2026
4 checks passed
@Lindeneg
Lindeneg deleted the LTET-26-bulk-import branch November 7, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant