Repository navigation
cms: bulk import of statement edits, with a preview and a stale-edit guard - #26
Merged
Merged
Conversation
|
review: ac13a9e |
|
review: 506a565 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes LTET-26
What changed
src/lib/statement-import.tsparses the file (parseImport: 2 MB of UTF-8 at most, valid JSON, aneditslist of 1 to 500 entries, each with anidtext and a non-emptysetwhose every path has ownfromandto), then plans it against the live statements (planImport). A path walks, by recursion, own keys of objects and existing indices of arrays only.__proto__,constructorandprototypein any segment, an empty segment,length, or a missing key is an unknown path, so no key is ever created and the prototype is never reached. Each statement is planned on a clone of its body. Skips: unknown id, an id that appears more than once in the file (all its edits), the pathid, an unknown path, overlapping paths in one set (scores.doomwithscores.doom.score), a stalefrom(deep-equal against the live value), and a result that failsstatementErrors(its errors listed). One skipped edit leaves the whole statement out. An edit whosetoequalsfromis no change; a statement with only those is listed as unchanged and not written.StatementService.planImportreads every statement once and plans.applyImportplans again from the same JSON and hands every ready statement toStatementRepository.replaceManyin one transaction. Each row is first locked withupdateMany where {id, updatedAt}(a write first, so the transaction waits on a busy writer instead of failing), and a row changed since the plan read it is left out and reported as "the statement changed during the import". Each written row gets a history entry with the notebulk import, and its published flag is untouched. The transaction hasmaxWait10 s andtimeout60 s, room for 500 statements. A database error writes nothing.onChangeruns once at the end, and not at all when nothing was written./admin/import(GET the form, POST/admin/import/previewand/admin/import/apply), under the Statements tab, linked as "Import edits" from the statements list. The preview lists each statement that will change with a Path / Before / After table (values as JSON), then the skipped statements with their reasons, then the unchanged ones, with an Apply button carrying the exact previewed JSON and the editable form below. Apply renders the result on the POST. A second Apply finds every applied edit stale and writes nothing./admin/import/skip it and the import router parses its two POST routes with a 7 MB limit (2 MB of JSON can triple when url-encoded), so that body is parsed only after the login and the same-origin check. Over 7 MB is the existing 413 answer.td codewraps anywhere, and the import form is full width (.stack.import). No side scroll at 390 px.New user-facing texts
Import editsPaste a JSON file of edits. Preview shows every change and every skipped edit, and nothing is written until you apply., the format line,A path is a dotted path into the statement, like venue, backlash.summary or scores.tone-deafness.score. It must exist already, and from must match the live value. A statement with any skipped edit is left as it is. At most 500 edits and 2 MB., labelEdits (JSON), buttonPreview<n> statement(s) will change, <n> skipped, <n> unchanged.,Apply to <n> statement(s),Nothing to apply., headingsWill change,Skipped,Unchanged,Edit and preview again, notesThese statements are left as they are.andEvery edit already matches., table headingsPath,Before,AfterApplied to <n> statement(s)., headingChangedno statement has this id,the id appears more than once in the file,no such path in the statement,the id cannot change,overlaps another path in the same edit,from does not match the live value, it was edited since,the statement changed during the import, or thestatementErrorsmessagesThe file is larger than 2 MB.,The file is not valid JSON.,The file needs an "edits" list.,The edits list is empty.,At most 500 edits per import.,edits[i] needs an "id" text.,edits[i] needs a "set" with at least one path.,edits[i].set["<path>"] needs "from" and "to".bulk importTested
The edge-case list on the issue, by number:
1, 2, 3, 4:
parseImportrejects invalid JSON / wrong shape (9 cases) / an empty list / an empty set; limits the number of edits, at and over; limits the bytes, at and over (and a 2-byte character over); e2e form error over 2 MB, a file over 256 KB accepted, 413 over 7 MB.5: plan skips an unknown id. 6, 8: plan skips unknown paths and never creates a key;
locaterefuses 19 paths. 7: plan skips__proto__,prototypeandconstructoras a top, nested or leaf segment and writes nothing, and never touches the prototype. 9: plan skips the id path. 10: plan skips a stale edit. 11: plan skips an invalid result with the rule's errors. 12: plan treats a no-op as no change; lists a statement whose every edit is a no-op as unchanged. 13: plan skips a repeated id as a whole. 14: plan skips overlapping paths. 15: plan andapplyImportleave a statement with one bad edit untouched and apply the others; e2e the same. 16: e2e the written statement stays unpublished. 17: plan finds every edit stale once applied; e2e a second apply writes nothing. 18: e2ereplaceManyskips a statement changed since it was read and writes the rest;applyImportreports it. 19:applyImportcalls onChange once, not at all when nothing is ready, changed or on a database error. 20: e2e GET and an anonymous POST redirect to the login and write nothing, a large anonymous POST too, a cross-origin POST is 403. 21: e2e history entry withbulk importand the old body; service passes the note.Also: router unit tests for the form, preview, no apply when nothing changes, apply, form errors keeping the text, a missing field, and a 500. Plans through an array index. The real
prisma/seed/imports/tone-deafness.json(95 edits, 35 KB) parses, and against the statements as they were before LTET-16 all 95 are ready with nothing skipped. Screenshots of the form and a preview at 390 and 1280 px, no side scroll, no console errors.npm run build,npm run test:unit(787),npm run test:e2e(1048),npm run format:check, all green on 506a565.Look at first
src/lib/statement-import.ts, thenStatementRepository.replaceMany.