[drietsch]

Fixes the paged.data defects the paged.media films recorded (DATA-1 to DATA-4; DATA-5 is reported below, most of it lives in core and the editor). Each fix landed with a test that failed first. Version: 0.1.0-canary.16 (the publish workflow ships it after a merge).

DATA-1: an unindexed barcode encoded the wrong record (A)

Cause. Every result was numbered by content: stable_order with no keys sorts by every column. So "record 0" of SELECT * FROM catalogue was the alphabetically first product, Bookmark set (2000000210209), not the first row, Linen notebook (2000000210018). Data Merge, the Dataset preview and InDesign number records in the order the data arrives, so an unindexed binding, the stepper's record 1 and the merge's first page all disagreed.

Fix. Record N is the Nth record as delivered when the query defines its order, and the Nth by content only when it does not (spec §6.1; ADR 553 amended). data-query::result_order_is_defined reads the SQL against DuckDB's order-preservation rules, and errs on the safe side:

• defined: a top-level ORDER BY, or a scan of one table through order-preserving clauses (WHERE, LIMIT, OFFSET, UNION ALL, a derived table or CTE over such a scan). DuckDB keeps insertion order there, because preserve_insertion_order is on by default.
• undefined: joins, grouping, DISTINCT, other set operations, windows with a partition or an order, sampling, subqueries outside FROM, and an empty statement.

Tables, record flows, rules, batch plans (plan_batch_ordered), the stepper and every per-record binding share that numbering (record_row / record_order). DP-4 still holds: for a query whose order is undefined, the same rows delivered in another order are no change.

Tests.

• data-conformance/tests/record_order.rs failed on origin/main: 3 of 4 tests (2000000210209 vs 2000000210018; table and stepper in content order; ORDER BY ignored).
• data-query order.rs unit tests.
• automation.rs keeps its content-order assertions over a join and adds the scan case.
• Pipeline Part C: record 0 of a scan is now its first delivered row.
• Perf pins W1/W3/W6 stabilizeCalls 1 → 0, as the harness asked: a scan sorts nothing.

A related fix found on the way: a barcode re-lower or preview step never replaced the symbol

Cause. The update-in-place batch removed the old modules (deleteFrame) and then drew the new ones (insertPath). Real core (protocol 71) refuses that batch with "position 30 out of range for parent Spread … (len 0)". The refusal was only a log line, so the old bars stayed. The existing relower test checked only the module count, and a refused batch keeps the count too.

Fix. Draw first, then remove, still in one batch and one undo step. The label still lands on the last new module ($created). The warning now includes core's error.

Test. relower-real-core now compares the bars and requires record 1's symbol to replace record 0's. It failed with the old order.

DATA-2: Data Merge dropped the template frame's fill and inset

Cause. Merged frames are minted fresh, because core cannot copy a frame whose story holds a placeholder hyperlink. Only the story-level formatting was copied onto them.

Fix. readMergeTemplate also reads each template frame's own properties. It keeps every property the frame sets except its place, threading, layer, lock and visibility (isCopiedFramePath), with the object style first. Values a fresh frame already has, such as InDesign's [Normal Text Frame] or one column, are skipped (isSetValue). The writer sets the kept properties on each merged frame in the same batch.

The new read is paid for: the session now reads the pages and the scene tree once for both the template and the re-merge plan. W8: host calls 18 → 17, reads 12 → 11, mutation ops unchanged (403). The 8 InDesign recordings still match.

Tests.

• test/films-real-core.spec.ts DATA-2 merges a card template that is filled, inset, stroked and centred, and checks every merged frame. It failed before: fill of ud: expected null to be 'Color/u1'.
• merge.test.ts covers the pure part.

DATA-3: the stepper started disabled; Data query Preview stayed disabled

Cause. The stepper's record total and position were panel state that started at 0 and was set only by stepping. Every mount, including a tab switch, began at "— / —". In the Data query panel, Preview and Save were enabled only by typed SQL, while the field showed the source's rows as a placeholder.

Fix.

• Stepper state lives in the session. The session keeps the records each query delivered (getState().records, taken from the delivered batch with no engine call) and the stepped-to record (getState().preview, setPreviewPosition). refreshData emits a change. The stepper shows "1 / N" once the query has run and keeps its place across a remount.
• "next ›" before any data runs the query. It then shows record 1.
• Data query. With no SQL typed, Preview and Save run what the field shows: SELECT * FROM "<source>".

Tests. All of these failed before the fix:

• panels.test.ts: the stepper opens at "1 / 24", keeps "2 / 24" across a remount, and "next ›" runs the data.
• query-panel.test.ts: Preview after Read columns.
• preview.test.ts: the session keeps counts and position.

DATA-4: Data Merge could not be started from a script

Cause. Scripts reach typed commands (paged.invoke, the object model). The data plugin had typed commands for refresh, apply and data sets, but not for the merge. Only the plain menu command ran it.

Fix. A new typed command, media.paged.data.merge, with flat optional arguments: query, mode, arrange, spacings, removeBlankLines, template, destination, pageId, documentName. It and the menu command both call runMergeCommand (src/merge-command.ts). A script gets { ok, records, pages, overset, message, diagnostics }.

Test. films-real-core DATA-4 checks that the command is listed and merges through h.objects.invoke, both with options and with none (the menu's defaults). It failed before because the command was not listed.

DATA-5 (C): mostly not in this repo

• Pages named "u5". insertPage mints a page with no Name, and duplicatePage copies the template's Name verbatim (every page "1"); seen in an exported package. The editor's filmstrip shows the spread label, and its id test (isIdLikeLabel, packages/shell/src/cockpit/spread-grouping.ts:39) needs at least 3 hex digits, so u5 is shown as a label. To fix in core (page names for minted pages) and/or the editor.
• Non-field template items on every page. This matches InDesign: a merged page is a copy of the template page. Both films rely on it for the running header, the card and the rule. Not changed.
• Blank filmstrip thumbnails. Core's mutation answer lists the new page ids (checked headless). The thumbnails come from the editor's snapshots map, which has no entry for pages minted in a batch (ThumbnailRail.tsx). Editor.
• View left at 3 %. On pageStructureChanged the editor rebuilds the document handle (packages/shell/src/PagedShell.tsx:1133). The view then fits all 24 pages. Editor.

Checks (local)

• cargo nextest run --workspace --profile ci: 317/317. Clippy -D warnings is clean, as are cargo fmt, the coverage gate (GREEN, 130 rows) and both dependency guards.
• pnpm typecheck passes. pnpm test with REQUIRE_REAL_ENGINE=1 REQUIRE_REAL_DUCKDB=1 REQUIRE_REAL_CORE=1: 398 passed, 2 skipped (+53 host-model). The manifest is valid and the package-hash check passes (canary.16 is new).
• cargo deny check: advisories ok, bans ok, licenses ok, sources ok (after the dependency swap below).

Dependency: serde_yaml replaced by serde_norway (RUSTSEC-2024-0449)

Cause. The Check job's cargo-deny step failed on unsafe-libyaml, which is unmaintained (RUSTSEC-2024-0449). It reached the workspace only through serde_yaml 0.9.34+deprecated, which reads the registry rows in the data-core and data-expr build scripts and in the coverage gate.

Fix. serde_norway 0.9.42, the maintained fork of serde_yaml 0.9: same parser, same Value, same from_str. It sits on unsafe-libyaml-norway. Licences are MIT OR Apache-2.0 and MIT, both inside deny.toml's allow-list. serde_yaml_ng was rejected because it still depends on unsafe-libyaml. The change is a crate rename in three files.

Identical behaviour.

• The generated code is byte-identical: funcs.rs sha256 f6f377e9… and dispatch.rs 30cd6df8…, before and after.
• All 23 registry YAML files parse to identical Values under both crates. A scratch harness compared their JSON and Debug renderings.
• The coverage gate prints the same report.
• The engine wasm is not byte-identical: 100 bytes and 2 functions smaller after stripping. That comes from the crates' metadata hashes changing with their build-dependency set, not from any code. Building the same sources with only __CARGO_DEFAULT_LIB_METADATA changed moves the code by 665 bytes and 10 functions.

🤖 Generated with Claude Code