[azimafroozeh]

What & why

A load csv overwrite stages its table through one Lance call, which writes the table's data files one after another. On SF30 that serial write is 104 s of the 503 s load (#1099's phase table), with every other writer idle while Comment's 67 M rows go out file by file. This PR cuts the validated batches at Lance's file target (1 M rows) into chunks, runs one uncommitted Lance write per chunk with up to stage_write_concurrency (default 8) in flight, and joins the chunks' fragments in chunk order into one Operation::Overwrite. Each chunk is exactly the file the single call would have cut and Lance numbers fragments in list order at commit, so the committed files, their order and the single commit are unchanged while the file writes overlap: SF30's write phase drops to 47 s and the load to 413 s.

Backing issue / RFC

• The bulk-load RFC, docs/rfcs/2026-10-15-bulk-load-as-a-query-plan.md (feat(load): plan CSV overwrites and reuse validation reads #1026, perf(load): check edge endpoints in an unchained integer key set #1085, feat(explain): cut pipelines at breakers on every route and measure a load's operators #1099): its sink stages one Lance call per table, so the fan-out inside that call owes the RFC a sentence; D11 (each phase holds its whole input) is unchanged.

Checklist

• Focused: one logical change, the overwrite's Lance writes fan out across file-sized chunks
• Tests: two Rust tests in table_store, since the width is a process setting and every GQT case stays under 1 M rows (one chunk): chunk order and exactly-once at a 4-row chunk target with 3 writers (14 rows in 3-row batches, fragments [4, 4, 4, 2], staged ids 1..=4, rows read back in input order before and after the commit), and the one-schema refusal before any writer starts (no data file written, HEAD unchanged, a zero-row batch of another schema dropped as before). The GQT corpus (412 cases) is unchanged.
• Public docs: docs/user/queries/settings.md, docs/user/mutations/index.md; docs/dev/writes.md for the mechanism
• Release note: changelog.d/load-csv-overwrite-writers.performance.md
• Invariants: reviewed against docs/dev/invariants.md, no Hard Invariant weakened, no deny-list item hit; invariant 4 (a mutation publishes once) holds, the chunks join into one Operation::Overwrite

Local verification

• LDBC SNB SF30, 31 load csv … mode overwrite statements through omnigraph mutate --json, release builds of main (b692610c) and this branch, one round each, phases read from executed.operators as in feat(explain): cut pipelines at breakers on every route and measure a load's operators #1099: load 503.0 → 412.9 s, write phase 104.3 → 47.3 s, Comment's write 18.2 → 5.6 s; parse (190 s, one thread) is now the long pole.
• LDBC SNB SF1, one binary at width 1 and 8 (OMNIGRAPH_LOAD_CONCURRENCY), 2 rounds interleaved: load 13.22 → 12.00 s, write 2.63 → 2.18 s; only the six tables above 1 M rows change (write 1.3× to 1.5× faster). main equals this branch at width 1 within 3 %.

Notes for reviewers

• No format change: an existing store reads unchanged; the one-schema check a single call ran now runs before any writer starts and names the first differing field, and a failed chunk fails the stage with HEAD untouched.
• A default changes: stage_write_concurrency (OMNIGRAPH_LOAD_CONCURRENCY) now also widens the Lance writes inside one load csv overwrite; 1 restores the serial write.
• Memory tradeoff: each writer keeps the full write_max_bytes (32 MiB default) as its Lance accumulation allowance, so a table above 1 M rows can hold up to stage_write_concurrency times that in writer buffers. The allowance was already per writer; dividing it by the width would shrink a wide table's pages, and the numbers above were taken per writer.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T16:43:24.616901Z	3e0f91f	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e0f91f916

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".