[laithalsaadoon]

On DuckDB 2.0.0, lake rebuild and lake verify run out of memory on the host's real data, inside the edges read_json at registry.py:904. atif-sql 0.4.0 (DuckDB 1.5.6) passes both on the same data. One commit fixes it.

Cause

The edges reader sized maximum_object_size from the largest edges.jsonl in the statement, plus a quarter and 1 MiB. A newline-delimited file's objects are its lines, though, and DuckDB allocates every read buffer at twice the bound (buffer_capacity = maximum_object_size * 2 in json_scan.cpp, the same in 1.5.6 and 2.0.0). So one large file set the buffers of every file in its lake batch. The interactive corpus holds a session whose edges.jsonl decodes to 120 MB, so its batch of 512 sessions read all 512 files with 302 MB buffers.

2.0 holds more of those buffers at once. Its read_json over a file list is a multi-file wrapper around a single-file scan (read_single_json_file.cpp). Every file gets its own scan state, with enable_parallel_scans true because one file is fewer than the thread count, and files are opened ahead of the scan. On 1.5, a list at least as long as the thread count read each file whole, with one buffer a thread.

Measured on the real snapshot at four threads (the lake writer's setting), the smallest memory_limit that read that batch's edges was 1,342 MiB on 1.5.6 and 2,046 MiB on 2.0.0. The writer's cap is 2,048 MiB, and the writer holds other tables while it reads, so the load failed. A batch size of 64 didn't help, because the bound and not the file count sets the buffer size.

Fix

The edges and session_events readers now take their bound from the longest line (_line_size_bound). A file that fits under DuckDB's 16 MiB default with its headroom counts at its own size and isn't read. A larger one is read once, a chunk at a time, by the new stored_artifacts.longest_line. The longest line across the 13,071 edges.jsonl files of the snapshot (1.9 GB decoded) is 960 bytes. Across their session_events.jsonl files it's 8,913 bytes. So in practice the bound is DuckDB's default, and a read's buffers no longer grow with its largest file. The trajectory reader keeps the file-sized bound, because a trajectory.json is one document.

This keeps the 2 GiB writer cap. With the line bound the real 512-session batch needs 79 MiB (it needed 2,046 MiB before), so a larger cap would only hide the next oversized bound.

Tests

• TestWriterMemoryCap (test_lake.py) rebuilds and then verifies an eight-session corpus holding a 64 MiB edges.jsonl, under a 448 MiB writer cap. It needs 207 MiB with the fix and 972 MiB without it. On main's source it fails with OutOfMemoryException: failed to allocate data of size 165.3 MiB (340.8 MiB/448.0 MiB used) at registry.py:904, the same line as the shadow failures. With the fix it passes in about 2.4 s.
• TestLineSizeBound covers the line rule: a large file of short lines stays at the floor, a long line gets the headroom, the longest line of any file wins, a .zst file is measured decompressed, a file the floor already covers isn't read, the 1 GiB cap holds, and a missing file counts as zero.
• test_longest_line_spans_read_chunks covers lines that straddle the 1 MiB read chunks, a last line with no newline, and an empty file, plain and compressed.
• The spy test now expects one file bound (trajectories) and two line bounds (session events, edges), all at the floor.

The docs citations into registry.py move with the code (docs:citations:fix). The contract map now says which bound each reader uses.

Evidence

On loopbox duckdb2-20261025 (m8a.4xlarge, 64 GiB), tree at a356141, venv from mise run install:ci (duckdb 2.0.0):

• mise run --force check: exit 0. lint, ty and pyright (0 errors), lint:imports (7 contracts kept), docs:prose, security:vex:check, and test: 2695 passed, 3 skipped, 1 xfailed in 167 s.
• mise run docs:install && mise run docs:gate: exit 0. docs:citations reports 0 problems in 2,468 citations, and docs:links 0 broken of 1,980 internal links.

Shadow rerun on the real snapshot

On loopbox duckdb2-shadow-20261025 (m8a.8xlarge), with the scripts in the job's shadow/setup/ (phaseJ.sh). Every run used a fresh reflink copy of the 2026-10-25 snapshot (five corpora, 13,071 sessions) under a systemd scope pinned to 16 CPUs, with 1.5 rerun back to back. The fix build is this branch's tree from uv sync --frozen, with the same uv.lock as main.

Command (scope)	Build	Exit	Wall	Peak RSS
lake verify, untouched lake (110G)	0.4.0 / DuckDB 1.5.6	0	2:37.6	3.22 GiB
	main 30bb3e2 / 2.0.0 (2026-10-25 run)	1 (OOM)	2:38.7	1.39 GiB
	this branch / 2.0.0	0	2:50.2	1.39 GiB
lake rebuild (24G)	0.4.0 / 1.5.6	0	3:45.4	3.17 GiB
	main 30bb3e2 / 2.0.0 (2026-10-25 run)	1 (OOM)	2:55.2	1.72 GiB
	this branch / 2.0.0	0	4:00.1	1.72 GiB
lake verify of the rebuilt lake (110G)	0.4.0 / 1.5.6	0	2:15.4	3.26 GiB
	this branch / 2.0.0	0	2:30.2	1.40 GiB

• All four verify reports are clean (0 mismatches) over the same five corpora and 13,071 sessions. Both rebuilds wrote the same corpora and 112 data files.
• The 2.0 rebuild peaks at about half of 1.5's RSS. It's 15 s slower than 1.5, and the new line scan isn't the cause: it reads 10 files over the floor (309 MiB decoded) in 0.22 s.
• The smallest memory_limit for the real 512-session edges batch is now 79 MiB, down from 2,046 MiB. The biggest single file alone needs 75 MiB, down from 639 MiB.
• Query diff, 1.5 on the untouched lake against this branch on its rebuilt lake: 39 of the 42 statements (the 40 skill and heavy queries plus the two corpus queries) match row for row. heavy09_regex_ops differs on both corpora, with the same hashes as the first 1.5-against-2.0 run. That's 2.0's partial-match ~: 0 matches on 1.5 against 16,406 on 2.0, the same as regexp_matches. No SQL text here uses ~. skill05_perceived_counts matches on every count, and its avg_confidence differs only in the 16th digit. It differs the same way between two runs of this branch, so that's summation order.