Memory scale for large extracted array CSVs

[htsukamoto5]

Background

Spun out from #72 (item 2). The other items in that issue (nested-object expansion depth and primitive array detail) have been resolved; this one remains open.

Problem

extractedArrays in JsPsychMetadata (packages/metadata/src/index.ts) is an in-memory Map<string, Array<Record<string, any>>> that accumulates every extracted row across all processed files before anything is written to disk. For large datasets this can be expensive:

• Real Tobii eye-tracking dataset (DataPipe exports, ~26–70 MB JSON, ~1,700 trials/file, 38–39 files): tobii_data alone produced 94,198 rows per file → roughly 3.5 M rows held in memory simultaneously across a full study.

What to investigate / decide

1. Is it actually a problem in practice? Profile memory usage on the full 39-file Tobii dataset. If Node's GC keeps up and peak RSS is acceptable, the current approach may be fine.
2. If it is a problem: consider streaming / chunked writes — flush each file's extracted rows to disk as they're processed rather than accumulating across all files.
3. Interaction with the browser frontend: the frontend holds the same data as FileSystemFileHandle blobs; the scale concern may be more acute in the browser than in Node.

Related

• Future checks: nested-object expansion depth, extracted-array scale, and bare primitive arrays #72 (original tracking issue)
• PRs fix(metadata): register array-element fields in variableMeasured (closes #82) #84, feat(metadata): recursively unnest data inside extracted array elements #86, feat(metadata): extract primitive arrays under a synthetic .value column #87 (the array extraction work that surfaces this concern)

[htsukamoto5]

Investigation findings

Profiled/traced the extraction paths. The core premise here needs correcting, and the remaining concern splits in two.

The "accumulates across all files" premise no longer holds

extractedArrays/extractedObjects are reset at the top of every generate() call (packages/metadata/src/index.ts), and both the CLI (processFile) and the frontend (DataUpload.tsx) call generate() per file and write/convert that file's sidecars before the next file. So peak is one file's extracted rows (~94K for the Tobii example), not the study-wide ~3.5M described above.

This was actually fixed in b876c2d ("Fix extractedArrays accumulation across multiple generate() calls", 2026-06-01) — before this issue was filed (2026-06-10). The concern was inherited from #72's framing and not re-checked against current code. So the suggested fix ("flush per file rather than accumulate across files") is effectively already in place.

Two real items remain

1. Double-parse (small, fixed in #120). Each file was parsed twice during processing — once in generate(), once in the caller for CSV conversion. #120 makes generate() accept pre-parsed rows so a file is parsed once. Bounded by the largest single file; no output change.

2. Browser holds all converted output at once (the genuine ceiling). DataUpload.tsx accumulates every file's converted CSV strings and every raw original's full content into one converted map held in React state, and validatePsychDS rebuilds it all as Blobs — all needed simultaneously to produce the download. There's no flush-to-disk escape hatch (the File System Access API was deliberately not used). For a large multi-file study this would OOM the tab; the CLI streams to disk and has no such limit.

Item 2 is a product/architecture question (is the browser meant for large studies, or is that the CLI's job?) — raising it separately rather than guessing at scope here. Leaving this issue open to track that decision; #120 covers the double-parse.