Optimize GTFS ingestion (~40% faster) and drop bulk-load PRAGMAs

[chtitux]

Summary

• Rewrites the CSV loader to parse each file only once, use positional arrays instead of per-row Record<string, string> objects, and reuse a single prepared INSERT per table rather than re-preparing a multi-row VALUES (?,?),(?,?),... statement per 1000-row batch.
• Replaces the pre-flight double-parse (used only to populate progress totals) with an O(bytes) newline-count estimate. ProgressInfo.totalRows is now documented as an estimate (typically exact, may be ±a few per file).
• Drops the bulk-load PRAGMA block (synchronous=OFF, journal_mode=MEMORY, temp_store=MEMORY, cache_size=-64000, locking_mode=EXCLUSIVE) and its post-ingest reset. Benchmarked effect on sql.js is within noise (≤1%); removal unblocks pluggable-adapter work.

Full design doc: documents/gtfs-optimize-ingestion.md.

Measured impact

Full GtfsSqlJs.fromZipData() wall time, 5 runs after warmup, median:

Feed	Before	After	Δ
ASTUCE Rouen (5 MB zip / 34 MB uncompressed / ~430k stop_times)	2647 ms	1669 ms	−37%
Car Jaune (1 MB zip / 4.7 MB uncompressed)	312 ms	188 ms	−40%

Benchmark scripts (scripts/bench-pragmas.ts, scripts/bench-ingest.ts, scripts/bench-after.ts) are included so the wins are reproducible on other feeds / machines.

Behaviour changes (CHANGELOG)

• ProgressInfo.totalRows semantics: exact → estimate. Consumers using it as a post-ingest row count should use SELECT COUNT(*) instead.
• locking_mode = EXCLUSIVE no longer set during ingestion. No known consumer was relying on this.

Public API signatures are unchanged.

Test plan

• npm run typecheck passes
• npm run lint passes
• npm test — all 139 tests pass, including a new tests/progress-callback.test.ts that guards totalRows estimate accuracy, percentComplete bounds, and the terminal complete event.
• npx tsx scripts/bench-after.ts /tmp/gtfs-bench/astuce.zip — median 1669 ms (under 1700 ms budget from the plan).
• npx tsx scripts/bench-after.ts /tmp/gtfs-bench/car-jaune.zip — median 188 ms.
• Manual smoke test on a very large feed (e.g. Île-de-France Mobilités) to confirm wins scale and no memory regression.

🤖 Generated with Claude Code