perf(shared-fs): add verified lazy range reads - #321
peerbit-org wants to merge 5 commits into
Conversation
|
Windows tarball validation caught an OS-dependent package-size failure: 2,763,849 bytes versus the existing 2,750,000-byte budget. The same commit packed to 2,737,006 bytes on Darwin/Linux because tracked sources and TypeScript output used different newline policies on Windows. Commit c7f1fc6 fixes the cause rather than raising the budget or stripping public declaration documentation:
The refreshed three-OS tarball matrix will verify the Windows result. No size budget was increased. |
|
Follow-up from the first normalized checkout: the size variance was gone, but the package-content guard correctly found that the package LICENSE was LF while the repository-root LICENSE remained CRLF on Windows. Commit |
|
Latest rebased head 427e1b3 passed Ubuntu/macOS portable tests, all native checks, package installs, and cross-OS interop. Windows completed 491 tests with 11 skipped; the sole failure remained persistent-multi-writer.test.ts. Raw failure sequence:
This is the same downstream lifecycle/readiness test blocker already exposed on #320. I am leaving the PR red—no blind retry, timeout inflation, or weakened assertion. The intended path is to refresh #300 onto the coherent release from upstream peerbit#1426, land the public readiness waiter update after a first-attempt three-OS pass, then refresh this PR. |
Summary
Compatibility and integrity
The marker is the existing content-addressed zero-byte chunk. Concatenating it changes neither file bytes nor the whole-file hash, so older readers can read new versions unchanged. Marked empty files use two references to the same zero-byte chunk, which distinguishes them from legacy empty versions without a schema change.
The range capability is advertised only by the built-in full-replica implementation. It validates the admitted signed
FileVersion, exact node/version/size/hash binding, canonical ordered layout, marker, each fetched chunk hash, and each positional chunk length. A writable upgrade materializes and whole-file-verifies once;O_TRUNCskips base materialization. Version and chunk leases make local/remote CUT recovery and GC retain data until all reads and lifecycle drains settle.Performance evidence
Manual process-isolated benchmark on macOS 26.6.2, Apple M3 Pro arm64, Node 24.13.1:
For the 64 MiB case, open fetched only the zero-byte marker; sixteen random 4 KiB reads fetched sixteen 512 KiB chunks. Warm-cache reads fetched no additional chunks.
These are algorithmic/process-local measurements, not physical cold-disk or network claims: each worker had just authored its file. A 4 KiB cold read still fetches a 512 KiB chunk, or 128x read amplification. Smaller/adaptive extents remain future work.
Validation
git diff --checkRelease
Includes minor changesets for
@peerbit/shared-fsand@peerbit/shared-fs-cli.This PR is independent of the experimental native IPC-width work in #320 and the upstream-cohort/readiness work in #300.