perf(shared-fs): bound exact-slot candidate queries and cache - #331
peerbit-org wants to merge 7 commits into
Conversation
|
Holding this PR without rerunning CI. Exact head |
|
Resource/performance audit: this remains on hold. Static review found no correctness blocker, and the 10k benchmark does support width-independent warm candidate selection. It does not yet establish production safety for a large namespace. Before merge, please refresh onto current master and provide one matched, fresh-process 100k unique-name benchmark with:
The current design retains a reverse-map entry and placement object per cached naming row. A single huge directory is not bounded by the parent-count limit, and the global The old-base package delta is modest (+25,512 unpacked bytes, +4,768 compressed), so runtime memory and cold-fill behavior—not package size—are the gate. No CI rerun requested. |
6ce570e to
5479d69
Compare
100k resource campaign and bounded bulk-build fixRefreshed this branch onto master The campaign found an O(N²) same-name bulk constructor: 2,000 rows incurred 4,006,000 identity reads. A 100,000-row same-name campaign exceeded its approximately three-minute budget; that is a censored timeout, not a measured install duration. The fixed bulk constructor takes 119–145 ms in the three observed same-name samples. Existing dynamic upsert/removal semantics remain unchanged. Independent review compared the actual old/new implementations over 10,000 randomized histories (2,513,641 rows), plus 50,000 randomized install/delete/eviction operations including cross-parent moves. Ordered cache rows, singleton/history representation, reverse placements, and eviction epochs matched. The remaining limits are material:
The report and raw JSON fixtures are in Local validation: 13 focused cache/bulk/race tests, both fresh-process smoke cases, build/lint/format, package checks, and the two full 100k campaigns passed without retries. Library package remains 83 files / 2,219,037 unpacked bytes; raw fixtures and benchmarks are excluded. Keep this PR held. Next gate is a bounded exact-slot query/cache and query-materialization policy that preserves same-ID replacement safety across names and parents, followed by this same campaign and filesystem-level validation on the next upstream cohort. This update does not unblock release #324. |
|
Exact head Ubuntu, macOS and Windows each passed 525 library tests and 33 CLI tests. Durable disposal passed 22/22 and persistent multiwriter 1/1 on the first attempt on every OS. Raw logs have no retry markers, failed tests, receipt-session failures or replicator timeouts. Native, interop, package installs, formatting, changesets and deployment validation are also green. Still held: this verifies cross-OS correctness checks on this branch's locked Peerbit 5.3.35 cohort, not the 100k cold-query/memory gate or the held 5.4.0 confirmation/session issue. Keep the bounded exact-slot/materialization work and the existing release hold; no merge or release performed. |
peerbit-org
changed the title
|
Additional public-API validation at exact head Real
The structural result is the useful gate: directory width did not increase examined or retained candidate rows for this path. Small-sample timing differences are descriptive, not evidence that wider directories are faster. This workload uses one in-process peer, the existing 5.3.35 cohort, and no mount or remote persisted receipts. It does not clear the full-suite retry, oversized-history/full-list, or latest-cohort holds. |
|
Strict CI audit for exact head Shared FS CI run 33954049406 completed successfully on workflow attempt 1, but workflow success is not a first-attempt test pass.
The macOS case was The naming-conflict/tombstone disposal case passed without retry in each CI job (Ubuntu 14.705 s, macOS 10.289 s, Windows 11.735 s); its separate earlier local full-suite retry remains recorded. No workflow rerun, timeout inflation, or retry-until-green was requested. All resource/history/latest-cohort gates in the PR body remain. A useful separate downstream cleanup is first-failure reporting for embedded retry-enabled tests, so future failures retain actionable phase/stack evidence. Final overall checks: Shared FS CI, cross-OS interop, Changesets and all three tarball installs passed. Hosting validation failed during dependency installation, before a shared-fs build/test result: |
|
Downstream follow-through; this PR remains held at exact head 9cde976.
The next small cleanup candidate is unused recursive DAG depth computation: no consumer reads it, while winner ordering uses stored causalDepth. It needs head/winner invariance tests and measurements, not a larger cache. No new production optimization is part of the experiment branch. |
Review against the new
|
|
Superseded by #359. It keeps this PR's exact-slot queries and bounded point cache and addresses the review above: it leaves the per-directory listing cache unchanged, checks capacity before evicting, keeps #303's test working, and drops the fixtures. It also adds FIFO admission and a width gate, so filesystems without wide directories never create the extra SQLite indexes (the owner's decision). The branch is kept. |
Path resolution, stat and create checks used to read the whole parent directory (a sweep) to find one name. In a wide directory that has not been listed, a cold lookup therefore cost O(width). Resolve a single (parentId, name) slot in this order: 1. a cached directory sweep (slotSweepCache, unchanged from master), which also proves absence with zero queries; 2. a separate bounded point cache of exact slot histories, including cached negatives (at most 4,096 slots and parent markers, 16,384 rows, about 8 MiB estimated, 64 distinct in-flight queries); 3. one exact kind+parentId+name index query, installed with the same fences the sweep fill uses: open generation, cache identity and the per-directory slot:<parentId> epoch. No global mutation epoch. The point cache decides admission before it mutates anything: a history too large to retain is returned whole, not cached, and evicts nothing. Change events keep cached slots exact, relocating same-id replacements by evicting the source slot. Directory listings and slotSweepCache are not changed. open(), retireOverlay() and close() replace both caches. Based on the point tier of #331 without its complete-parent tracking.
Env-gated (PEERBIT_SHARED_FS_SLOT_CACHE_BENCH=1); skipped in normal runs. Adapted from #331's slot-candidate bench: an end-to-end case over writeBatch-built directories and a seeded 100k-row SQLite index case. It only uses fields present on master too, so the same file measures a baseline. Gates are structural (query kinds and counts); timings are descriptive.
…359) Lookups in unlisted directories read them with a bounded query (at most 2,049 rows, same index as the listing sweep) and cache narrow ones whole as before. Directories found or listed wider than 2,048 naming rows use exact (parentId, name) queries with a bounded point cache. Filesystems without wide directories never create the slot indexes. Point-tier waiters are admitted per key in FIFO order; sweep fills are fenced by cache identity. Cold hit at 100k rows: 1,147 ms -> 337 ms; later cold miss 913 ms -> 0.23 ms; bulk creates at parity. Supersedes #331.
1 participant
Summary
Bound exact-name candidate caching and query the exact naming slot on a cold miss, preserving complete raw histories and the existing naming winner/conflict/tombstone semantics.
Matched 100k evidence
The identical v2 worker ran in four fresh processes against legacy production
5479d691and bounded production4529da1c, using the same executed Peerbit 5.3.35 / Documents 15.0.16 / shared-log 16.0.15 cohort. No install or dedupe. These are real disk-backed SQLite candidate-query measurements, not public file writes, mounted throughput, cold disk, cold join, or the held 5.4.0 rebaseline.The first query includes lazy SQL index/planner creation. Actual prepared SQL/bindings and EXPLAIN confirm composite kind/parentId/name access. Each lookup's exact candidate identity set is checked outside timing; raw reports include source/lock/module hashes and GC/memory checkpoints.
Tradeoffs: A 100k-row single-name naming history exceeds the budget and repeats at roughly 985 ms instead of a cached 0.102 ms. This fixture has 100k distinct node claimants, not 100k content edits to one file. A full sweep after partial warming reads all 100,100 rows (1.46 s for unique), while legacy already holds that parent in memory. New unseen-name creates perform point absence queries (0.156 ms median versus legacy's 0.067 ms after its first full sweep). Full enumeration, oversized-history transient memory, and incoming caller backlog are not bounded by the retained-cache limits. The same-name campaign still sampled 1,181 MiB heap before GC. Do not interpret this as an across-the-board speedup.
See SLOT_CACHE_RESOURCE.md for historical v1 evidence, matched v2 raw fixtures, reproduction, exact limitations, and package sizes.
Validation and hold
Keep this PR held. Fresh cross-platform first-attempt validation, the coherent upstream rebaseline, public namespace/mounted workload coverage, and history/enumeration materialization tradeoffs remain gates. No release is authorized by these microbenchmarks alone.