server2: overlap forecast partition lookups and cache Doc weights (#2849) by hwanjjang · Pull Request #2852 · WizardFactory/TodayWeather

hwanjjang · 2026-10-22T01:53:09Z

Speeds up an empty-cache domestic forecast build for #2849. Stacked on #2841 (D2, fix/domestic-warm-cache-2819 at 307defeb).

Changes

Ordered concurrent partition lookups (forecast_cache.rs, records.rs): up to 12 lookups in flight (READ_AHEAD, at most the process S3 gate), consumed in partition order. Docs, order, exclusion counts and the first error equal sequential reading. Scope caps are applied per consumed partition (ForecastScope::admit).
Forecast store limits (mod.rs): 12 lookups and 12 raw loads, 4 MiB output and 1 MiB retained controls per lookup. Process-wide in-flight bound 72.75 MiB (D2: 82.125 MiB).
Cached Doc weights (doc.rs, grid.rs): each Doc caches its exact heap weight, so full Doc walks grow linearly (256 stores: 768 walks, D2 67,840). Each store still sums the cached weights of the retained Docs. Grid rows are compared without copying text.
D2 owner hand-off fix (forecast_cache.rs): Workspace returns its 192 MiB lease before the owner permit wakes the next owner. Before this fix the next owner could fail with forecast cache: raw storage: Capacity, the message of the four domestic failures in server2: reuse complete domestic forecast views within bounded memory #2841 CI.
Measurement harness (peer latency control, aged mode, tools/cold_build_matrix.py), architecture/operations docs, the D2 card and the cold-read sequence diagram.
tests/runtime_services/inventory.rs ported unchanged from Retain domestic forecast views across writes and restarts (#2850) #2851 (0c163912, test only). The D2 prose test fails on 307defeb, and CI reached it once the domestic tests passed.

Evidence

Local (macOS arm64, loopback peer, modeled latency):

13/13 golden Docs and the reference HTTP body a6462aba…. Fold time: Short −41 %, Shortest −29 %.
Empty-cache request at 0 ms: fresh process 1.78 → 1.10 s, running-server rebuild 1.52 → 1.03 s. At 10 ms the views are ready after 3.1 s (running server) and 4.7 s (fresh process).

Seoul t3.small (SM3; loopback peer on the same two vCPUs):

No binary answered inside 3 s at 0/10/20 ms.
Running-server views ready: candidate 4.6 / 5.3 / 6.4 s; D2 6.7 / 8.2 / 8.3 s with few views mapped.
The 0 ms floor (4.6 s) points to CPU, not S3 concurrency. A 24-permit gate variant (not committed) only removes part of the latency share (10 ms: 4.8 s).

The 2.5 s p95 goal is not met locally at 10 ms or on t3.small.

Selected evidence

Checks: CI on 7eadefb6 all green. Local per-target workspace run (see evidence). Cross-provider review (OpenAI Codex gpt-6.1-sol): rounds 1–3 findings fixed; see the comments.

Pre-merge only: no merge or deployment authority. Merging #2841 first is a prerequisite. #2851 also changes forecast_cache.rs on the same base, so the two PRs will conflict.

🤖 Generated with Claude Code

Prepare the empty-cache forecast build work: SDLC intent, spec and plan, the cold-read sequence diagram, the ColdBuildSpeed2849 placement declaration and the measurement harness. - The owned loopback S3 peer accepts a latency control that delays every GET, HEAD and LIST by the set milliseconds (PUT is not delayed); its self-check covers the control. - The D2 smoke example uses the server's 12-permit S3 gate (was 4) and gains an aged mode that repeats the cold request after the 30-second view freshness, rebuilding the forecast views in a warm process. - cold_build_matrix.py runs two or more smoke binaries at modeled S3 delays with one seeded corpus and reports status, time, body identity and forecast reads; pure tests cover its report logic. No product code changes in this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An empty-cache view build read its partitions one at a time through a forecast store that admitted two lookups, so one request chained about 700 catalog, descriptor and raw reads and grew by about 2 s per millisecond of S3 latency. - read_partitions keeps up to READ_AHEAD (12) lookups in flight with an ordered buffered stream and consumes them in partition order, so the retained acquisitions, the failing scope partition and the first error equal a sequential read; returning early drops the rest. - forecast_lookup returns each partition's acquisitions, excluded groups and scope totals (every S06 acquisition counted); ForecastScope::admit applies the unchanged caps per consumed partition. - The forecast store admits 12 lookups and 12 raw loads (a raw load beyond its permits is refused, not queued) with 4 MiB of records and 1 MiB of controls per lookup: 12 x (4 + 1 + 1.0625) MiB in flight over all views, below D2's 2 x (32 + 8 + 1.0625) MiB. Tests: overlapped reads inside the 12-permit gate (130 partitions at 10 ms: peak 12 lookups, 0.55 s versus 3.9 s sequential), jitter equality, first error in partition order, vanished catalog, owner deadline and lookup timeout cancellation, two cold keys with another reader, the 4 MiB boundary, store limits and scope-cap equivalence with the per-acquisition checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The fold re-weighed the whole retained collection on every store and the whole Doc on every bounded field set, so storing n Docs walked about n^2 Docs (256 Docs: 67,840 walks). - A Doc keeps its heap_weight in a Weight cell (0 = unknown; a real weight is at least 64). try_set stores the exact new weight from the stored key, value and Vec capacity after its unchanged projected check; set, remove and clones start unknown; a failed push leaves it unknown. Weight compares equal and Debug prints the previous form. - checked_store_with_limit and every caller now read cached weights: storing 256 Docs walks 768 Docs, 32 field sets walk once. - The logical weight of the same data grows by 8 bytes per Doc and per stored field (Doc carries the cell); limits and their arithmetic are unchanged. Tests: cached weights equal an independent full walk after random sets, refused sets, removes and clones; collection admission equals a full recomputation at limit - 1, limit and limit + 1; a walk counter shows linear stores and field sets (Red on the base: 67,840 walks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Workspace declared its owner permit before its 192 MiB byte lease, so a retiring owner released the slot first. Tokio wakes the next owner inside that release and ByteBudget::reserve refuses rather than waits, so an owner woken on another worker could fail with Capacity while the old lease was still counted ("forecast cache: raw storage: Capacity", the four failures of base CI run 39243883911 and one local full-suite failure that passed alone). Declare the lease first: struct fields drop in declaration order. No size, owner, queue or waiter limit changes. Test: a waker registered by the next owner records the workspace in use at its wake-up; on the old order it saw 201,326,592 bytes, now 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A profile of the empty-cache build after the previous changes put about 200 of 1,150 build_view samples in Row::get string copies, their allocation and frees (find_or_create per item, complete per expected time and row), and 45 in js_number of the grid coordinate per item. - Row::str_ref borrows the string Row::get reads (own value, or the template string unless it needs the value layer's canonical form). - find_or_create and valid compare date and time through it; complete compares a row's date + time with the expected stamp without building it; row_at stays where a set needs the owned stamp. - raw_items_checked formats the coordinate once per acquisition and compares the publication without a per-item string. Tests: rows found exactly as through Row::get (own, template, numeric and stand-in values; written before the change); str_ref and the borrowed stamp comparison equal the copied reads. The isolated smoke keeps 13/13 golden Docs and the reference HTTP body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A cold or aged request cut by the 3-second native deadline leaves its forecast owners building until their 8-second bound, so a 502 alone does not show how far the build was from fitting. - The smoke example waits, outside the request timing, until the owners started by a cold or aged request are idle and records views_ready_ms; the S3 counters then include the whole build. - cold_build_matrix.py reports the ready time (median, p95, runs that were not ready after 10 s) and the mapped views per cell. Test: summary of ready times and not-ready runs (written first). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

At 10 and 20 ms both builds missed the 3-second deadline on the cold request, and the example stopped the aged run there, so the rebuild in a warm process (forecast views stale after 30 seconds) was never measured. That rebuild is the common empty-cache case: about 2,230 forecast S3 operations and about 20 others, against about 1,400 other operations in a fresh process. - An aged run still makes its rebuild request after a cut cold one; the run stays failed and the matrix accepts that order only there. - The matrix counts unlabelled S3 operations and returned bytes separately (it had summed both into one operation count) and reports their median per cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The first complete local matrix ran every cell but lost its report: the final latency reset failed with "Can't assign requested address" (local ports exhausted at that moment) and the tool exited before writing matrix.json. The reset was unnecessary because the peer stops right after. An exception inside the run loop now ends the loop, and the report keeps every finished run with a "stopped early" problem (exit 1). The reset is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Architecture D2 section: ordered read-ahead of 12 partition lookups, forecast store admission 12/12 with 4 MiB records and 1 MiB controls per lookup (72.75 MiB in flight over all views, D2 82.125 MiB), linear Doc accounting, the owner hand-off order, local evidence and the remaining gap to the 2.5-second goal. - Operator guide: new limits, the hand-off note, the base-versus- candidate latency matrix procedure; D2 run figures labelled as such. The native verification guide links the procedure. - Architecture diagram: D2 card and forecast sources at 8946cec; the cold-read sequence is bound to the implementation (Archify finalize and visual check pass for both). - Intent rev 3, spec rev 5 and plan carry the dated amendments (process- wide memory bound, owner hand-off scope); selected evidence under docs/evidence/tasks/issue-2849-cold-build-speed. No code change: server2/ is unchanged by this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A cut cold request excused any later exit, so a child that hung after a 502, or an aged run without its rebuild, passed as a measurement. Timeouts and signals are now always problems and the aged mode must record both phases. Review finding CX-2852-001. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Full Doc walks grow linearly with stores, but each store still reads every retained Doc's cached weight. The architecture text and the cold-read diagram said the accounting was linear. Review finding CX-2852-002; diagram regenerated with Archify 3.2.0 (all gates pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

runtime_services::inventory::factory_national_prose_reader_survives_factory_reconstruction fails on the D2 head 307defe and passes on its merge base. Once the owner hand-off fix let the domestic tests pass, CI on this stack reached that test and failed there. The test file is taken unchanged from 0c16391 (PR #2851, commits 24fb0fb and 0c16391; test only): each reconstruction phase prepares its weather once and ends detached owners and pools at its runtime boundary. Locally runtime_services passes 48/48 on this branch with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hwanjjang · 2026-10-22T02:18:07Z

Review round 1 (OpenAI Codex gpt-6.1-sol, medium, at af08acc4): CHANGES_REQUESTED

Required, CX-2852-001 server2/tools/cold_build_matrix.py: after a cut cold request the tool accepted a hung child (TIMEOUT) or an aged run without its rebuild as a measurement. Fixed in 207b176d: timeouts and signals are always problems, and aged runs must record both phases. 4 regression cases fail before the fix and pass after.
Recommended, CX-2852-002 docs/architecture/server2.md: "linear accounting" overstated the change, because each store still sums every retained Doc's cached weight. Applied in b7aaf689 (text and cold-read diagram; Archify gates pass).

CI at af08acc4: domestic passed 181/0, so the four Capacity failures from #2841 are gone. runtime_services then failed on the pre-existing D2 test factory_national_prose_reader_survives_factory_reconstruction. 7eadefb6 ports that test fixture unchanged from #2851 (test only, declared); locally runtime_services passes 48/0.

hwanjjang · 2026-10-22T03:00:28Z

Review round 2 (same reviewer thread, at 7eadefb6): PASS, no new findings. CX-2852-001 and CX-2852-002 are confirmed resolved; the ported inventory.rs is byte-identical to 0c163912; server2/src is unchanged since af08acc4. CI at 7eadefb6: all checks pass. The Seoul host evidence (SM3) is still running and will get its own review round.

On one Spot t3.small (2 vCPUs) with the loopback peer, no binary answered an empty-cache request inside 3 s at 0, 10 or 20 ms. A running server's views are ready after 4.6 / 5.3 / 6.4 s with the candidate and 6.7 / 8.2 / 8.3 s with D2, whose owners stop at 8 s with few views. The 0 ms floor is parse and fold CPU; a 24-permit gate variant cut the 10/20 ms times to 4.8 / 5.3 s without lowering it. Also documents the matrix tool's stricter problem rule (CX-2852-001) and the test fixture ported from #2851 in the plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Only the Linux candidate and gate24 builds returned the reference body during qualification; the D2 build did not. D2's owners did not all stop at the 8 s bound: at 0 ms two rebuilds failed early without a logged cause. The 4.6 s at 0 ms is a floor of a setup in which the loopback peer shares the two vCPUs, not a measured parse and fold time. Review findings CX-2852-003, -004 and -005. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hwanjjang · 2026-10-22T04:00:05Z

Seoul t3.small matrix (SM3) at 982055d9. One Spot t3.small (2 vCPUs, role-less, operator-only SSH, no egress), 02:08–03:51 UTC, then terminated; the security group and volume deletions were read back. Estimated cost about USD 0.10. Loopback peer on the same host, 20 fresh-process and 10 running-server runs per cell, no harness problem.

No binary answered inside 3 s at 0/10/20 ms (0/270 cold, 0/90 aged).
Running-server views ready: candidate 4.6 / 5.3 / 6.4 s; D2 6.7 / 8.2 / 8.3 s with a median of only 3.5 / 1 / 0.5 of 7 views. The uncommitted 24-permit gate variant took 4.6 / 4.8 / 5.3 s.
Even at 0 ms the floor is 4.6 s with the server, the peer and loopback HTTP sharing two vCPUs. That points to CPU, not S3 concurrency; parse/fold time alone was not separated on the host. Max RSS about 88 MiB for all three.

Review round 3 (Codex, at 982055d9): CHANGES_REQUESTED. All three findings are addressed in af1caa4a:

Required, CX-2852-003: qualification verified the body of the Linux candidate and gate24 builds only, not D2. Fixed (claim narrowed).
Required, CX-2852-004: not all D2 times came from the 8 s owner bound; two 0 ms D2 rebuilds failed early (2.2 / 2.4 s) and the log records no cause. Fixed (described as consistent with the hand-off defect, not confirmed).
Recommended, CX-2852-005: "fold CPU floor" was stronger than the data. Applied (shared host CPU; parse/fold not isolated).

The fresh-process and running-server D2 operation counts were given as one series, D2's 10 ms owners were said to reach the 8 s bound although two stayed active until 9.46 s, and the one local 10 ms 200 took a client-measured 3.003 s, not under 3 s. Additional review finding GK-2852-001 and reviewer notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hwanjjang · 2026-10-22T06:13:28Z

Readiness at bd7122ae (pre-merge; no merge, auto-merge or deployment authority)

Required review (OpenAI Codex gpt-6.1-sol, medium): round 5 PASS. All findings CX-2852-001 to 005 are resolved.
Additional reviews: Gemini (gemini-3.8-flash-high) PASS. Grok (grok-4.7, high) PASS with one Recommended finding, GK-2852-001 (the D2 operation series), applied in bd7122ae and confirmed in Codex round 5. Its notes on the 9.46 s D2 owners and the 3.003 s local 200 are also corrected.
CI: every check SUCCESS. The first foundation attempt at bd7122ae failed on the ported D2 prose test (/v000903/nation/KR returned 500 four times in a row). The rerun passed, and the same server2 tree passed at 7eadefb6 and af1caa4a, so the test is flaky. It belongs to server2: reuse complete domestic forecast views within bounded memory #2841/Retain domestic forecast views across writes and restarts (#2850) #2851.
Merge order: server2: reuse complete domestic forecast views within bounded memory #2841 first. Retain domestic forecast views across writes and restarts (#2850) #2851 conflicts with this PR in forecast_cache.rs, records.rs, the smoke example and the operator guide.
Not met: the 2.5 s p95 goal of server2: review ways to speed up one domestic weather request on an empty forecast cache #2849, locally at 10 ms and on t3.small.

hwanjjang and others added 13 commits October 22, 2026 05:49

Point the plan at the implementation-bound design (#2849)

af08acc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hwanjjang and others added 2 commits October 22, 2026 12:54

hwanjjang changed the title ~~server2: overlap forecast partition lookups and make fold accounting linear (#2849)~~ server2: overlap forecast partition lookups and cache Doc weights (#2849) Oct 22, 2026

hwanjjang marked this pull request as ready for review October 22, 2026 06:13

hwanjjang mentioned this pull request Oct 22, 2026

server2: review ways to speed up one domestic weather request on an empty forecast cache #2849

Open

4 tasks

Navigation Menu

Repository navigation

server2: overlap forecast partition lookups and cache Doc weights (#2849) - #2852

server2: overlap forecast partition lookups and cache Doc weights (#2849)#2852
hwanjjang wants to merge 16 commits into
fix/domestic-warm-cache-2819from
fix/cold-build-speed-2849

hwanjjang commented Oct 22, 2026 •

edited

Loading

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hwanjjang commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Evidence

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

hwanjjang commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hwanjjang commented Oct 22, 2026 •

edited

Loading