server2: retain domestic forecast views until invalid (per-key invalidation, boundary expiry, SSD tier) #2850

Parent: #2614 · Follows: #2819 D2 (PR #2841) · Supports: #2705 · Related: #2849, #2840 · Dependencies: PR #2841 merged (or stack on fix/domestic-warm-cache-2819)

Problem / Goal

D2 discards a built forecast view after at most 30 s, only within the wall-clock minute that built it, and drops every town's view whenever any forecast acquisition starts or ends. An expired view is rebuilt in full (8.0–9.1 s on t3.small, see #2849), and the memory budget holds about three towns. A request for a town that was not requested moments ago therefore probably rebuilds; this is inferred from code, not measured. Keep a built view until it is actually invalid, check external changes cheaply, and let a disposable SSD tier hold the views memory cannot, without changing responses or making local state authoritative. #2849 makes one uncached build faster; this issue makes uncached builds rarer.

Context

Code at PR #2841 head 307defeb, server2/src/domestic/kma/source/forecast_cache.rs (a code reading, not a measurement):

• Time limit. View::fresh (:150) accepts a view for at most 30 s after admission and only in the wall-clock minute that built it. The 30 s is a D2 design default ("Root accepts maximum 30s external-write/new-storage-fault observation delay", intent/issue-2819-domestic-warm-cache.md), not a recorded AK decision. The spec lists publication, retention, month and day boundaries as the expiry points; the code uses the simpler minute rule.
• Global invalidation. Writer::acquire (:874–896) and Writing::drop (:861–869) bump one epoch and call map.clear() at the start and end of any forecast acquisition for any grid, RSS or mid key. While one runs, Cache::get (:312) skips the map and build (:471–477) does not install a view. Review finding DC-CLAUDE-003 recorded these "multi-town/global-generation limits" and kept them as bounded local preparation.
• No renewal. The D2 spec says "Expired or failed refresh never serves/renews the old view", so an unchanged view is rebuilt in full. S07 responses also never extend their TTL.
• Budget. 128 MiB / 128 entries (:247–253). The 13 views of one town weigh 35.7 MiB logical, so about three towns fit. Logical weight is not serialized size.
• SSD. server2/src/cache/disk.rs (S15b-4c) is a disposable, hash-verified, LRU-bounded cache (default 8 GiB), off unless SERVER2_LOCAL_CACHE_DIR is set. It holds immutable raw objects only; docs/operations/server2-infrastructure.md leaves "decoded caching or O01 packs" to S19/S20 evidence. The D2 smoke example does not enable it. Raw objects from SSD would not remove the parse and fold CPU that dominates a rebuild.
• What a view depends on. Raw acquisitions are immutable, and an acquisition request names the grid and family (or mid region) it writes. A view depends on now only through step functions with computable change points: acquisitions becoming visible, the oldest partition, the latest publication and retention anchor, the RSS run time, and the month.

Constraints:

• Known changes invalidate immediately; external changes appear after a bounded delay (S07 rule, plans/issue-2692.md). Keep a bounded delay, but make checking it cheap.
• S3 stays authoritative. The local disk cache is bounded, disposable and hash-verified; a missing or corrupt file is a normal miss (plan amendment). A view on disk is a cache validated against S3 identities, not stored truth (server2 O01: conditionally reduce raw GET fan-out with exact-raw packs #2707: "Never persist normalized weather").
• Responses stay byte-identical (13 view goldens; HTTP reference sha256 a6462aba…). No runtime verification mode or flag. No legacy dependency. One API process first.

Required changes

1. Per-key invalidation. An acquisition invalidates only the view keys it can change. Other keys keep serving from memory while it runs. Keep the active-writer and epoch fences for the affected key, cancellation and panic safety, the own-write overlay and the original S16 identity.
2. Boundary expiry. Replace the 30 s / same-minute rule with an expiry computed from the existing pure selectors: the earliest next publication, retention, oldest-partition, month or day boundary, or future-fetched time. A wall-clock rewind still misses.
3. Cheap external-change check. Keep an explicit bounded delay for changes this process cannot see. At that point, compare the view's canonical acquisition identities with S3 using listings or catalogs only. If unchanged, renew the view without raw reads, parse or fold; if changed, rebuild, or fold only the newer acquisitions if server2: review ways to speed up one domestic weather request on an empty forecast cache #2849 shows that is safe. Renewal departs from the D2 spec and from S07's "TTL never extends"; record the reason, the delay value and the S3 request cost in the spec.
4. SSD tier for built views. A view evicted from memory, and views at process start, can be restored from a bounded local directory. Store the folded result with its canonical acquisition identities and parser/schema revision; verify the hash; accept it only if the identities match the current S3 listing. Missing, corrupt, stale or over-cap files fall back to a rebuild from S3. Reuse the existing disk cache's directory, cap and ownership rules; add no setting that exists only for verification. Measure serialized size and restore time for the 13 views, and state how many towns memory and an 8 GiB cap hold.
5. Scope. Implementation under server2/ only; outside exceptions limited to maintained docs (docs/architecture/server2.md, docs/operations/server2-domestic-cache.md, Archify JSON). Declare exact paths before editing. Split into reviewable PRs in the order above if that is simpler.

Completion criteria

• Per-key isolation: with two towns, an acquisition for town A leaves town B's view served from memory with zero raw reads, parse and fold, while A's view is rebuilt afterwards; writer cancellation and panic leave no stale installed view.
• Boundary expiry: fixed-clock tests show a view reused across minute boundaries until exactly the next publication, retention, month or future-fetched boundary and rebuilt there; wall-clock rewind misses; all 13 view goldens match before and after.
• Cheap revalidation: when S3 content is unchanged, the end of the external-change delay renews the view with zero raw GET, parse and fold; after an external write of a newer acquisition, the first request after the delay includes it. The delay and the check's S3 request count are in the spec.
• SSD restore: with a memory budget smaller than the working set, an evicted view, and a view at process restart, is restored from disk with zero raw GET, parse and fold and a byte-identical HTTP body. Corrupt, truncated, wrong-identity and over-cap files fall back to an S3 rebuild with identical bytes; disk use never exceeds the configured cap. The evidence records serialized size, restore time and towns held.
• Contract and docs: the D2 section of docs/architecture/server2.md, the operator guide and the Archify JSON describe the new retention rules; no runtime flag was added; changed paths match the placement declaration.

No merge, deployment, new cloud resource, real S3 call or provider call is authorized by this issue.