server2: overlap forecast partition lookups and cache Doc weights (#2849)

docs/architecture/diagrams/server2-cold-read.sequence.json

@@ -0,0 +1,255 @@
+{
+  "schema_version": 1,
+  "diagram_type": "sequence",
+  "meta": {
+    "title": "Empty-cache forecast build: ordered concurrent partition lookups",
+    "output": "docs/architecture/diagrams/server2-cold-read.html",
+    "quality_profile": "showcase",
+    "locale": "en",
+    "viewBox": [
+      1040,
+      590
+    ],
+    "column_fit": "spread",
+    "repository": {
+      "url": "https://github.com/WizardFactory/TodayWeather",
+      "revision": "8946cecd899228b8585bfc2c16de3fd6f1b20321",
+      "link_mode": "local-only"
+    }
+  },
+  "participants": [
+    {
+      "id": "owner",
+      "type": "backend",
+      "label": "Cache owner",
+      "sublabel": "one flight per view",
+      "sources": [
+        {
+          "path": "server2/src/domestic/kma/source/forecast_cache.rs",
+          "line": 389,
+          "end_line": 486,
+          "label": "Owner admission, workspace and install"
+        }
+      ]
+    },
+    {
+      "id": "reader",
+      "type": "backend",
+      "label": "Partition reader",
+      "sublabel": "ordered consumer",
+      "sources": [
+        {
+          "path": "server2/src/domestic/kma/source/forecast_cache.rs",
+          "line": 487,
+          "end_line": 618,
+          "label": "Read and ordered partition read-ahead"
+        },
+        {
+          "path": "server2/src/domestic/kma/source/records.rs",
+          "line": 35,
+          "end_line": 95,
+          "label": "Scope totals admitted in partition order"
+        }
+      ]
+    },
+    {
+      "id": "store",
+      "type": "backend",
+      "label": "Forecast store",
+      "sublabel": "admission 12 lookups",
+      "sources": [
+        {
+          "path": "server2/src/domestic/kma/source/mod.rs",
+          "line": 114,
+          "end_line": 131,
+          "label": "Store admission 12 and lookup bounds"
+        },
+        {
+          "path": "server2/src/storage/catalog.rs",
+          "line": 684,
+          "end_line": 700,
+          "label": "Lookup chain"
+        }
+      ]
+    },
+    {
+      "id": "gate",
+      "type": "backend",
+      "label": "Process S3 gate",
+      "sublabel": "12 permits, unchanged",
+      "sources": [
+        {
+          "path": "server2/src/gateway/runtime.rs",
+          "line": 30,
+          "end_line": 32,
+          "label": "Existing combined in-flight limit"
+        }
+      ]
+    },
+    {
+      "id": "s3",
+      "type": "database",
+      "label": "S3 records",
+      "sublabel": "authoritative",
+      "sources": [
+        {
+          "path": "server2/src/storage/catalog.rs",
+          "line": 684,
+          "end_line": 700,
+          "label": "Catalog, descriptor and raw reads"
+        }
+      ]
+    },
+    {
+      "id": "fold",
+      "type": "backend",
+      "label": "Fold",
+      "sublabel": "blocking, cached Doc weights",
+      "sources": [
+        {
+          "path": "server2/src/domestic/kma/source/forecast_cache.rs",
+          "line": 619,
+          "end_line": 711,
+          "label": "build_view on the blocking pool"
+        },
+        {
+          "path": "server2/src/domestic/kma/source/doc.rs",
+          "line": 175,
+          "end_line": 300,
+          "label": "Exact cached Doc weight"
+        }
+      ]
+    }
+  ],
+  "segments": [
+    {
+      "from": 240,
+      "to": 410,
+      "label": "Overlapped reads, ordered results"
+    },
+    {
+      "from": 420,
+      "to": 475,
+      "label": "Fold"
+    }
+  ],
+  "messages": [
+    {
+      "id": "step-0",
+      "from": "owner",
+      "to": "reader",
+      "y": 170,
+      "label": "Build key (workspace held)",
+      "variant": "default"
+    },
+    {
+      "id": "step-1",
+      "from": "reader",
+      "to": "s3",
+      "y": 198,
+      "label": "LIST partitions by month",
+      "variant": "default"
+    },
+    {
+      "id": "step-2",
+      "from": "s3",
+      "to": "reader",
+      "y": 226,
+      "label": "Sorted partition ids",
+      "variant": "return"
+    },
+    {
+      "id": "step-3",
+      "from": "reader",
+      "to": "store",
+      "y": 254,
+      "label": "Start lookups 1..12 in order",
+      "variant": "emphasis"
+    },
+    {
+      "id": "step-4",
+      "from": "store",
+      "to": "gate",
+      "y": 282,
+      "label": "Per lookup: catalog, descriptor, raw",
+      "variant": "default"
+    },
+    {
+      "id": "step-5",
+      "from": "gate",
+      "to": "s3",
+      "y": 310,
+      "label": "At most 12 operations in flight",
+      "variant": "emphasis"
+    },
+    {
+      "id": "step-6",
+      "from": "s3",
+      "to": "store",
+      "y": 338,
+      "label": "Objects, decoded and verified",
+      "variant": "return"
+    },
+    {
+      "id": "step-7",
+      "from": "store",
+      "to": "reader",
+      "y": 366,
+      "label": "Lookup n ready (any finish order)",
+      "variant": "return"
+    },
+    {
+      "id": "step-8",
+      "from": "reader",
+      "to": "owner",
+      "y": 394,
+      "label": "Partition order; caps; first error wins",
+      "variant": "return"
+    },
+    {
+      "id": "step-9",
+      "from": "owner",
+      "to": "fold",
+      "y": 440,
+      "label": "build_view on blocking thread",
+      "variant": "emphasis"
+    },
+    {
+      "id": "step-10",
+      "from": "fold",
+      "to": "owner",
+      "y": 468,
+      "label": "Same Docs, cached Doc weights",
+      "variant": "return"
+    }
+  ],
+  "cards": [
+    {
+      "dot": "emerald",
+      "title": "Contract",
+      "items": [
+        "Lookups overlap; results are consumed in partition order, so Docs, order and errors equal the sequential read.",
+        "Store admission rises 2 to 12 (raw loads too); per-lookup output falls 32 to 4 MiB and controls 8 to 1 MiB: in-flight reads of all views stay within 72.75 MiB (D2: 82.125 MiB).",
+        "No new mode, timer, retry or provider call."
+      ]
+    },
+    {
+      "dot": "emerald",
+      "title": "Preserved",
+      "items": [
+        "Owner deadline, 3 second waiter, writer fences and install rules are unchanged; the next owner wakes after the workspace returns.",
+        "The 12-permit process S3 gate is shared with other families and is not resized here.",
+        "Every capacity cap keeps its exact boundary."
+      ]
+    },
+    {
+      "dot": "slate",
+      "title": "Loopback evidence",
+      "items": [
+        "130 partitions at 10 ms per operation: 0.55 s with 12 lookups in flight, 3.9 s sequential.",
+        "Local, modeled 10 ms: views ready after 3.1 s in a running server (D2: none within its 8 s owner bound); the 3 s deadline is still missed.",
+        "Seoul t3.small: views ready after 4.6 s even at 0 ms (D2: about 3 of 7 views); host CPU, shared with the loopback peer, sets the floor. Real S3 and TLS are not measured."
+      ]
+    }
+  ]
+}

docs/architecture/diagrams/server2.json

@@ -8,7 +8,7 @@
     "locale": "en",
     "repository": {
       "url": "https://github.com/WizardFactory/TodayWeather",
-      "revision": "f1a0e0940f14341b62138f8bf17f568f1c061cfb",
+      "revision": "8946cecd899228b8585bfc2c16de3fd6f1b20321",
       "provider": "github",
       "link_mode": "web"
     }
@@ -153,13 +153,13 @@
         {
           "path": "server2/src/domestic/kma/source/forecast_cache.rs",
           "line": 21,
-          "end_line": 321,
-          "label": "Forecast limits and fresh generation lookup"
+          "end_line": 618,
+          "label": "Limits, lookup and ordered partition read-ahead"
         },
         {
           "path": "server2/src/domestic/kma/source/forecast_cache.rs",
-          "line": 856,
-          "end_line": 901,
+          "line": 874,
+          "end_line": 921,
           "label": "Actual writer lifetime invalidates generation"
         },
         {
@@ -308,6 +308,7 @@
       "title": "D2 forecast memory",
       "items": [
         "Grid / RSS / Mid complete views; other owners still run",
+        "12 partition lookups read ahead, consumed in order",
         "128 MiB retained; 48 MiB per entry; logical, not RSS",
         "Epoch plus active writer fence; no stale generation install",
         "Local CPU / queue limits; global integration remains open"

docs/architecture/server2.md

@@ -4136,3 +4136,68 @@ per request; no valid-zone recovery is claimed. Cold observation stubs are denie
 These are local source/cache results, not complete API parity, host p95 or an
 eligible rollback release. See the [operator guide](../operations/server2-domestic-cache.md)
 and [selected verification](../evidence/tasks/issue-2819-domestic-warm-cache/verification.md).
+
+### D2 empty-cache reads (#2849)
+
+An empty forecast cache rebuilds a view from every partition in its lookback; the
+Seoul fixture town needs about 700 partition lookups (about 2,220 forecast GETs).
+D2 read them one at a time through a forecast store that admitted two lookups, so
+one request grew by about 2 s per millisecond of S3 latency. The
+[cold-read sequence](diagrams/server2-cold-read.html)
+([editable source](diagrams/server2-cold-read.sequence.json)) shows the change.
+
+- A view build keeps up to 12 partition lookups (`READ_AHEAD`) in flight and
+  consumes them in ascending partition order. Retained acquisitions, the scope
+  caps (1,024 partitions, 2,048 acquisitions, 8,192 pages, 1 MiB pages, 32 MiB raw)
+  and the first error therefore equal the sequential read; an error, a cap or the
+  owner deadline drops the lookups still in flight.
+- The forecast store admits 12 lookups and 12 raw loads (a raw load beyond its
+  permits is refused, not queued) and bounds each lookup to 4 MiB of records and
+  1 MiB of catalog and descriptor bytes. With one compressed body per load, its
+  in-flight bytes over all views are at most 12 × (4 + 1 + 1.0625) MiB = 72.75 MiB,
+  below D2's 2 × (32 + 8 + 1.0625) MiB = 82.125 MiB; one view may now hold all of
+  it (D2: 41.06 MiB). A partition above 4 MiB is a `Capacity` error; fixture
+  partitions decode to at most about 0.2 MiB. The 12-permit process S3 gate is
+  unchanged and shared, so a cold build can occupy it while it reads.
+- A Doc keeps its exact weight once known (bounded field sets maintain it; other
+  changes and clones start unknown), so a store sums one cached weight per
+  retained Doc instead of walking every Doc's fields: full Doc walks now grow
+  linearly with the stores (768 for 256 stores; D2 67,840). The per-store sum
+  still reads every retained Doc's cached weight. The same data weighs 8 bytes
+  more per Doc and per stored field. Grid row reads compare date and time
+  without copying.
+- A retiring owner returns its 192 MiB workspace before its slot wakes the next
+  owner. D2 released the slot first, and an owner woken on another worker could
+  fail with `forecast cache: raw storage: Capacity` under load.
+
+Owner, queue, waiter, workspace, freshness and lookback limits are unchanged. Local
+evidence (macOS, owned loopback peer with modeled per-operation latency): 130
+partitions at 10 ms take 0.55 s with 12 lookups in flight against a 3.9 s
+sequential estimate; the fold of `short` falls from 1,171 to 687 ms and `shortest`
+from 418 to 297 ms with all 13 golden Docs unchanged. An empty-cache request at 0 ms
+takes 1.10 s instead of 1.78 s in a fresh process and 1.03 s instead of 1.52 s in a
+running server whose views went stale. At a modeled 10 ms the candidate's views are
+ready after 3.1 s in a running server (one of five such requests answered 200, at
+a client-measured 3.003 s) and after 4.7 s in a fresh process, while D2's owners stop at their 8-second
+bound with at most one of seven views; at 20 ms the running-server rebuild needs
+5.4 s. Through
+the 12-permit process gate, a fresh request's 3,628 S3 operations cannot finish in
+under about 3.0 s at 10 ms, and a running-server rebuild's 2,245 in under about
+1.9 s, so the 2.5-second goal needs fewer S3 operations per view or more S3
+concurrency, both separate decisions.
+
+On the Seoul t3.small (2 vCPUs, Unlimited; same loopback peer on the same host, so
+the peer shares the CPUs) no binary answered an empty-cache request inside 3 s at
+0, 10 or 20 ms, in 20 fresh-process and 10 running-server runs per cell. Views
+were ready (owners idle) in a running server after 4.6 / 5.3 / 6.4 s with the
+candidate and 6.7 / 8.2 / 8.3 s with D2, whose owners mapped a median of only 3.5,
+1 and 0.5 of 7 views (two of its 0 ms rebuilds failed early without a logged
+cause); in a fresh process after 5.7 / 6.6 / 8.7 s and 7.7 / 8.2 / 8.4 s. Even at
+0 ms the candidate needs 4.6 s with the server, peer and loopback HTTP on the same
+two vCPUs, which points to CPU rather than S3 concurrency; parse and fold alone were
+not timed on the host. An uncommitted variant with a 24-permit gate and
+read-ahead measured 4.6 / 4.8 / 5.3 s; the largest resident set of any process
+stayed about 88 MiB for all three. These are loopback measurements, not real
+S3/TLS, p95 capacity or production results. See the
+[operator guide](../operations/server2-domestic-cache.md#measure-empty-cache-builds-at-modeled-s3-latency)
+and [selected verification](../evidence/tasks/issue-2849-cold-build-speed/verification.md).