gc(tenuring): the survival-rate lock latches S=1 only on steady-state evidence, never from the startup cohort

cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md

@@ -0,0 +1,152 @@
+# Tenuring steady-evidence lock report
+
+Code SHA: `d229d7d715cee4725776a83fa210ac5727ce1862`
+
+Branch: `perf/tenuring-evidence-lock`
+
+## Lock map before and after this change
+
+The copying collector builds the cohort-scoped signal in
+`crates/perry-runtime/src/gc/copying.rs:621-640`: `eden_copied_bytes` counts
+fresh Eden objects copied into the survivor space, while
+`survivor_first_round_live_bytes` counts age-1 objects from that same cohort
+which return alive on the next copying minor. The completed minor passes those
+values to `retune_after_scavenge` at `copying.rs:1895-1909`.
+
+`crates/perry-runtime/src/gc/tenuring.rs:600-639` swaps the current
+`eden_copied_bytes` into `PREV_COPIED_BYTES`. A rated round is, as before, a
+cycle with `prev_cohort_copied > 0`: a previous copying minor admitted a fresh
+cohort, so this minor can divide its `first_round_live_bytes` by that cohort's
+intake. Before this change that single condition was also sufficient to make
+the first startup cohort eligible for the lock. That is why TN4's startup
+cohort qualified: the first minor copied it, the second minor returned it
+nearly intact, and the code had no process-phase evidence.
+
+The entry decision is now at `tenuring.rs:676-694`. A rated round contributes
+only when it is post-startup, its previous fresh intake is at least the existing
+`desired / 4` substantial-volume bar, and its survival is at least the existing
+90% bar. Any rated round that fails those conditions resets
+`PROMOTE_LOCK_STREAK`. `PROMOTE_LOCK` latches only when that streak reaches
+three.
+
+Startup is the first two **rated survivor cohorts**
+(`tenuring.rs:195-205, 613-627`), not a wall-clock duration and not the
+allocation-census flag. This uses the lock's own threshold-invariant evidence
+and cannot re-pace collection. The allocation census is deliberately not the
+marker: it seeds halfway to the first nursery cap, before the first copying
+minor, so it was already true for precisely the TN4 startup cohort that must be
+excluded. Two excluded ratings cover the observed two-minor startup phase; K=3
+is the smallest steady window that rejects a one-off or two-cycle phase
+boundary while still reaching S=1 within five rated cohorts on a truly
+non-dying workload (two startup plus three deciding rounds).
+
+The unlock path remains semantically unchanged at `tenuring.rs:642-664`:
+while locked, substantial Eden influx holds S=1; two consecutive cycles below
+`desired / 4` clear the lock and resume at S=2. The only added bookkeeping is
+clearing the entry streak when unlocking, so stale entry evidence cannot be
+reused after a later phase change.
+
+`seed_promote_lock_from_sweep` remains the same two-condition census rule
+(occupancy computes S=1 and Eden survival is at least 90%), but
+`tenuring.rs:838-890` now refuses even a qualifying census while fewer than two
+survivor cohorts have been rated. After startup it still latches immediately
+from those two conditions and hands off to the unchanged unlock path.
+
+## Prices recorded, not yet used to decide
+
+The 90% constant was not replaced. TN4 does not contain the two aligned unit
+prices needed to validate the structural inequality, so changing the decision
+rule now would merely substitute another assumption.
+
+Under `PERRY_GC_DIAG` only, each copying minor accumulates total copied and
+promoted bytes (`copying.rs:1817-1824`, `instruments.rs:350-371`). The existing
+cumulative minor pause is the copy-side numerator; cumulative
+`step_us + remark_us` is the promote-side numerator
+(`instruments.rs:391-402`). TN5 can therefore compute:
+
+```text
+copy_cost    = copy_pause_us / tenuring_copied_bytes
+promote_cost = promote_us / tenuring_promoted_bytes
+keep aging while mortality > copy_cost / promote_cost
+```
+
+The new byte atomics are not touched when diagnostics are off.
+
+## Diagnostic format
+
+Every adaptive S transition now has this format (`tenuring.rs:910-939`):
+
+```text
+[gc-tenuring] survivals FROM -> TO (REASON, eden_live_bytes=N desired=N rounds_rated=N streak=N survival_permille=N copied_bytes=N startup=true|false copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N)
+```
+
+For a sweep-seed transition, `copied_bytes` is the live Eden cohort the next
+minor would otherwise copy; the adjacent `sweep-seed` line prints live/dead,
+the two-condition verdict, and `startup=`. The process-exit block is
+(`gc/mod.rs:1410-1419`):
+
+```text
+[gc-time] wall_us=N step_us=N remark_us=N minor_us=N full_sync_us=N share_permille=N copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N
+```
+
+## Tests and explicit sabotages
+
+- `startup_shaped_survivors_do_not_contribute_to_the_lock_streak`: remove the
+  `!startup` entry conjunct; the two startup ratings plus the first steady
+  rating reach K and latch S=1.
+- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy`: raise K or
+  stop advancing the streak; the exact-K final round fails to latch.
+- `mortality_inside_the_steady_window_resets_the_lock_streak`: retain the
+  streak on a below-bar rated round; the final surviving cohort becomes the
+  cumulative Kth and latches S=1.
+- `sweep_seed_cannot_latch_from_a_startup_census`: remove the sweep startup
+  conjunct; a census satisfying both original conditions immediately latches.
+- Existing pinned-S coverage and
+  `occupancy_may_not_claim_the_ceiling_before_any_round_is_measured` were left
+  in place. They were not executable locally because of the disk gate below.
+
+## Gates
+
+Not run (zero tests/builds executed):
+
+- `cargo test -p perry-runtime --release --lib -- --test-threads=1`
+- `cargo build --release -p perry-runtime --features wasm-host`
+- `cargo build --release -p perry`
+
+Reason: `df -g /` was below the binding 12 GB floor before every possible
+Cargo invocation. It was polled every 60 seconds for 30 minutes, declining
+from 2 GB to 0 GB free. No Cargo command was invoked.
+
+Non-Cargo checks run and green:
+
+- `rustfmt --check` on all four changed Rust files
+- `git diff --check`
+- `python3 scripts/gc_runtime_root_holders.py`
+- `python3 scripts/gc_runtime_root_holders.py --self-test`
+- `scripts/check_file_size.sh`
+
+The new pointer-free TLS counters have explicit custody verdicts. The
+`PASS1_MARKED` non-moving-snapshot pin was re-audited and refreshed because
+`gc/mod.rs` changed only in the process-exit diagnostic path, outside its
+mark-complete to sweep-entry window.
+
+## Exact perrymaster request and falsifiable predictions
+
+Request TN on **main6 + this branch**: arms `unset`,
+`PERRY_GC_TENURING_SURVIVALS=1`, and
+`PERRY_GC_TENURING_SURVIVALS=2`; workloads 3300 and 400 characters; two rounds
+per arm/workload; four turns per process with graceful exit. Capture complete
+`PERRY_GC_DIAG=1` output and run both `tn_summary` and `tenure_an`, including S
+history, transition evidence, promoted bytes by turn, peak/settled RSS, and the
+four cumulative price counters.
+
+Predictions to falsify:
+
+- On cc, `unset` shows no startup latch: S-history is S2 only, or any latch is
+  after a post-startup three-round steady window.
+- Turn 1 equals pinned S2 within noise; turns 2-4 equal pinned S2.
+- Promoted bytes in turns 2-4 are approximately 12 MB.
+- Peak and settled RSS match the pinned-S2 arm (within the allowed 1-10% RSS
+  range).
+- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy` is the kill
+  condition proving the adaptive S1 lock still exists for non-dying workloads.

changelog.d/9851-lock-rates-one-cohort.md

@@ -0,0 +1,49 @@
+### Fixed
+
+- **The tenuring survival-rate lock now rates one fresh cohort, not the whole
+  survivor space — a well-formed ratio that stopped describing what it is named
+  after as soon as the threshold it sets rose above 2.**
+
+  The lock exists to answer "did an aging round filter anything?" and, when the
+  answer is no, to promote on first copy. It tested
+
+  ```
+  prev_copied >= substantial && survivor_live_bytes * 10 >= prev_copied * 9
+  ```
+
+  where `survivor_live_bytes` is every live byte leaving the from-survivor space
+  this cycle, of any age, and `prev_copied` is the previous cycle's whole intake
+  into that space. Those two scopes match — the survivor spaces are a strict
+  semispace pair, so the from-space holds exactly what the last cycle copied —
+  and the ratio cannot exceed 1. **The defect is not the arithmetic; it is which
+  population the ratio rates, and that is chosen by the very threshold the lock
+  sets.** At a threshold of 2 the space holds one fresh cohort and the ratio is
+  one aging round's survival. At 3 or 4 it also holds objects that have already
+  survived a round and are therefore selected for longevity, so the aggregate
+  clears the 90 % bar while a fresh cohort does not. The rule reads its own
+  setting back as evidence.
+
+  This was invisible while the occupancy rule sealed the loop at S=1, because
+  there `copied_bytes` is 0 and the lock's guard can never be satisfied.
+  Removing that seal handed the lock its guard back, and it became the dominant
+  route to promote-on-first-copy.
+
+  Measured on the compiled claude-code TUI, one binary, three arms via
+  `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies, 4 turns in one process:
+
+  | arm | minors | promoted | S=1 share of promotion | reached 1 via the lock |
+  |---|---|---|---|---|
+  | `=1` (pre-clamp equivalent) | 356 | 1055 MB | 100 % | - |
+  | occupancy clamp only | 368 / 384 | 982 / 980 MB | 85 % / 84 % | **8 / 12** |
+  | `=2` (positive control) | 380 | 785 MB | 0 % | n/a |
+
+  The copier now also accounts the fresh half of each cycle: `eden_copied_bytes`
+  (bytes copied out of *Eden* into the to-survivor space, no re-copies) and
+  `survivor_first_round_live_bytes` (live bytes leaving the from-survivor space
+  whose stored survival age is 1, i.e. members of exactly the cohort the
+  previous cycle's `eden_copied_bytes` counted). The lock rates those two. Both
+  are on the `[gc-copy-minor]` diagnostic line, so first-round mortality is
+  readable from any build rather than only from an instrumented one.
+
+  Reaching 1 still belongs to the paths that measure mortality; what changes is
+  that the measurement is now of one aging round at every threshold.

changelog.d/9851-occupancy-may-not-promote-on-first-copy.md

@@ -0,0 +1,67 @@
+### Fixed
+
+- **The adaptive tenuring loop's occupancy rule can no longer conclude
+  "promote on first copy" — a claim about lifetime that it has no evidence
+  for, and which destroys the evidence that would refute it.**
+
+  `retune_after_scavenge` picks a survival threshold from
+  `S = 1 + desired / influx`: the largest S whose projected survivor occupancy
+  `(S-1) x influx` fits the desired survivor size. With integer division, any
+  influx above `desired` yields exactly **1** — there is no rung at 2 or 3.
+  Measured on the compiled claude-code TUI, the first drop reads
+  `eden_live_bytes=12075344` against `desired=1048576`.
+
+  S=1 does not reduce the surviving data; it relocates it, from the survivor
+  space — where the next minor re-examines it for free — to the old generation,
+  which only a full collection can reclaim. The occupancy formula has no term
+  for that. And S=1 is **self-sealing**: nothing is copied, so `copied_bytes` is
+  0, so next cycle `prev_copied` is 0, so the survival-rate lock's guard
+  (`prev_copied >= substantial`) is false forever. Both remaining exits — the
+  occupancy recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits,
+  which say nothing about lifetime.
+
+  Measured, 4 streamed turns in one process, both arms from one binary via the
+  diagnostic knob `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies:
+
+  | | adaptive | pinned S=2 |
+  |---|---|---|
+  | minors at S=1 | **351 of 352**, carrying 100 % of promotion | 0 |
+  | threshold transitions in the whole run | **1** | 7 |
+  | survivor-round mortality samples | **1** | **393** |
+  | median mortality | **0.9 %** | **26.1 %** |
+  | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % |
+  | promoted | 1057 MB | 792 MB |
+
+  The loop takes its one and only mortality measurement on the **first minor of
+  the process** — before any steady state, when the cohort really is immortal —
+  reads 99.1 % survival, drops to 1, and can never sample again. In steady
+  state an aging round filters about **a quarter** of each cohort.
+
+  The occupancy rule now stops at the lowest threshold that still *produces*
+  that measurement. That value is 2 by construction, not by tuning: at S=1
+  nothing enters the survivor space, at S=2 exactly one cohort does. The
+  arithmetic is untouched — `compute_target_survivals` still computes 1, and
+  its test asserts so byte-identically; only what the loop may do with the
+  result changes.
+
+  **Reaching 1 still belongs to the two paths that measure mortality** — the
+  survival-rate lock (a substantial cohort of which >= 90 % came back alive)
+  and the sweep seed (the mark-sweep's own Eden live/dead split). Both are
+  untouched, so the rule is self-limiting: on a workload whose cohort genuinely
+  does not die, the lock fires after one cohort's copy and takes the loop back
+  to 1.
+
+  On claude-code it **does** fire, 8-12 times per four-turn run, and the
+  companion entry below is why: once the clamp lets the ladder climb past 2 the
+  lock is rating a population its own threshold selected. An earlier version of
+  this entry claimed the opposite ("5 of 358 substantial cohorts sit under the
+  lock's threshold, so the clamp holds rather than oscillating"); that figure
+  was measured with the threshold *pinned*, where every cohort the lock can
+  rate is a first-round cohort, and it does not describe the rule running.
+
+  Two existing tests change their expected value from 1 to 2 and keep their
+  names, structure and invariants: `drops_immediately_and_rises_debounced`
+  protects the *asymmetric response* (immediate drop, debounced rise), which
+  4 -> 2 demonstrates exactly as well as 4 -> 1; and
+  `steady_heavy_influx_is_a_fixed_point` protects *fixed-pointness*, which is
+  unchanged with 2 as the fixed point.

crates/perry-runtime/src/gc/copying.rs

@@ -617,8 +617,29 @@ impl CopyingNurseryCollector {
         // moved somewhere at any threshold), which is what makes the loop's
         // fixed point stable.
         match ptr.kind {
-            CopyingPointerKind::Eden => self.stats.eden_live_bytes += total,
-            _ => self.stats.survivor_live_bytes += total,
+            CopyingPointerKind::Eden => {
+                self.stats.eden_live_bytes += total;
+                // #9851 follow-up: the fresh half of `copied_bytes`. The
+                // survival-rate lock's denominator must be the intake of ONE
+                // cohort; `copied_bytes` also carries survivor residents being
+                // re-copied, which at a threshold above 2 is most of it.
+                if !promote {
+                    self.stats.eden_copied_bytes += total;
+                }
+            }
+            _ => {
+                self.stats.survivor_live_bytes += total;
+                // ...and the matching numerator. A from-survivor object whose
+                // stored age is 1 entered from Eden on the previous cycle, so
+                // it is a member of exactly the cohort `eden_copied_bytes`
+                // counted then. Ages above 1 have already survived a round and
+                // are a population selected for longevity; including them is
+                // what made the ratio drift above the lock's bar as the
+                // threshold rose.
+                if prior_age == 1 {
+                    self.stats.survivor_first_round_live_bytes += total;
+                }
+            }
         }
         new_user as usize
     }
@@ -1796,6 +1817,10 @@ pub(super) fn run_copied_minor_attempt(
     // enable, so it releases the latch that suppressed a repeat handoff.
     note_copying_minor_completed();
     super::instruments::note_copying_minor_pause_us(start.elapsed().as_micros() as u64);
+    super::instruments::note_tenuring_price_bytes(
+        collector.stats.copied_bytes,
+        collector.stats.promoted_bytes,
+    );
     // #7604: the process-wide liveness counters. A copying minor ran, and this
     // is how much it actually relocated -- the only evidence that distinguishes
     // "the instrument was armed" from "the instrument fired".
@@ -1867,14 +1892,24 @@ pub(super) fn run_copied_minor_attempt(
             .copied_objects
             .saturating_add(collector.stats.promoted_objects),
     );
-    retune_after_scavenge(
-        collector.stats.eden_live_bytes,
+    // #9851 follow-up: the survival-rate lock is fed the FRESH cohort's intake
+    // and that same cohort's survival, not the whole survivor space's. See
+    // `retune_after_scavenge`.
+    #[cfg(test)]
+    test_record_cohort_split(
         collector.stats.copied_bytes,
+        collector.stats.eden_copied_bytes,
         collector.stats.survivor_live_bytes,
+        collector.stats.survivor_first_round_live_bytes,
+    );
+    retune_after_scavenge(
+        collector.stats.eden_live_bytes,
+        collector.stats.eden_copied_bytes,
+        collector.stats.survivor_first_round_live_bytes,
     );
     if crate::gc::gc_diag_enabled() {
         eprintln!(
-            "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} trigger={:?} declared_safepoint={}",
+            "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} eden_copied_bytes={} survivor_live_bytes={} survivor_first_round_live_bytes={} trigger={:?} declared_safepoint={}",
             collector.stats.in_place_promotion,
             untraced,
             super::untraced_promotion_cycles(),
@@ -1890,6 +1925,9 @@ pub(super) fn run_copied_minor_attempt(
             freed_bytes,
             collector.stats.tenuring_survivals,
             collector.stats.eden_live_bytes,
+            collector.stats.eden_copied_bytes,
+            collector.stats.survivor_live_bytes,
+            collector.stats.survivor_first_round_live_bytes,
             _trigger_kind,
             super::policy::GC_AT_DECLARED_SAFEPOINT.with(std::cell::Cell::get)
         );
@@ -1908,6 +1946,41 @@ pub(super) fn run_copied_minor_attempt(
     }))
 }
 
+/// Test-only witness for the #9851 follow-up: the whole-space pair against the
+/// fresh-cohort pair, as the copier computed them for one cycle. Without this
+/// the change is unfalsifiable from a test — the two quantities are equal on
+/// every heap whose survivor space holds a single generation, which is every
+/// heap at a threshold of 2 or below.
+#[cfg(test)]
+thread_local! {
+    static LAST_COHORT_SPLIT: std::cell::Cell<(usize, usize, usize, usize)> =
+        const { std::cell::Cell::new((0, 0, 0, 0)) };
+}
+
+#[cfg(test)]
+fn test_record_cohort_split(
+    copied_bytes: usize,
+    eden_copied_bytes: usize,
+    survivor_live_bytes: usize,
+    first_round_live_bytes: usize,
+) {
+    LAST_COHORT_SPLIT.with(|c| {
+        c.set((
+            copied_bytes,
+            eden_copied_bytes,
+            survivor_live_bytes,
+            first_round_live_bytes,
+        ))
+    });
+}
+
+/// `(copied_bytes, eden_copied_bytes, survivor_live_bytes, first_round_live_bytes)`
+/// from the most recent copying minor on this thread.
+#[cfg(test)]
+pub(super) fn test_last_cohort_split() -> (usize, usize, usize, usize) {
+    LAST_COHORT_SPLIT.with(std::cell::Cell::get)
+}
+
 fn finalize_dead_copied_minor_from_space_side_allocations() {
     crate::map::finalize_dead_copied_minor_from_space_maps();
     crate::set::finalize_dead_copied_minor_from_space_sets();