diff --git a/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md b/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md new file mode 100644 index 0000000000..3debf9aed5 --- /dev/null +++ b/cc-perf-campaign/codex/REPORT_tenuring_evidence_lock.md @@ -0,0 +1,152 @@ +# Tenuring steady-evidence lock report + +Code SHA: `d229d7d715cee4725776a83fa210ac5727ce1862` + +Branch: `perf/tenuring-evidence-lock` + +## Lock map before and after this change + +The copying collector builds the cohort-scoped signal in +`crates/perry-runtime/src/gc/copying.rs:621-640`: `eden_copied_bytes` counts +fresh Eden objects copied into the survivor space, while +`survivor_first_round_live_bytes` counts age-1 objects from that same cohort +which return alive on the next copying minor. The completed minor passes those +values to `retune_after_scavenge` at `copying.rs:1895-1909`. + +`crates/perry-runtime/src/gc/tenuring.rs:600-639` swaps the current +`eden_copied_bytes` into `PREV_COPIED_BYTES`. A rated round is, as before, a +cycle with `prev_cohort_copied > 0`: a previous copying minor admitted a fresh +cohort, so this minor can divide its `first_round_live_bytes` by that cohort's +intake. Before this change that single condition was also sufficient to make +the first startup cohort eligible for the lock. That is why TN4's startup +cohort qualified: the first minor copied it, the second minor returned it +nearly intact, and the code had no process-phase evidence. + +The entry decision is now at `tenuring.rs:676-694`. A rated round contributes +only when it is post-startup, its previous fresh intake is at least the existing +`desired / 4` substantial-volume bar, and its survival is at least the existing +90% bar. Any rated round that fails those conditions resets +`PROMOTE_LOCK_STREAK`. `PROMOTE_LOCK` latches only when that streak reaches +three. + +Startup is the first two **rated survivor cohorts** +(`tenuring.rs:195-205, 613-627`), not a wall-clock duration and not the +allocation-census flag. This uses the lock's own threshold-invariant evidence +and cannot re-pace collection. The allocation census is deliberately not the +marker: it seeds halfway to the first nursery cap, before the first copying +minor, so it was already true for precisely the TN4 startup cohort that must be +excluded. Two excluded ratings cover the observed two-minor startup phase; K=3 +is the smallest steady window that rejects a one-off or two-cycle phase +boundary while still reaching S=1 within five rated cohorts on a truly +non-dying workload (two startup plus three deciding rounds). + +The unlock path remains semantically unchanged at `tenuring.rs:642-664`: +while locked, substantial Eden influx holds S=1; two consecutive cycles below +`desired / 4` clear the lock and resume at S=2. The only added bookkeeping is +clearing the entry streak when unlocking, so stale entry evidence cannot be +reused after a later phase change. + +`seed_promote_lock_from_sweep` remains the same two-condition census rule +(occupancy computes S=1 and Eden survival is at least 90%), but +`tenuring.rs:838-890` now refuses even a qualifying census while fewer than two +survivor cohorts have been rated. After startup it still latches immediately +from those two conditions and hands off to the unchanged unlock path. + +## Prices recorded, not yet used to decide + +The 90% constant was not replaced. TN4 does not contain the two aligned unit +prices needed to validate the structural inequality, so changing the decision +rule now would merely substitute another assumption. + +Under `PERRY_GC_DIAG` only, each copying minor accumulates total copied and +promoted bytes (`copying.rs:1817-1824`, `instruments.rs:350-371`). The existing +cumulative minor pause is the copy-side numerator; cumulative +`step_us + remark_us` is the promote-side numerator +(`instruments.rs:391-402`). TN5 can therefore compute: + +```text +copy_cost = copy_pause_us / tenuring_copied_bytes +promote_cost = promote_us / tenuring_promoted_bytes +keep aging while mortality > copy_cost / promote_cost +``` + +The new byte atomics are not touched when diagnostics are off. + +## Diagnostic format + +Every adaptive S transition now has this format (`tenuring.rs:910-939`): + +```text +[gc-tenuring] survivals FROM -> TO (REASON, eden_live_bytes=N desired=N rounds_rated=N streak=N survival_permille=N copied_bytes=N startup=true|false copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N) +``` + +For a sweep-seed transition, `copied_bytes` is the live Eden cohort the next +minor would otherwise copy; the adjacent `sweep-seed` line prints live/dead, +the two-condition verdict, and `startup=`. The process-exit block is +(`gc/mod.rs:1410-1419`): + +```text +[gc-time] wall_us=N step_us=N remark_us=N minor_us=N full_sync_us=N share_permille=N copy_pause_us=N tenuring_copied_bytes=N promote_us=N tenuring_promoted_bytes=N +``` + +## Tests and explicit sabotages + +- `startup_shaped_survivors_do_not_contribute_to_the_lock_streak`: remove the + `!startup` entry conjunct; the two startup ratings plus the first steady + rating reach K and latch S=1. +- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy`: raise K or + stop advancing the streak; the exact-K final round fails to latch. +- `mortality_inside_the_steady_window_resets_the_lock_streak`: retain the + streak on a below-bar rated round; the final surviving cohort becomes the + cumulative Kth and latches S=1. +- `sweep_seed_cannot_latch_from_a_startup_census`: remove the sweep startup + conjunct; a census satisfying both original conditions immediately latches. +- Existing pinned-S coverage and + `occupancy_may_not_claim_the_ceiling_before_any_round_is_measured` were left + in place. They were not executable locally because of the disk gate below. + +## Gates + +Not run (zero tests/builds executed): + +- `cargo test -p perry-runtime --release --lib -- --test-threads=1` +- `cargo build --release -p perry-runtime --features wasm-host` +- `cargo build --release -p perry` + +Reason: `df -g /` was below the binding 12 GB floor before every possible +Cargo invocation. It was polled every 60 seconds for 30 minutes, declining +from 2 GB to 0 GB free. No Cargo command was invoked. + +Non-Cargo checks run and green: + +- `rustfmt --check` on all four changed Rust files +- `git diff --check` +- `python3 scripts/gc_runtime_root_holders.py` +- `python3 scripts/gc_runtime_root_holders.py --self-test` +- `scripts/check_file_size.sh` + +The new pointer-free TLS counters have explicit custody verdicts. The +`PASS1_MARKED` non-moving-snapshot pin was re-audited and refreshed because +`gc/mod.rs` changed only in the process-exit diagnostic path, outside its +mark-complete to sweep-entry window. + +## Exact perrymaster request and falsifiable predictions + +Request TN on **main6 + this branch**: arms `unset`, +`PERRY_GC_TENURING_SURVIVALS=1`, and +`PERRY_GC_TENURING_SURVIVALS=2`; workloads 3300 and 400 characters; two rounds +per arm/workload; four turns per process with graceful exit. Capture complete +`PERRY_GC_DIAG=1` output and run both `tn_summary` and `tenure_an`, including S +history, transition evidence, promoted bytes by turn, peak/settled RSS, and the +four cumulative price counters. + +Predictions to falsify: + +- On cc, `unset` shows no startup latch: S-history is S2 only, or any latch is + after a post-startup three-round steady window. +- Turn 1 equals pinned S2 within noise; turns 2-4 equal pinned S2. +- Promoted bytes in turns 2-4 are approximately 12 MB. +- Peak and settled RSS match the pinned-S2 arm (within the allowed 1-10% RSS + range). +- `k_steady_fully_surviving_rounds_latch_promote_on_first_copy` is the kill + condition proving the adaptive S1 lock still exists for non-dying workloads. diff --git a/changelog.d/9851-lock-rates-one-cohort.md b/changelog.d/9851-lock-rates-one-cohort.md new file mode 100644 index 0000000000..deb0ad2d56 --- /dev/null +++ b/changelog.d/9851-lock-rates-one-cohort.md @@ -0,0 +1,49 @@ +### Fixed + +- **The tenuring survival-rate lock now rates one fresh cohort, not the whole + survivor space — a well-formed ratio that stopped describing what it is named + after as soon as the threshold it sets rose above 2.** + + The lock exists to answer "did an aging round filter anything?" and, when the + answer is no, to promote on first copy. It tested + + ``` + prev_copied >= substantial && survivor_live_bytes * 10 >= prev_copied * 9 + ``` + + where `survivor_live_bytes` is every live byte leaving the from-survivor space + this cycle, of any age, and `prev_copied` is the previous cycle's whole intake + into that space. Those two scopes match — the survivor spaces are a strict + semispace pair, so the from-space holds exactly what the last cycle copied — + and the ratio cannot exceed 1. **The defect is not the arithmetic; it is which + population the ratio rates, and that is chosen by the very threshold the lock + sets.** At a threshold of 2 the space holds one fresh cohort and the ratio is + one aging round's survival. At 3 or 4 it also holds objects that have already + survived a round and are therefore selected for longevity, so the aggregate + clears the 90 % bar while a fresh cohort does not. The rule reads its own + setting back as evidence. + + This was invisible while the occupancy rule sealed the loop at S=1, because + there `copied_bytes` is 0 and the lock's guard can never be satisfied. + Removing that seal handed the lock its guard back, and it became the dominant + route to promote-on-first-copy. + + Measured on the compiled claude-code TUI, one binary, three arms via + `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies, 4 turns in one process: + + | arm | minors | promoted | S=1 share of promotion | reached 1 via the lock | + |---|---|---|---|---| + | `=1` (pre-clamp equivalent) | 356 | 1055 MB | 100 % | - | + | occupancy clamp only | 368 / 384 | 982 / 980 MB | 85 % / 84 % | **8 / 12** | + | `=2` (positive control) | 380 | 785 MB | 0 % | n/a | + + The copier now also accounts the fresh half of each cycle: `eden_copied_bytes` + (bytes copied out of *Eden* into the to-survivor space, no re-copies) and + `survivor_first_round_live_bytes` (live bytes leaving the from-survivor space + whose stored survival age is 1, i.e. members of exactly the cohort the + previous cycle's `eden_copied_bytes` counted). The lock rates those two. Both + are on the `[gc-copy-minor]` diagnostic line, so first-round mortality is + readable from any build rather than only from an instrumented one. + + Reaching 1 still belongs to the paths that measure mortality; what changes is + that the measurement is now of one aging round at every threshold. diff --git a/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md b/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md new file mode 100644 index 0000000000..40d6d0a057 --- /dev/null +++ b/changelog.d/9851-occupancy-may-not-promote-on-first-copy.md @@ -0,0 +1,67 @@ +### Fixed + +- **The adaptive tenuring loop's occupancy rule can no longer conclude + "promote on first copy" — a claim about lifetime that it has no evidence + for, and which destroys the evidence that would refute it.** + + `retune_after_scavenge` picks a survival threshold from + `S = 1 + desired / influx`: the largest S whose projected survivor occupancy + `(S-1) x influx` fits the desired survivor size. With integer division, any + influx above `desired` yields exactly **1** — there is no rung at 2 or 3. + Measured on the compiled claude-code TUI, the first drop reads + `eden_live_bytes=12075344` against `desired=1048576`. + + S=1 does not reduce the surviving data; it relocates it, from the survivor + space — where the next minor re-examines it for free — to the old generation, + which only a full collection can reclaim. The occupancy formula has no term + for that. And S=1 is **self-sealing**: nothing is copied, so `copied_bytes` is + 0, so next cycle `prev_copied` is 0, so the survival-rate lock's guard + (`prev_copied >= substantial`) is false forever. Both remaining exits — the + occupancy recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits, + which say nothing about lifetime. + + Measured, 4 streamed turns in one process, both arms from one binary via the + diagnostic knob `PERRY_GC_TENURING_SURVIVALS`, 3300-character replies: + + | | adaptive | pinned S=2 | + |---|---|---| + | minors at S=1 | **351 of 352**, carrying 100 % of promotion | 0 | + | threshold transitions in the whole run | **1** | 7 | + | survivor-round mortality samples | **1** | **393** | + | median mortality | **0.9 %** | **26.1 %** | + | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % | + | promoted | 1057 MB | 792 MB | + + The loop takes its one and only mortality measurement on the **first minor of + the process** — before any steady state, when the cohort really is immortal — + reads 99.1 % survival, drops to 1, and can never sample again. In steady + state an aging round filters about **a quarter** of each cohort. + + The occupancy rule now stops at the lowest threshold that still *produces* + that measurement. That value is 2 by construction, not by tuning: at S=1 + nothing enters the survivor space, at S=2 exactly one cohort does. The + arithmetic is untouched — `compute_target_survivals` still computes 1, and + its test asserts so byte-identically; only what the loop may do with the + result changes. + + **Reaching 1 still belongs to the two paths that measure mortality** — the + survival-rate lock (a substantial cohort of which >= 90 % came back alive) + and the sweep seed (the mark-sweep's own Eden live/dead split). Both are + untouched, so the rule is self-limiting: on a workload whose cohort genuinely + does not die, the lock fires after one cohort's copy and takes the loop back + to 1. + + On claude-code it **does** fire, 8-12 times per four-turn run, and the + companion entry below is why: once the clamp lets the ladder climb past 2 the + lock is rating a population its own threshold selected. An earlier version of + this entry claimed the opposite ("5 of 358 substantial cohorts sit under the + lock's threshold, so the clamp holds rather than oscillating"); that figure + was measured with the threshold *pinned*, where every cohort the lock can + rate is a first-round cohort, and it does not describe the rule running. + + Two existing tests change their expected value from 1 to 2 and keep their + names, structure and invariants: `drops_immediately_and_rises_debounced` + protects the *asymmetric response* (immediate drop, debounced rise), which + 4 -> 2 demonstrates exactly as well as 4 -> 1; and + `steady_heavy_influx_is_a_fixed_point` protects *fixed-pointness*, which is + unchanged with 2 as the fixed point. diff --git a/crates/perry-runtime/src/gc/copying.rs b/crates/perry-runtime/src/gc/copying.rs index 2eb134d64f..8d9cdb3681 100644 --- a/crates/perry-runtime/src/gc/copying.rs +++ b/crates/perry-runtime/src/gc/copying.rs @@ -617,8 +617,29 @@ impl CopyingNurseryCollector { // moved somewhere at any threshold), which is what makes the loop's // fixed point stable. match ptr.kind { - CopyingPointerKind::Eden => self.stats.eden_live_bytes += total, - _ => self.stats.survivor_live_bytes += total, + CopyingPointerKind::Eden => { + self.stats.eden_live_bytes += total; + // #9851 follow-up: the fresh half of `copied_bytes`. The + // survival-rate lock's denominator must be the intake of ONE + // cohort; `copied_bytes` also carries survivor residents being + // re-copied, which at a threshold above 2 is most of it. + if !promote { + self.stats.eden_copied_bytes += total; + } + } + _ => { + self.stats.survivor_live_bytes += total; + // ...and the matching numerator. A from-survivor object whose + // stored age is 1 entered from Eden on the previous cycle, so + // it is a member of exactly the cohort `eden_copied_bytes` + // counted then. Ages above 1 have already survived a round and + // are a population selected for longevity; including them is + // what made the ratio drift above the lock's bar as the + // threshold rose. + if prior_age == 1 { + self.stats.survivor_first_round_live_bytes += total; + } + } } new_user as usize } @@ -1796,6 +1817,10 @@ pub(super) fn run_copied_minor_attempt( // enable, so it releases the latch that suppressed a repeat handoff. note_copying_minor_completed(); super::instruments::note_copying_minor_pause_us(start.elapsed().as_micros() as u64); + super::instruments::note_tenuring_price_bytes( + collector.stats.copied_bytes, + collector.stats.promoted_bytes, + ); // #7604: the process-wide liveness counters. A copying minor ran, and this // is how much it actually relocated -- the only evidence that distinguishes // "the instrument was armed" from "the instrument fired". @@ -1867,14 +1892,24 @@ pub(super) fn run_copied_minor_attempt( .copied_objects .saturating_add(collector.stats.promoted_objects), ); - retune_after_scavenge( - collector.stats.eden_live_bytes, + // #9851 follow-up: the survival-rate lock is fed the FRESH cohort's intake + // and that same cohort's survival, not the whole survivor space's. See + // `retune_after_scavenge`. + #[cfg(test)] + test_record_cohort_split( collector.stats.copied_bytes, + collector.stats.eden_copied_bytes, collector.stats.survivor_live_bytes, + collector.stats.survivor_first_round_live_bytes, + ); + retune_after_scavenge( + collector.stats.eden_live_bytes, + collector.stats.eden_copied_bytes, + collector.stats.survivor_first_round_live_bytes, ); if crate::gc::gc_diag_enabled() { eprintln!( - "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} trigger={:?} declared_safepoint={}", + "[gc-copy-minor] ran in_place={} untraced={} untraced_cycles={} untraced_objects={} in_place_blocks={} in_place_dead_bytes={} sparse_blocks={} survival_permille={} copied_objects={} copied_bytes={} promoted_objects={} promoted_bytes={} freed_bytes={} tenuring_survivals={} eden_live_bytes={} eden_copied_bytes={} survivor_live_bytes={} survivor_first_round_live_bytes={} trigger={:?} declared_safepoint={}", collector.stats.in_place_promotion, untraced, super::untraced_promotion_cycles(), @@ -1890,6 +1925,9 @@ pub(super) fn run_copied_minor_attempt( freed_bytes, collector.stats.tenuring_survivals, collector.stats.eden_live_bytes, + collector.stats.eden_copied_bytes, + collector.stats.survivor_live_bytes, + collector.stats.survivor_first_round_live_bytes, _trigger_kind, super::policy::GC_AT_DECLARED_SAFEPOINT.with(std::cell::Cell::get) ); @@ -1908,6 +1946,41 @@ pub(super) fn run_copied_minor_attempt( })) } +/// Test-only witness for the #9851 follow-up: the whole-space pair against the +/// fresh-cohort pair, as the copier computed them for one cycle. Without this +/// the change is unfalsifiable from a test — the two quantities are equal on +/// every heap whose survivor space holds a single generation, which is every +/// heap at a threshold of 2 or below. +#[cfg(test)] +thread_local! { + static LAST_COHORT_SPLIT: std::cell::Cell<(usize, usize, usize, usize)> = + const { std::cell::Cell::new((0, 0, 0, 0)) }; +} + +#[cfg(test)] +fn test_record_cohort_split( + copied_bytes: usize, + eden_copied_bytes: usize, + survivor_live_bytes: usize, + first_round_live_bytes: usize, +) { + LAST_COHORT_SPLIT.with(|c| { + c.set(( + copied_bytes, + eden_copied_bytes, + survivor_live_bytes, + first_round_live_bytes, + )) + }); +} + +/// `(copied_bytes, eden_copied_bytes, survivor_live_bytes, first_round_live_bytes)` +/// from the most recent copying minor on this thread. +#[cfg(test)] +pub(super) fn test_last_cohort_split() -> (usize, usize, usize, usize) { + LAST_COHORT_SPLIT.with(std::cell::Cell::get) +} + fn finalize_dead_copied_minor_from_space_side_allocations() { crate::map::finalize_dead_copied_minor_from_space_maps(); crate::set::finalize_dead_copied_minor_from_space_sets(); diff --git a/crates/perry-runtime/src/gc/instruments.rs b/crates/perry-runtime/src/gc/instruments.rs index 267a8bd1d5..d629dbc219 100644 --- a/crates/perry-runtime/src/gc/instruments.rs +++ b/crates/perry-runtime/src/gc/instruments.rs @@ -347,6 +347,10 @@ static REMARK_TOTAL_US: AtomicU64 = AtomicU64::new(0); static MINOR_TOTAL_US: AtomicU64 = AtomicU64::new(0); /// Total microseconds spent inside synchronous `js_gc_collect` calls. static FULL_TOTAL_US: AtomicU64 = AtomicU64::new(0); +/// Bytes copied into a survivor space, for the tenuring copy-price diagnostic. +static TENURING_COPIED_BYTES: AtomicU64 = AtomicU64::new(0); +/// Bytes promoted to old-gen, for the tenuring promotion-price diagnostic. +static TENURING_PROMOTED_BYTES: AtomicU64 = AtomicU64::new(0); /// Record one copying minor's pause duration. #[inline] @@ -354,6 +358,19 @@ pub(crate) fn note_copying_minor_pause_us(us: u64) { MINOR_TOTAL_US.fetch_add(us, Ordering::Relaxed); } +/// Record the byte denominators for the tenuring price experiment. +/// +/// These are diagnostic-only: when `PERRY_GC_DIAG` is off the two atomics are +/// untouched, so measuring the future decision rule adds no release-path cost. +#[inline] +pub(crate) fn note_tenuring_price_bytes(copied_bytes: usize, promoted_bytes: usize) { + if !crate::gc::gc_diag_enabled() { + return; + } + TENURING_COPIED_BYTES.fetch_add(copied_bytes as u64, Ordering::Relaxed); + TENURING_PROMOTED_BYTES.fetch_add(promoted_bytes as u64, Ordering::Relaxed); +} + /// Record one synchronous full collection's wall duration. #[inline] pub(crate) fn note_full_collect_us(us: u64) { @@ -371,6 +388,20 @@ pub fn gc_time_totals_us() -> (u64, u64, u64, u64) { ) } +/// `(copy pause us, copied bytes, step+remark us, promoted bytes)` cumulative +/// counters for pricing aging against promotion. Byte counters remain zero +/// unless diagnostics are enabled. +pub(crate) fn tenuring_price_counters() -> (u64, u64, u64, u64) { + ( + MINOR_TOTAL_US.load(Ordering::Relaxed), + TENURING_COPIED_BYTES.load(Ordering::Relaxed), + STEP_TOTAL_US + .load(Ordering::Relaxed) + .saturating_add(REMARK_TOTAL_US.load(Ordering::Relaxed)), + TENURING_PROMOTED_BYTES.load(Ordering::Relaxed), + ) +} + /// Microseconds since the first call in this process (the same epoch the /// mark-barrier timer uses) — the wall-clock denominator for the share line. pub(crate) fn wall_us_since_epoch() -> u64 { diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 64409430cb..ec32b4438f 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -1409,10 +1409,12 @@ fn emit_incremental_liveness_diag() { /// disjoint by construction. fn emit_gc_time_share_diag() { let (step_us, remark_us, minor_us, full_us) = instruments::gc_time_totals_us(); + let (copy_pause_us, tenuring_copied_bytes, promote_us, tenuring_promoted_bytes) = + instruments::tenuring_price_counters(); let wall_us = instruments::wall_us_since_epoch().max(1); let pause_us = step_us + remark_us + minor_us; eprintln!( - "[gc-time] wall_us={wall_us} step_us={step_us} remark_us={remark_us} minor_us={minor_us} full_sync_us={full_us} share_permille={}", + "[gc-time] wall_us={wall_us} step_us={step_us} remark_us={remark_us} minor_us={minor_us} full_sync_us={full_us} share_permille={} copy_pause_us={copy_pause_us} tenuring_copied_bytes={tenuring_copied_bytes} promote_us={promote_us} tenuring_promoted_bytes={tenuring_promoted_bytes}", pause_us.saturating_mul(1000) / wall_us, ); } diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index ac62c3b605..e8c48255e1 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -275,6 +275,18 @@ pub(super) struct CopyingNurseryTraceStats { /// Live bytes re-copied/promoted out of the from-survivor space this /// cycle — the re-copy tax the adaptive loop exists to bound. pub(super) survivor_live_bytes: usize, + /// #9851 follow-up: the FRESH half of `copied_bytes` — bytes copied out of + /// Eden into the to-survivor space this cycle, excluding survivor-space + /// residents being re-copied. This is the intake of exactly one cohort, + /// and it is the denominator the survival-rate lock must use. + pub(super) eden_copied_bytes: usize, + /// The matching numerator: live bytes moved out of the from-survivor space + /// this cycle whose stored survival age was 1 — i.e. objects that entered + /// the survivor space from Eden on the PREVIOUS cycle, and nothing older. + /// `survivor_live_bytes` rates the whole space, whose composition changes + /// with the threshold; this rates one aging round of one fresh cohort, + /// which is what the lock's conclusion is about. + pub(super) survivor_first_round_live_bytes: usize, pub(super) large_excluded_objects: usize, pub(super) large_excluded_bytes: usize, pub(super) reset_blocks: usize, @@ -1151,6 +1163,8 @@ impl GcCycleTrace { "tenuring_survivals": self.copying_nursery.tenuring_survivals, "eden_live_bytes": self.copying_nursery.eden_live_bytes, "survivor_live_bytes": self.copying_nursery.survivor_live_bytes, + "eden_copied_bytes": self.copying_nursery.eden_copied_bytes, + "survivor_first_round_live_bytes": self.copying_nursery.survivor_first_round_live_bytes, "large_excluded_objects": self.copying_nursery.large_excluded_objects, "large_excluded_bytes": self.copying_nursery.large_excluded_bytes, "reset_blocks": self.copying_nursery.reset_blocks, diff --git a/crates/perry-runtime/src/gc/tenuring.rs b/crates/perry-runtime/src/gc/tenuring.rs index 1abb9dd99c..b7ca897288 100644 --- a/crates/perry-runtime/src/gc/tenuring.rs +++ b/crates/perry-runtime/src/gc/tenuring.rs @@ -47,14 +47,14 @@ //! default — 40 bytes under) settles at S=2 and still copies every //! surviving byte exactly once for nothing, because 100% of each cohort //! survives its survivor round and gets promoted a cycle later anyway. -//! The **survival-rate lock** closes that: when last cycle's survivor -//! intake (`copied_bytes`) was substantial and ≥90% of it came back out -//! alive this cycle (`survivor_live_bytes`), the aging round demonstrably -//! filters nothing, so the threshold locks to 1 (promote on first copy) -//! until the influx goes quiet. The lock's exit signal — influx below -//! `desired/4` for two consecutive cycles — stays measurable while -//! locked, unlike survivor occupancy, which is zero at S=1 and would -//! leave the loop blind. +//! The **survival-rate lock** closes that, but only from steady evidence. The +//! first two rated cohorts belong to process startup and cannot decide the +//! lock. After that, three consecutive substantial survivor rounds must each +//! return ≥90% of the cohort copied in the preceding cycle before the threshold +//! locks to 1 (promote on first copy). Any rated round below that bar resets the +//! streak. The lock's exit signal — influx below `desired/4` for two consecutive +//! cycles — stays measurable while locked, unlike survivor occupancy, which is +//! zero at S=1 and would leave the loop blind. //! //! There is no env knob here (see CLAUDE.md's GC knob kill-policy): the //! loop is always on, and its neutral state — influx below `desired`, @@ -63,9 +63,11 @@ //! //! ## Seeding the lock from a non-copying collection (#7598) //! -//! The survival-rate lock above is correct but **one cycle late by -//! construction**: it keys on `prev_copied`, so a *previous copying minor* -//! must already have filled the survivor space. On a workload with one +//! The survival-rate measurement is **one cycle late by construction**: it +//! keys on `prev_copied`, so a *previous copying minor* must already have filled +//! the survivor space. After the startup evidence boundary, a sweep seed can +//! still establish the lock before a later copying minor pays that round trip. +//! On a workload with one //! long-lived burst (`json_pipeline`: `out.push({…})` 500k times) the first //! copying minor therefore always pays the wasted copy — measured 268 MB //! Eden→survivor on cycle 3 and the same 268 MB survivor→old on cycle 4, @@ -138,13 +140,70 @@ use super::*; -/// Ceiling and power-on value: the previous fixed threshold. +/// Ceiling and previous fixed threshold. pub(super) const GC_TENURING_SURVIVALS_MAX: u8 = GC_COPY_PROMOTION_SURVIVALS; +/// The lowest threshold the **occupancy rule** may select. +/// +/// Not a tuned number: it is the lowest S at which `copied_bytes > 0`, i.e. the +/// lowest value that still PRODUCES the survivor-round measurement. At S=1 +/// nothing enters the survivor space; at S=2 exactly one cohort does. +/// +/// Why the occupancy rule must not reach 1 (#9851). A threshold of 1 is a claim +/// about **lifetime** — "this cohort will not die, promote it on first copy" — +/// and the occupancy rule measures **space**: `(S-1) * influx <= desired` asks +/// only whether one cohort fits in the desired survivor size. When it does not, +/// S=1 does not reduce the surviving data; it relocates it, from the survivor +/// space (where the next minor re-examines it for free) to the old generation +/// (which only a full can reclaim). The formula has no term for that. +/// +/// Worse, S=1 is **self-sealing**: with nothing copied, `copied_bytes` is 0, so +/// next cycle `prev_copied` is 0, so the survival-rate lock's guard +/// (`prev_copied >= substantial`) is false forever. The state destroys the only +/// measurement that could refute it, and both remaining exits — the occupancy +/// recompute and `PROMOTE_LOCK`'s unlock — are *quiet-influx* exits, which say +/// nothing about lifetime. +/// +/// Measured on the compiled claude-code TUI, 4 streamed turns in one process, +/// both arms from one binary via `PERRY_GC_TENURING_SURVIVALS` (3300-char): +/// +/// | | adaptive | pinned S=2 | +/// |---|---|---| +/// | minors at S=1 | 351 of 352, carrying 100 % of promotion | 0 | +/// | survivor-round mortality samples | **1** | **393** | +/// | median mortality | **0.9 %** — the first minor of the process | **26.1 %** | +/// | ...in steady turns 2 / 3 / 4 | not measurable | 26.1 / 26.1 / 26.1 % | +/// | promoted | 1057 MB | 792 MB | +/// +/// The loop takes its one and only mortality sample on the first minor of the +/// process — before any steady state, when the cohort really is immortal — +/// concludes "nothing dies", and can never sample again. In steady state an +/// aging round filters about **a quarter** of the cohort. +/// +/// Reaching 1 still belongs to the two paths that actually MEASURE mortality: +/// the survival-rate lock (`prev_copied` substantial and >=90 % of it came back +/// alive) and the sweep seed (the mark-sweep's own Eden live/dead split). The +/// evidence gate below delays, rather than removes, those paths. So the rule is +/// self-limiting: on a workload whose steady cohorts genuinely do not die, the +/// lock reaches 1 after its bounded window. +pub(super) const OCCUPANCY_MIN_SURVIVALS: u8 = 2; + /// Consecutive cycles the computed target must exceed the current threshold /// before it is raised (by one step). const RAISE_DEBOUNCE_CYCLES: u8 = 2; +/// Rated survivor cohorts excluded as process startup. TN4 observed the lock +/// rating exactly the cohort formed by the first two copying minors, before the +/// first application turn. Counting rated cohorts (rather than wall time or an +/// allocation threshold) uses the lock's own threshold-invariant evidence and +/// cannot change collection pacing. +const STARTUP_RATED_ROUNDS: u64 = 2; + +/// Consecutive qualifying post-startup cohorts required to lock. Three is the +/// smallest window that both rejects a one-off/two-cycle phase boundary and +/// reaches S=1 promptly on a genuinely non-dying workload. +const PROMOTE_LOCK_RATED_ROUNDS: u8 = 3; + /// Ceiling for the influx-driven nursery cap scale: 16 MB × 4 = 64 MB. /// Bounds the young-gen RSS contribution on live-set-bound workloads while /// still cutting their collection count 4× (each collection carries a fixed @@ -153,11 +212,25 @@ const RAISE_DEBOUNCE_CYCLES: u8 = 2; const NURSERY_CAP_SCALE_MAX: u8 = 4; crate::perry_thread_local! { - static TENURING_SURVIVALS: Cell = const { Cell::new(GC_TENURING_SURVIVALS_MAX) }; + /// Power-on threshold. This is `OCCUPANCY_MIN_SURVIVALS`, not the ceiling: + /// see `SURVIVOR_ROUND_MEASURED`. Starting at the ceiling is a claim that + /// young objects live long, made before a single object has been given the + /// chance to die, and it is the expensive direction of that claim -- every + /// survivor is copied three times before it can be promoted. + static TENURING_SURVIVALS: Cell = const { Cell::new(OCCUPANCY_MIN_SURVIVALS) }; + /// Has any survivor round been RATED yet on this thread -- i.e. did some + /// cycle put a cohort into the survivor space that the next cycle could + /// then follow? Until this is true the loop has no lifetime evidence of + /// any kind, and the occupancy rule may not move off the floor. + static SURVIVOR_ROUND_MEASURED: Cell = const { Cell::new(false) }; static RAISE_STREAK: Cell = const { Cell::new(0) }; /// Survival-rate lock: promote-on-first-copy until influx goes quiet. static PROMOTE_LOCK: Cell = const { Cell::new(false) }; static UNLOCK_STREAK: Cell = const { Cell::new(0) }; + /// Number of survivor cohorts whose first round has been rated. + static RATED_ROUNDS: Cell = const { Cell::new(0) }; + /// Consecutive substantial, ≥90%-surviving post-startup rated cohorts. + static PROMOTE_LOCK_STREAK: Cell = const { Cell::new(0) }; /// Bytes the previous copying minor put into the to-survivor space — /// the denominator of this cycle's survival rate. static PREV_COPIED_BYTES: Cell = const { Cell::new(0) }; @@ -179,17 +252,50 @@ crate::perry_thread_local! { static OBJECT_CENSUS_SEEDED: Cell = const { Cell::new(false) }; } +#[cfg(test)] +thread_local! { + /// Scoped threshold pin for tests of mechanisms that require a particular + /// promotion age. This is thread-local for the same reason as the adaptive + /// state: runtime tests share one process and may run on different threads. + static TENURING_SURVIVALS_TEST_OVERRIDE: Cell> = const { Cell::new(None) }; +} + /// The survivals threshold the next copying minor should promote at: /// `next_age >= tenuring_survivals()` tenures. In `1..=4`; 4 is the /// original fixed policy, 1 promotes every live nursery object on first /// copy. pub(super) fn tenuring_survivals() -> u8 { + #[cfg(test)] + if let Some(forced) = TENURING_SURVIVALS_TEST_OVERRIDE.with(Cell::get) { + return forced; + } if let Some(forced) = tenuring_survivals_override() { return forced; } TENURING_SURVIVALS.with(Cell::get) } +/// Pin the promotion age for a threshold-sensitive test on this thread. +/// Restores the previous pin on drop; the adaptive policy continues to run +/// underneath it, but every copying minor snapshots the explicitly pinned age. +#[cfg(test)] +pub(super) fn set_survivals_for_test(survivals: u8) -> TenuringSurvivalsTestGuard { + assert!((1..=GC_TENURING_SURVIVALS_MAX).contains(&survivals)); + TenuringSurvivalsTestGuard( + TENURING_SURVIVALS_TEST_OVERRIDE.with(|cell| cell.replace(Some(survivals))), + ) +} + +#[cfg(test)] +pub(super) struct TenuringSurvivalsTestGuard(Option); + +#[cfg(test)] +impl Drop for TenuringSurvivalsTestGuard { + fn drop(&mut self) { + TENURING_SURVIVALS_TEST_OVERRIDE.with(|cell| cell.set(self.0)); + } +} + /// `PERRY_GC_TENURING_SURVIVALS=` pins the promotion age, overriding the /// adaptive threshold (#7432). Diagnostic only; unset means adaptive. /// @@ -469,20 +575,68 @@ pub(super) fn compute_target_survivals(eden_live_bytes: usize, desired_bytes: us /// Feed one finished copying-minor cycle into the feedback loop. /// `eden_live_bytes` is the cycle's Eden survivor influx (bytes moved out -/// of Eden, whether copied to a survivor space or promoted); -/// `copied_bytes` is what this cycle put into the to-survivor space; -/// `survivor_live_bytes` is what came back out of the from-survivor space -/// alive (numerator of the survival rate against the *previous* cycle's -/// `copied_bytes`). +/// of Eden, whether copied to a survivor space or promoted). +/// +/// The other two are **one cohort's** intake and that same cohort's survival, +/// and they must stay that way (#9851 follow-up): +/// `eden_copied_bytes` is what this cycle copied out of *Eden* into the +/// to-survivor space — a fresh cohort, no re-copies — and +/// `first_round_live_bytes` is what came back out of the from-survivor space +/// alive with a stored age of 1, i.e. members of the cohort that the +/// *previous* cycle's `eden_copied_bytes` counted. +/// +/// Why not the whole space. The survivor spaces are a strict semispace pair, +/// so the from-space at cycle N holds exactly what cycle N-1 copied, and +/// `survivor_live_bytes / prev_copied_bytes` is a well-formed survival ratio — +/// of the whole space. But *what that space contains* is set by the very +/// threshold this loop controls: at S<=2 it is one fresh cohort, at S=3-4 it +/// also holds age-2 and age-3 objects, which have already survived a round and +/// are therefore selected for longevity. Rating that mixture and concluding +/// "the aging round filters nothing" applies a measurement of an aged, +/// self-selected population to first-round cohorts. Measured on the compiled +/// claude-code TUI: a fresh cohort survives at 74 %, and the loop still reached +/// the lock's 90 % bar 8-12 times per four-turn run once #9851's clamp let the +/// ladder climb past 2. pub(super) fn retune_after_scavenge( eden_live_bytes: usize, - copied_bytes: usize, - survivor_live_bytes: usize, + eden_copied_bytes: usize, + first_round_live_bytes: usize, ) { retune_nursery_cap_scale(eden_live_bytes); let desired = desired_survivor_bytes(); let substantial = desired / 4; - let prev_copied = PREV_COPIED_BYTES.with(|c| c.replace(copied_bytes)); + let prev_cohort_copied = PREV_COPIED_BYTES.with(|c| c.replace(eden_copied_bytes)); + // A cohort went into the survivor space last cycle, so THIS cycle is the + // one that could follow it: from here on the loop has lifetime evidence and + // the occupancy rule is allowed to move off the floor. + let rated = prev_cohort_copied > 0; + let rounds_rated = if rated { + SURVIVOR_ROUND_MEASURED.with(|m| m.set(true)); + RATED_ROUNDS.with(|rounds| { + let next = rounds.get().saturating_add(1); + rounds.set(next); + next + }) + } else { + RATED_ROUNDS.with(Cell::get) + }; + let startup = if rated { + rounds_rated <= STARTUP_RATED_ROUNDS + } else { + rounds_rated < STARTUP_RATED_ROUNDS + }; + let survival_permille = if rated { + first_round_live_bytes.saturating_mul(1000) / prev_cohort_copied + } else { + 0 + }; + let mut evidence = TransitionEvidence { + rounds_rated, + streak: PROMOTE_LOCK_STREAK.with(Cell::get), + survival_permille, + copied_bytes: prev_cohort_copied, + startup, + }; let current = TENURING_SURVIVALS.with(Cell::get); if PROMOTE_LOCK.with(Cell::get) { @@ -494,11 +648,13 @@ pub(super) fn retune_after_scavenge( if streak >= RAISE_DEBOUNCE_CYCLES { PROMOTE_LOCK.with(|l| l.set(false)); UNLOCK_STREAK.with(|s| s.set(0)); + PROMOTE_LOCK_STREAK.with(|s| s.set(0)); RAISE_STREAK.with(|s| s.set(0)); // Resume the ladder one step up rather than snapping to the // ceiling; the normal debounced rise takes it the rest of // the way if the workload stays quiet. - set_survivals(current, 2, eden_live_bytes, "unlock"); + evidence.streak = 0; + set_survivals(current, 2, eden_live_bytes, "unlock", evidence); } else { UNLOCK_STREAK.with(|s| s.set(streak)); } @@ -508,18 +664,65 @@ pub(super) fn retune_after_scavenge( return; } - // Survival-rate lock: last cycle's survivor intake was substantial and + // Survival-rate lock: last cycle's FRESH COHORT was substantial and // (nearly) all of it came back out alive, so the aging round filters // nothing — every copied byte is a byte that will be promoted anyway. - if prev_copied >= substantial && survivor_live_bytes.saturating_mul(10) >= prev_copied * 9 { + // + // Both sides are scoped to that one cohort (#9851 follow-up). Rating the + // whole survivor space instead makes the ratio rise with the threshold + // this rule sets, because a higher threshold is precisely what keeps + // already-aged objects in the space; the rule then reads its own setting + // back as evidence. See `retune_after_scavenge`'s header. + if rated { + let qualifies = !startup + && prev_cohort_copied >= substantial + && first_round_live_bytes.saturating_mul(10) + >= prev_cohort_copied.saturating_mul(PROMOTE_LOCK_LIVE_TENTHS); + let streak = if qualifies { + PROMOTE_LOCK_STREAK.with(Cell::get).saturating_add(1) + } else { + 0 + }; + PROMOTE_LOCK_STREAK.with(|s| s.set(streak)); + evidence.streak = streak; + } + if evidence.streak >= PROMOTE_LOCK_RATED_ROUNDS { PROMOTE_LOCK.with(|l| l.set(true)); UNLOCK_STREAK.with(|s| s.set(0)); RAISE_STREAK.with(|s| s.set(0)); - set_survivals(current, 1, eden_live_bytes, "lock"); + set_survivals(current, 1, eden_live_bytes, "lock", evidence); return; } - let target = compute_target_survivals(eden_live_bytes, desired); + // #9851: the occupancy rule measures SPACE and may not conclude 1, which is + // a claim about LIFETIME — see `OCCUPANCY_MIN_SURVIVALS`. Deliberately + // clamped HERE and not inside `compute_target_survivals`: that pure function + // has a second caller, `full_seed_promotes_on_first_copy`, which gates the + // sweep seed on `... != 1` ("would occupancy alone already promote on first + // copy?"). Clamping the shared function would silently disarm the sweep + // seed, which is one of the two paths that IS allowed to reach 1. + // + // The startup follow-up makes that rule SYMMETRIC. `1 + desired / influx` + // returns the ceiling for a tiny influx and for a zero one, so on the first + // minors of a process — when the heap is nearly empty and no cohort has + // ever been followed — occupancy claims the maximum. That is the same + // category error in the other direction: a claim about LIFETIME from a + // measurement of SPACE, made before any evidence exists, and the expensive + // one, because every survivor is then copied up to three times before it + // may be promoted. Measured on the compiled claude-code TUI, the whole + // adaptive-vs-pinned difference was this startup excursion — + // `4 -> 2 (occupancy) -> 1 (lock)` inside turn 1 and nothing afterwards, + // worth +0.35..0.45 s at 3300 chars and +50 % at 400. + // + // So until one survivor round has actually been rated, occupancy holds at + // the floor: the lowest threshold that PRODUCES the measurement it needs to + // say anything at all. Evidence, not the ladder, is what lets it move. + let measured = SURVIVOR_ROUND_MEASURED.with(Cell::get); + let target = if measured { + compute_target_survivals(eden_live_bytes, desired).max(OCCUPANCY_MIN_SURVIVALS) + } else { + OCCUPANCY_MIN_SURVIVALS + }; let next = if target < current { RAISE_STREAK.with(|s| s.set(0)); target @@ -536,7 +739,7 @@ pub(super) fn retune_after_scavenge( RAISE_STREAK.with(|s| s.set(0)); current }; - set_survivals(current, next, eden_live_bytes, "occupancy"); + set_survivals(current, next, eden_live_bytes, "occupancy", evidence); } /// Grow the nursery cap one ×2 step (to at most ×4) when survivor influx @@ -583,7 +786,7 @@ fn retune_nursery_cap_scale(eden_live_bytes: usize) { /// promote-on-first-copy lock: ≥90% of the Eden bytes the sweep classified /// must have been live. That is the "the aging round would filter nothing" /// proof, measured directly instead of inferred from a survivor round-trip. -const FULL_SEED_LIVE_TENTHS: usize = 9; +const PROMOTE_LOCK_LIVE_TENTHS: usize = 9; /// Would a completed mark-sweep's Eden census justify promote-on-first-copy? /// @@ -607,7 +810,9 @@ const FULL_SEED_LIVE_TENTHS: usize = 9; /// program whose nursery is atypically mostly-live at one sweep promotes one /// Eden's worth of short-lived objects and pays an old-gen reclaim to get them /// back. Exposure is bounded by one nursery cap and by the existing unlock -/// path; requiring BOTH conditions is what keeps it narrow. +/// path; requiring BOTH conditions is what keeps it narrow. This pure census +/// predicate intentionally does not know process phase; the stateful caller +/// adds the same post-startup gate as the survivor-round path. pub(super) fn full_seed_promotes_on_first_copy( eden_live_bytes: usize, eden_dead_bytes: usize, @@ -618,7 +823,7 @@ pub(super) fn full_seed_promotes_on_first_copy( } let classified = eden_live_bytes.saturating_add(eden_dead_bytes); classified > 0 - && eden_live_bytes.saturating_mul(10) >= classified.saturating_mul(FULL_SEED_LIVE_TENTHS) + && eden_live_bytes.saturating_mul(10) >= classified.saturating_mul(PROMOTE_LOCK_LIVE_TENTHS) } /// Feed one finished mark-sweep's Eden census into the loop. `eden_live_bytes` @@ -627,11 +832,17 @@ pub(super) fn full_seed_promotes_on_first_copy( /// /// Callers must exclude budgeted cycles and cycles that ran the conservative /// native-stack scan — see the module header for why those two inputs are not -/// sound liveness measurements. +/// sound liveness measurements. Even a qualifying census is refused until the +/// lock's first two survivor cohorts have been rated, so a startup sweep cannot +/// bypass the survivor path's startup exclusion. pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_bytes: usize) { let already_locked = PROMOTE_LOCK.with(Cell::get); let desired = desired_survivor_bytes(); - let seeds = full_seed_promotes_on_first_copy(eden_live_bytes, eden_dead_bytes, desired); + let rounds_rated = RATED_ROUNDS.with(Cell::get); + let startup = rounds_rated < STARTUP_RATED_ROUNDS; + let census_qualifies = + full_seed_promotes_on_first_copy(eden_live_bytes, eden_dead_bytes, desired); + let seeds = !startup && census_qualifies; // Diagnostic, not a knob: print the census AND the verdict on every // mark-sweep, including refusals. A policy that silently declines is // indistinguishable from one that never ran (#7024/#7025), and the @@ -644,7 +855,7 @@ pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_byt eden_live_bytes * 100 / classified }; eprintln!( - "[gc-tenuring] sweep-seed eden_live_bytes={eden_live_bytes} eden_dead_bytes={eden_dead_bytes} live_pct={pct} desired={desired} seeds={seeds} already_locked={already_locked}" + "[gc-tenuring] sweep-seed eden_live_bytes={eden_live_bytes} eden_dead_bytes={eden_dead_bytes} live_pct={pct} desired={desired} census_qualifies={census_qualifies} startup={startup} seeds={seeds} already_locked={already_locked}" ); } if already_locked || !seeds { @@ -656,7 +867,27 @@ pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_byt RAISE_STREAK.with(|s| s.set(0)); // PREV_COPIED_BYTES is deliberately untouched: it is the survival-rate // lock's denominator, owned by the copying path. - set_survivals(current, 1, eden_live_bytes, "sweep-seed"); + let classified = eden_live_bytes.saturating_add(eden_dead_bytes); + let survival_permille = if classified == 0 { + 0 + } else { + eden_live_bytes.saturating_mul(1000) / classified + }; + set_survivals( + current, + 1, + eden_live_bytes, + "sweep-seed", + TransitionEvidence { + rounds_rated, + streak: PROMOTE_LOCK_STREAK.with(Cell::get), + survival_permille, + // The live Eden cohort is the volume the next copying minor would + // otherwise copy; the sweep reads that denominator one cycle early. + copied_bytes: eden_live_bytes, + startup, + }, + ); } fn diag_cap_scale(from: u8, to: u8, eden_live_bytes: usize) { @@ -667,28 +898,57 @@ fn diag_cap_scale(from: u8, to: u8, eden_live_bytes: usize) { } } -fn set_survivals(current: u8, next: u8, eden_live_bytes: usize, why: &str) { +#[derive(Clone, Copy)] +struct TransitionEvidence { + rounds_rated: u64, + streak: u8, + survival_permille: usize, + copied_bytes: usize, + startup: bool, +} + +fn set_survivals( + current: u8, + next: u8, + eden_live_bytes: usize, + why: &str, + evidence: TransitionEvidence, +) { if next == current { return; } TENURING_SURVIVALS.with(|s| s.set(next)); if crate::gc::gc_diag_enabled() { + let (copy_pause_us, tenuring_copied_bytes, promote_us, tenuring_promoted_bytes) = + super::instruments::tenuring_price_counters(); eprintln!( - "[gc-tenuring] survivals {} -> {} ({why}, eden_live_bytes={} desired={})", + "[gc-tenuring] survivals {} -> {} ({why}, eden_live_bytes={} desired={} rounds_rated={} streak={} survival_permille={} copied_bytes={} startup={} copy_pause_us={} tenuring_copied_bytes={} promote_us={} tenuring_promoted_bytes={})", current, next, eden_live_bytes, - desired_survivor_bytes() + desired_survivor_bytes(), + evidence.rounds_rated, + evidence.streak, + evidence.survival_permille, + evidence.copied_bytes, + evidence.startup, + copy_pause_us, + tenuring_copied_bytes, + promote_us, + tenuring_promoted_bytes, ); } } #[cfg(test)] pub(super) fn reset_for_test() { - TENURING_SURVIVALS.with(|s| s.set(GC_TENURING_SURVIVALS_MAX)); + TENURING_SURVIVALS.with(|s| s.set(OCCUPANCY_MIN_SURVIVALS)); + SURVIVOR_ROUND_MEASURED.with(|m| m.set(false)); RAISE_STREAK.with(|s| s.set(0)); PROMOTE_LOCK.with(|l| l.set(false)); UNLOCK_STREAK.with(|s| s.set(0)); + RATED_ROUNDS.with(|s| s.set(0)); + PROMOTE_LOCK_STREAK.with(|s| s.set(0)); PREV_COPIED_BYTES.with(|c| c.set(0)); NURSERY_CAP_SCALE.with(|s| s.set(1)); CAP_GROW_STREAK.with(|s| s.set(0)); @@ -710,6 +970,20 @@ mod tests { const MB: usize = 1024 * 1024; + /// Advance past the startup boundary with cohorts that visibly die, so + /// tests below can exercise post-startup policy without contributing to + /// the promote-lock streak themselves. + fn finish_startup_with_mortality(desired: usize) { + let cohort = 3 * desired; + retune_after_scavenge(16 * desired, cohort, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * desired, cohort, 0); + } + assert_eq!(RATED_ROUNDS.with(Cell::get), STARTUP_RATED_ROUNDS); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + assert!(!PROMOTE_LOCK.with(Cell::get)); + } + /// #7929: the constant band must buy a CONSTANT NUMBER OF OBJECTS. /// /// The discriminating quantity is deliberately `band / mean` (the object @@ -878,22 +1152,35 @@ mod tests { fn drops_immediately_and_rises_debounced() { reset_for_test(); let desired = desired_survivor_bytes(); - assert_eq!(tenuring_survivals(), 4); - - // Heavy influx: instant drop to 1. + // Power-on is the FLOOR now, not the ceiling (startup follow-up): the + // ladder may not claim a lifetime in either direction without evidence. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + + // Give the loop its evidence, because this test is about the ladder's + // ASYMMETRY and not about the startup gate. Two cycles with a cohort + // that fully dies: the second rates the first, so a survivor round has + // been measured, and 0 % survival keeps the lock out of it. + retune_after_scavenge(desired * 2, 3 * desired, 0); + retune_after_scavenge(desired * 2, 3 * desired, 0); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + + // Heavy influx: instant drop, no debounce. #9851 changed the FLOOR this + // lands on (2, not 1 — the occupancy rule may not claim a lifetime), not + // the asymmetry this test is named for: 4 -> 2 in one cycle is the same + // "drops immediately" property that 4 -> 1 was. retune_after_scavenge(desired * 2, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // One quiet cycle: no rise yet (debounce). retune_after_scavenge(0, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // Second quiet cycle: rise by exactly one step, not to the target. retune_after_scavenge(0, 0, 0); - assert_eq!(tenuring_survivals(), 2); + assert_eq!(tenuring_survivals(), 3); // Heavy again: streak resets and threshold drops straight back. retune_after_scavenge(desired * 2, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // Sustained quiet recovers to the ceiling two cycles per step. for _ in 0..6 { @@ -911,10 +1198,13 @@ mod tests { // every cycle even while the cap scale walks up underneath it. An // influx only marginally above the base desired is a different case: // the growing cap re-classifies it as moderate, which is correct. + // #9851: the fixed point is now the occupancy floor (2) rather than 1. + // Fixed-POINTNESS is what this test protects — no oscillation while the + // cap scale walks up underneath — and that is unchanged. let heavy = gc_scavenge_nursery_cap_bytes(); for _ in 0..10 { retune_after_scavenge(heavy, 0, 0); - assert_eq!(tenuring_survivals(), 1); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); } assert_eq!( scavenge_nursery_cap_effective_bytes(), @@ -924,6 +1214,260 @@ mod tests { reset_for_test(); } + /// #9851, both halves of the rule in one test, in the #7909 two-phase shape + /// so the decline is ATTRIBUTED rather than merely absent. + /// + /// Phase 1 — the occupancy rule alone, on an influx far above `desired`, + /// must stop at 2 and NOT claim promote-on-first-copy. That is the whole + /// change: 2 is the lowest threshold that still puts a cohort through the + /// survivor space, so the loop keeps producing the measurement that could + /// refute it. + /// + /// Phase 2 — the same heap, once a substantial cohort HAS come back fully + /// alive, must still reach 1 through the survival-rate lock. The rule + /// removes an unmeasured conclusion, not the measured one, and this half is + /// what makes it self-limiting rather than a blanket floor. + /// + /// Sabotage: drop the `.max(OCCUPANCY_MIN_SURVIVALS)` in + /// `retune_after_scavenge` and phase 1 fails (the loop reports 1 with no + /// evidence). Drop the lock instead and phase 2 fails. + #[test] + fn occupancy_alone_never_claims_promote_on_first_copy_but_the_lock_still_can() { + reset_for_test(); + let d = desired_survivor_bytes(); + + // Phase 1: influx 16x the desired survivor size — the occupancy formula + // computes 1 (integer division: 1 + desired/influx). No cohort has been + // rated yet, so there is NO lifetime evidence on this heap. + assert_eq!( + compute_target_survivals(16 * d, d), + 1, + "precondition: the occupancy ARITHMETIC still computes 1 — this \ + change clamps what the loop may do with it, not the formula" + ); + for _ in 0..5 { + retune_after_scavenge(16 * d, 0, 0); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "occupancy measures SPACE and must not conclude promote-on-first-copy" + ); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "and it must not have taken the lock's route to get there" + ); + + // Phase 2: now substantial cohorts go through the survivor space and + // come back fully alive. Startup evidence is excluded, then exactly K + // steady ratings must still reach 1. + retune_after_scavenge(16 * d, 3 * d, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * d, 3 * d, 3 * d); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + for round in 1..=PROMOTE_LOCK_RATED_ROUNDS { + retune_after_scavenge(16 * d, 3 * d, 3 * d); + if round < PROMOTE_LOCK_RATED_ROUNDS { + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + } + assert_eq!( + tenuring_survivals(), + 1, + "K steady substantial cohorts that fully survive must still lock \ + promote-on-first-copy" + ); + assert!(PROMOTE_LOCK.with(Cell::get), "...through the lock"); + reset_for_test(); + } + + /// STARTUP FOLLOW-UP — the occupancy rule may not claim the CEILING either. + /// + /// `1 + desired / influx` returns the ceiling for a tiny influx and for a + /// zero one, so on the first minors of a process — heap nearly empty, no + /// cohort ever followed — occupancy claims the maximum. That is the same + /// category error as claiming 1: a statement about LIFETIME derived from a + /// measurement of SPACE, made before any evidence exists. It is also the + /// expensive direction, because every survivor is then copied up to three + /// times before it may be promoted. + /// + /// Measured on the compiled claude-code TUI, this was the WHOLE difference + /// between the adaptive loop and a pinned threshold: a single startup + /// excursion `4 -> 2 (occupancy) -> 1 (lock)` inside turn 1, nothing + /// afterwards, worth +0.35..0.45 s at 3300 characters and +50 % at 400. + /// + /// Sabotage: delete the `SURVIVOR_ROUND_MEASURED` gate in + /// `retune_after_scavenge` (or restore the power-on value to + /// `GC_TENURING_SURVIVALS_MAX`) and phase 1 fails — the loop reports the + /// ceiling on a heap where nothing has ever been rated. + #[test] + fn occupancy_may_not_claim_the_ceiling_before_any_round_is_measured() { + reset_for_test(); + let d = desired_survivor_bytes(); + + // Precondition: the ARITHMETIC still says "ceiling" for a startup-sized + // influx. This change gates what the loop may do with that, exactly as + // #9851 did at the other end of the range. + assert_eq!(compute_target_survivals(0, d), GC_TENURING_SURVIVALS_MAX); + assert_eq!( + compute_target_survivals(d / 64, d), + GC_TENURING_SURVIVALS_MAX + ); + + // Phase 1: power-on, then many startup-shaped minors — tiny influx, + // nothing copied, so nothing rateable. The loop must sit at the floor + // and never climb. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + for _ in 0..8 { + retune_after_scavenge(d / 64, 0, 0); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "no survivor round has been rated, so occupancy has no lifetime \ + evidence and may not leave the floor" + ); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "and it must not have reached the floor via the lock either" + ); + + // Phase 2: once a cohort has actually gone through the survivor space + // and been followed, the ladder is allowed to move again. A cohort that + // fully dies keeps the lock out, so what is observed here is the + // occupancy rule being re-enabled and nothing else. + retune_after_scavenge(d / 64, 3 * d, 0); + retune_after_scavenge(d / 64, 3 * d, 0); + for _ in 0..8 { + retune_after_scavenge(d / 64, 0, 0); + } + assert_eq!( + tenuring_survivals(), + GC_TENURING_SURVIVALS_MAX, + "with a round measured and the influx quiet, the debounced rise must \ + still reach the ceiling — the gate delays the claim until there is \ + evidence, it does not remove the ladder" + ); + reset_for_test(); + } + + /// #9851: a cohort that DIES in its survivor round must keep the loop at the + /// occupancy floor rather than being locked to 1 — the case cc actually is. + /// Measured there: 26.1 % of each cohort dies in one survivor round, in + /// steady state, on 393 samples; the lock needs >=90 % survival, so it + /// correctly stays out. + /// + /// The arguments are the FRESH COHORT's intake and survival (#9851 + /// follow-up). Fed the whole survivor space instead — which is what the + /// call site used to pass — this same heap locks, because above a threshold + /// of 2 that space also holds objects already selected for longevity. That + /// is not a hypothetical: on cc the clamp alone left 85 % of promotion at + /// S=1, reached through this lock 8-12 times per four-turn run. + #[test] + fn a_cohort_that_dies_in_its_round_holds_at_the_occupancy_floor() { + reset_for_test(); + let d = desired_survivor_bytes(); + // Heavy influx (occupancy says 1) AND a substantial cohort of which + // ~26 % dies — cc's steady state, in miniature. + for _ in 0..8 { + retune_after_scavenge(16 * d, 4 * d, 3 * d); + } + assert!( + !PROMOTE_LOCK.with(Cell::get), + "74 % survival is below the lock's 90 % bar: the lock must stay out" + ); + assert_eq!( + tenuring_survivals(), + OCCUPANCY_MIN_SURVIVALS, + "so the loop holds at the occupancy floor and keeps aging the cohort" + ); + reset_for_test(); + } + + /// STARTUP EVIDENCE LOCK: two fully-surviving startup cohorts may not + /// contribute to the steady-state streak. The first post-startup survivor + /// therefore leaves the streak at one and S at the occupancy floor. + /// + /// Sabotage: remove `!startup` from the qualifying predicate. The two + /// startup ratings then count as the first two streak members and the final + /// call reaches K, latching S=1 and failing this test. + #[test] + fn startup_shaped_survivors_do_not_contribute_to_the_lock_streak() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + + // Establish the denominator, then rate two startup cohorts at 100%. + retune_after_scavenge(16 * d, cohort, 0); + for _ in 0..STARTUP_RATED_ROUNDS { + retune_after_scavenge(16 * d, cohort, cohort); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + + // One genuinely post-startup rating is not a K-round window. + retune_after_scavenge(16 * d, cohort, cohort); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 1); + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + + /// Kill condition from the design: the lock remains reachable, in bounded + /// time, for a workload whose steady cohorts genuinely do not die. + /// + /// Sabotage: raise `PROMOTE_LOCK_RATED_ROUNDS` or fail to advance the + /// streak; the exact-K final assertion fails. + #[test] + fn k_steady_fully_surviving_rounds_latch_promote_on_first_copy() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + finish_startup_with_mortality(d); + + for round in 1..=PROMOTE_LOCK_RATED_ROUNDS { + retune_after_scavenge(16 * d, cohort, cohort); + if round < PROMOTE_LOCK_RATED_ROUNDS { + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + } + } + assert!(PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), 1); + reset_for_test(); + } + + /// A mortality observation breaks consecutiveness even when the process + /// has accumulated K qualifying observations in total. + /// + /// Sabotage: retain the streak on a rated below-bar round. The last survivor + /// becomes the Kth streak member and this test observes an S=1 latch. + #[test] + fn mortality_inside_the_steady_window_resets_the_lock_streak() { + reset_for_test(); + let d = desired_survivor_bytes(); + let cohort = 3 * d; + finish_startup_with_mortality(d); + + for _ in 0..PROMOTE_LOCK_RATED_ROUNDS - 1 { + retune_after_scavenge(16 * d, cohort, cohort); + } + assert_eq!( + PROMOTE_LOCK_STREAK.with(Cell::get), + PROMOTE_LOCK_RATED_ROUNDS - 1 + ); + + retune_after_scavenge(16 * d, cohort, cohort / 2); + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 0); + retune_after_scavenge(16 * d, cohort, cohort); + + assert_eq!(PROMOTE_LOCK_STREAK.with(Cell::get), 1); + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + #[test] fn survival_rate_lock_breaks_a_saturated_pipeline() { reset_for_test(); @@ -938,11 +1482,13 @@ mod tests { 2, "first cycle has no prior intake to rate, so occupancy decides" ); - retune_after_scavenge(influx, 3 * d, 3 * d); + for _ in 0..STARTUP_RATED_ROUNDS + u64::from(PROMOTE_LOCK_RATED_ROUNDS) { + retune_after_scavenge(influx, 3 * d, 3 * d); + } assert_eq!( tenuring_survivals(), 1, - "a substantial intake that fully survives its round must lock promote-on-first-copy" + "the bounded steady window must lock" ); // The lock holds through occupancy readings that would say S=2. for _ in 0..5 { @@ -999,15 +1545,23 @@ mod tests { let d = desired_survivor_bytes(); // Medium-lived objects: a substantial intake of which only half // survives its survivor round. Aging is filtering — the lock must - // stay out and the occupancy ladder must decide. + // stay out and the occupancy ladder must age from the power-on floor. + let mut seen = Vec::new(); for _ in 0..6 { retune_after_scavenge(d / 2, d / 2, d / 4); assert!( - tenuring_survivals() >= 3, - "a cohort that dies in the survivor space must keep aging (got {})", - tenuring_survivals() + !PROMOTE_LOCK.with(Cell::get), + "50% survival is below the lock's 90% bar" ); + seen.push(tenuring_survivals()); } + assert_eq!( + seen, + [2, 2, 3, 3, 3, 3], + "power-on is the floor now, not the ceiling: after a survivor round \ + is measured, the debounced occupancy ladder must keep the dying \ + cohort aging rather than claim promote-on-first-copy" + ); reset_for_test(); } @@ -1079,26 +1633,49 @@ mod tests { // ── #7598's mark-sweep seed ───────────────────────────────────────── // - // The lock above cannot engage before the SECOND copying minor. These - // exercise the seed that reads the same proof off a completed mark-sweep, - // one collection earlier. + // The survivor lock needs a previous copying minor. These exercise the seed + // that reads the same proof off a completed mark-sweep after startup, one + // collection earlier, plus the startup refusal itself. + /// A sweep census may satisfy both of the seed's existing conditions and + /// still be forbidden to decide S=1 while the process is in startup. + /// + /// Sabotage: remove the startup conjunct from `seeds`; this qualifying + /// census immediately latches S=1 and fails both final assertions. #[test] - fn sweep_seed_decides_before_the_first_copying_minor_snapshots_the_threshold() { + fn sweep_seed_cannot_latch_from_a_startup_census() { + reset_for_test(); + let d = desired_survivor_bytes(); + let eden_live = 4 * d; + assert!( + full_seed_promotes_on_first_copy(eden_live, 0, d), + "precondition: both original sweep-seed conditions must qualify" + ); + + seed_promote_lock_from_sweep(eden_live, 0); + + assert!(!PROMOTE_LOCK.with(Cell::get)); + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); + reset_for_test(); + } + + #[test] + fn sweep_seed_decides_before_a_post_startup_copying_minor_snapshots_the_threshold() { // `copying.rs` snapshots `tenuring_survivals()` in - // `CopyingNurseryCollector::new`, so the only value that can change - // what the first big minor does is the one standing BEFORE any - // `retune_after_scavenge` for that cycle has run. That is precisely - // what the survival-rate lock cannot reach and this seed can. + // `CopyingNurseryCollector::new`; after startup, a completed sweep may + // still decide what the next copying minor does one cycle earlier than + // a survivor round-trip. reset_for_test(); let d = desired_survivor_bytes(); let eden_live = d * 4; assert_eq!( tenuring_survivals(), - 4, - "with no input the loop is at the ceiling: the wasted copy state" + OCCUPANCY_MIN_SURVIVALS, + "power-on is the floor now, not the ceiling: with no lifetime \ + evidence the loop may not claim either extreme" ); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(eden_live, eden_live / 50); assert_eq!( tenuring_survivals(), @@ -1118,6 +1695,7 @@ mod tests { let d = desired_survivor_bytes(); let eden_live = d * 4; let eden_dead = eden_live * 9; + finish_startup_with_mortality(d); assert_eq!( compute_target_survivals(eden_live, d), 1, @@ -1127,8 +1705,9 @@ mod tests { seed_promote_lock_from_sweep(eden_live, eden_dead); assert_eq!( tenuring_survivals(), - 4, - "10% Eden survival must not seed promote-on-first-copy" + OCCUPANCY_MIN_SURVIVALS, + "10% Eden survival must leave the loop at the power-on floor, not \ + seed promote-on-first-copy by claiming a threshold below 2" ); reset_for_test(); } @@ -1141,8 +1720,12 @@ mod tests { // reading, and seeding off it would lock every program at S=1. reset_for_test(); let d = desired_survivor_bytes(); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(d / 8, 0); - assert_eq!(tenuring_survivals(), 4); + // Unchanged from power-on, which is the floor now rather than the + // ceiling (startup follow-up). The property under test is that the + // sweep seed REFUSED — it left the threshold where it found it. + assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); reset_for_test(); } @@ -1174,6 +1757,7 @@ mod tests { // differently at S=1 than at S=4 and would oscillate. reset_for_test(); let d = desired_survivor_bytes(); + finish_startup_with_mortality(d); seed_promote_lock_from_sweep(d * 4, 0); assert_eq!(tenuring_survivals(), 1); diff --git a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs index 9abfa5a6e2..b8259420ac 100644 --- a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs +++ b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs @@ -27,22 +27,24 @@ fn heavy_influx_lowers_threshold_and_promotes_next_cycle() { let _guard = CopyingNurseryTestGuard::new(SLOTS); assert_eq!( crate::gc::tenuring::tenuring_survivals(), - GC_COPY_PROMOTION_SURVIVALS, - "guard must start every test at the power-on threshold" + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS, + "guard must start every test at the power-on floor, not the ceiling" ); fill_slots_with_heavy_influx(); let before = (js_shadow_slot_get(0) & POINTER_MASK) as usize; assert!(crate::arena::pointer_in_nursery(before)); - // Cycle 1 runs at the power-on threshold: the cohort is copied into a - // survivor space (ages to 1), and its influx re-tunes the threshold down - // to promote-on-first-copy. + // Cycle 1 runs at the power-on floor: the cohort is copied into a survivor + // space (ages to 1), and heavy influx must not take occupancy below that + // floor by claiming promote-on-first-copy without lifetime evidence. let _ = gc_collect_minor(); assert_eq!( crate::gc::tenuring::tenuring_survivals(), - 1, - "a >desired Eden survivor influx must drop the threshold to 1" + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS, + "a >desired Eden survivor influx must drop the threshold to the \ + occupancy floor (#9851: the occupancy rule measures space and may not \ + claim promote-on-first-copy, which is a claim about lifetime)" ); let after_first = (js_shadow_slot_get(0) & POINTER_MASK) as usize; assert!( @@ -51,7 +53,11 @@ fn heavy_influx_lowers_threshold_and_promotes_next_cycle() { ); // Cycle 2 promotes the whole cohort instead of re-copying it: this is - // the ping-pong the adaptive threshold exists to break. + // the ping-pong the adaptive threshold exists to break. #9851 did NOT + // weaken this half — the cohort was copied once in cycle 1, so its + // `next_age` here is 2, which still satisfies `next_age >= 2`. The test's + // named invariant ("lowers threshold AND promotes next cycle") is intact; + // only the literal threshold moved. let _ = gc_collect_minor(); for slot in 0..SLOTS { let addr = (js_shadow_slot_get(slot) & POINTER_MASK) as usize; @@ -215,7 +221,13 @@ fn quiet_cycles_restore_power_on_threshold_debounced() { fill_slots_with_heavy_influx(); let _ = gc_collect_minor(); - assert_eq!(crate::gc::tenuring::tenuring_survivals(), 1); + // #9851: the occupancy floor, not 1. What this test protects — a DEBOUNCED + // restore, at most one step per cycle, ending at the power-on threshold — + // is asserted structurally below and is unchanged. + assert_eq!( + crate::gc::tenuring::tenuring_survivals(), + crate::gc::tenuring::OCCUPANCY_MIN_SURVIVALS + ); // Promote the cohort out of the nursery so later cycles are quiet. let _ = gc_collect_minor(); diff --git a/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs b/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs index 45c6c25f82..18963db4af 100644 --- a/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs +++ b/crates/perry-runtime/src/gc/tests/copying/promoted_remembered_7803.rs @@ -62,6 +62,8 @@ fn young_padded_closure_capturing(bits: u64) -> usize { #[test] fn drain_promoted_parent_keeps_its_young_child_edge_remembered() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); // parent captures a young leaf; intermediate captures parent. Only the // INTERMEDIATE is rooted, so the parent is reached — and, on the @@ -95,9 +97,8 @@ fn drain_promoted_parent_keeps_its_young_child_edge_remembered() { deref(capture_bits_of(spacer)) }; - // Age everyone to the brink of promotion (power-on threshold: promote on - // the fourth survival — pinned by - // `test_copying_minor_promotes_survivor_on_fourth_survival`). + // Age everyone to the explicitly pinned promotion boundary: the fourth + // survival. This test exercises drain promotion at S=4, not power-on. for _ in 0..3 { let _ = gc_collect_minor(); } diff --git a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs index aed0b6c681..b6104a63f5 100644 --- a/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs +++ b/crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs @@ -3,6 +3,8 @@ use super::*; #[test] fn test_copying_minor_promotes_survivor_on_fourth_survival() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); js_shadow_slot_set(0, ptr_bits(child)); @@ -53,6 +55,8 @@ fn test_copying_minor_preserves_old_page_accounting_for_defrag_policy() { pinned_header: std::ptr::null_mut(), }; let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); clear_marks(); clear_mark_seeds(); @@ -176,6 +180,8 @@ fn test_copying_minor_preserves_old_page_accounting_for_defrag_policy() { #[test] fn test_copying_minor_sticky_old_to_survivor_edge_promotes_on_fourth_cycle() { let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); let (old_arr, elements) = unsafe { alloc_old_test_array(1) }; unsafe { @@ -964,6 +970,8 @@ fn test_movable_regexp_evacuation_migrates_all_address_owned_state() { #[test] fn test_copied_minor_promotable_census_filtered_walk_matches_unfiltered() { let _guard = CopyingNurseryTestGuard::new(1); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let child = young_leaf(); js_shadow_slot_set(0, ptr_bits(child)); @@ -1065,3 +1073,85 @@ fn nursery_regexp_that_dies_young_is_finalized_by_the_copied_minor() { ); js_shadow_slot_set(0, 0); } + +/// #9851 follow-up — THE PREMISE OF THE LOCK REWIRE, on a real heap. +/// +/// The survival-rate lock used to rate `survivor_live_bytes` (every live byte +/// leaving the from-survivor space, of any age) against the previous cycle's +/// whole `copied_bytes`. Those two scopes match — the survivor spaces are a +/// strict semispace pair, so the from-space holds exactly what the last cycle +/// copied — and the ratio is well-formed. What is wrong is *which population* +/// it rates, and that is chosen by the threshold the lock itself sets: at a +/// threshold of 2 the space holds one fresh cohort, at 3 or 4 it also holds +/// objects that have already survived a round and are therefore selected for +/// longevity. +/// +/// This test pins the fact that makes the rewire meaningful rather than a +/// rename: **at a threshold above 2 the whole-space number and the fresh-cohort +/// number are different numbers**, with the aged resident in the first and not +/// in the second. On cc that difference is the whole finding — the aggregate +/// clears the lock's 90 % bar while a fresh cohort survives at 74 %. +/// +/// Shape: at an explicitly pinned threshold above 2 (promote on the 4th +/// survival) two rooted objects are introduced one cycle apart, so by the +/// third minor the from-survivor space holds one age-2 object and one age-1 +/// object. +#[test] +fn the_survivor_space_and_the_fresh_cohort_are_different_numbers_above_threshold_two() { + // TWO shadow slots: the test needs two independently rooted objects + // introduced one cycle apart, so that the survivor space holds two age + // classes at once. With one slot B is unrooted, dies immediately, and the + // fresh-cohort number is trivially zero. + let _guard = CopyingNurseryTestGuard::new(2); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); + + // Cycle 1: A enters the survivor space from Eden. The from-survivor space + // was empty, so both numbers are zero and the cohort is all of nothing. + let a = young_leaf(); + js_shadow_slot_set(0, ptr_bits(a)); + let _ = gc_collect_minor(); + let (_, _, survivor_live_1, first_round_1) = crate::gc::copying::test_last_cohort_split(); + assert_eq!( + (survivor_live_1, first_round_1), + (0, 0), + "cycle 1 evacuates Eden only: nothing came out of the survivor space" + ); + + // Cycle 2: A is re-copied (age 1 -> 2) and B enters from Eden. The + // from-survivor space held ONLY A, which is a first-round object, so the + // two numbers must still agree — this is the regime the lock was designed + // in, and the assertion that the split is not simply always different. + let b = young_leaf(); + js_shadow_slot_set(1, ptr_bits(b)); + let _ = gc_collect_minor(); + let (_, _, survivor_live_2, first_round_2) = crate::gc::copying::test_last_cohort_split(); + assert!( + survivor_live_2 > 0, + "A must have come back out of the survivor space" + ); + assert_eq!( + survivor_live_2, first_round_2, + "with a single generation resident the whole-space number IS the \ + fresh-cohort number — at threshold <= 2 the old rule was correct" + ); + + // Cycle 3: the from-survivor space now holds A (age 2) and B (age 1). + // `survivor_live_bytes` counts both; the fresh cohort is B alone. + let _ = gc_collect_minor(); + let (_, _, survivor_live_3, first_round_3) = crate::gc::copying::test_last_cohort_split(); + assert!( + first_round_3 > 0, + "B is a first-round survivor and must be counted as one" + ); + assert!( + survivor_live_3 > first_round_3, + "the aged resident A is in the whole-space number and must NOT be in \ + the fresh-cohort number: whole-space {survivor_live_3}, cohort \ + {first_round_3}. If these are equal the lock is still rating a \ + population its own threshold selected." + ); + + js_shadow_slot_set(0, 0); + js_shadow_slot_set(1, 0); +} diff --git a/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs b/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs index 581e22d237..1ea175c046 100644 --- a/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs +++ b/crates/perry-runtime/src/gc/tests/copying/weak_holder_registry.rs @@ -242,6 +242,8 @@ fn test_full_weak_processing_work_is_independent_of_unrelated_heap_size() { #[test] fn test_registry_tracks_holder_across_three_moving_minors() { let _guard = CopyingNurseryTestGuard::new(3); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let map = crate::weakref::js_weakmap_new(); let live_key = crate::object::js_object_alloc(0, 0); diff --git a/crates/perry-runtime/src/gc/tests/oldgen.rs b/crates/perry-runtime/src/gc/tests/oldgen.rs index 589adc3132..79b1453fd2 100644 --- a/crates/perry-runtime/src/gc/tests/oldgen.rs +++ b/crates/perry-runtime/src/gc/tests/oldgen.rs @@ -1328,6 +1328,8 @@ fn test_minor_skips_whole_heap_old_to_young_rebuild() { #[test] fn test_minor_preserves_old_to_young_edge_across_minors() { let _isolation = copying_nursery_isolation_lock(); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); let _barrier_guard = GeneratedWriteBarrierTestGuard::active(); reset_remembered_set(); diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs index 29217da550..cc65a5e865 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/hook_dispatch_handles.rs @@ -131,6 +131,8 @@ fn test_timer_tick_roots_callback_args_and_previous_context_across_hooks() { let _async_hook_guard = AsyncHookRuntimeTestGuard::new(); let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); register_runtime_handle_root_scanner_for_tests(); gc_register_mutable_root_scanner(crate::async_hooks::scan_async_hooks_roots_mut); @@ -289,6 +291,8 @@ fn test_array_map_runtime_handles_survive_callback_copied_minor_gc() { #[test] fn test_map_materializers_runtime_handles_survive_copied_minor_gc() { let _guard = CopyingNurseryTestGuard::new(0); + let _tenuring = + crate::gc::tenuring::set_survivals_for_test(crate::gc::tenuring::GC_TENURING_SURVIVALS_MAX); let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); register_runtime_handle_root_scanner_for_tests(); diff --git a/crates/perry-runtime/src/gc/tests/support.rs b/crates/perry-runtime/src/gc/tests/support.rs index 130c0f7a08..d0cbecce02 100644 --- a/crates/perry-runtime/src/gc/tests/support.rs +++ b/crates/perry-runtime/src/gc/tests/support.rs @@ -414,9 +414,10 @@ pub(crate) struct CopyingNurseryTestGuard { } pub(super) fn reset_copying_nursery_runtime_test_state() { - // Age-sensitive tests assume the power-on tenuring threshold (promote at - // the 4th survival); pin it so a heavy-influx test earlier on the same - // thread cannot leak a lowered adaptive threshold in. + // Restore the adaptive policy to its power-on floor. Tests of mechanisms + // that require a particular promotion age pin it explicitly with + // `tenuring::set_survivals_for_test`, so a power-on policy change cannot + // silently change the mechanism they exercise. crate::gc::tenuring::reset_for_test(); // #7645: the young-pin latch is process-wide and monotone, so one // earlier pinning test would otherwise leave every later copying test diff --git a/scripts/gc_runtime_root_holders.json b/scripts/gc_runtime_root_holders.json index f1582b60c2..5ef98d06c2 100644 --- a/scripts/gc_runtime_root_holders.json +++ b/scripts/gc_runtime_root_holders.json @@ -266,6 +266,30 @@ "verdict": "not_a_gc_pointer", "why": "Boolean census request latch, set by census_arm and consumed at full-sweep entry; contains no address or JS value." }, + { + "file": "crates/perry-runtime/src/gc/copying.rs", + "name": "LAST_COHORT_SPLIT", + "verdict": "test_only", + "why": "Declared under #[cfg(test)] at crates/perry-runtime/src/gc/copying.rs:1951; this Cell<(usize, usize, usize, usize)> holds only the byte counts of the last survivor-space/fresh-cohort split for the tenuring lock tests. It is absent from shipped binaries." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "SURVIVOR_ROUND_MEASURED", + "verdict": "not_a_gc_pointer", + "why": "Declared at crates/perry-runtime/src/gc/tenuring.rs:225; this Cell records whether any survivor round has been rated on this thread, gating the occupancy rule off its floor. A boolean, never a heap pointer." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "RATED_ROUNDS", + "verdict": "not_a_gc_pointer", + "why": "Steady-state tenuring evidence counter. Cell stores only how many survivor cohorts have been rated; it never holds an address or NaN-boxed value." + }, + { + "file": "crates/perry-runtime/src/gc/tenuring.rs", + "name": "PROMOTE_LOCK_STREAK", + "verdict": "not_a_gc_pointer", + "why": "Tenuring evidence streak. Cell stores only the count of consecutive qualifying rated cohorts; it never holds an address or NaN-boxed value." + }, { "file": "crates/perry-runtime/src/gc/census.rs", "name": "LABEL", @@ -276,7 +300,7 @@ "file": "crates/perry-runtime/src/gc/census.rs", "name": "PASS1_MARKED", "verdict": "non_moving_snapshot", - "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged.", + "why": "Real GC header addresses, deliberately untraced so the diagnostic does not keep its observed objects alive. Populated only at the end of mark propagation of a synchronous full cycle; consumed at sweep entry in the same run_to_completion invocation. The intervening full-cycle phases do not relocate or run JS callbacks. The Vec is used for membership comparisons and dropped with the census before sweep. Budgeted and minor cycles skip both boundaries. Pin re-audited 2026-09-05 after #9760 touched `gc/mod.rs`: that change is `mod heap_stats;` plus a `pub(crate) use` re-export and alters no mark/sweep control flow. `heap_stats()` is reached only from `js_bun_jsc_heap_stats` (the JS-facing `bun:jsc.heapStats()`), i.e. from mutator code, never inside a cycle, and its own module contract forbids allocation or collection during its walk. The mark-complete \u2192 sweep-entry window is unchanged. Re-audited 2026-09-05 (train125) after #9769 and #9771 touched pinned files. #9769 adds one `reg_scanner!` registration to `gc/mod.rs`; #9771 adds a feature-gated `alloc_census_init()` there and a feature-gated Rust-heap dump inside `take_census`. `alloc-census` is not in the default feature set, and decisively: `census_take_if_armed_at_full_sweep_start` does `PASS1_MARKED.with(|p| p.borrow_mut().take())` BEFORE calling `take_census`, so the snapshot has already left the thread-local by the time #9771's code runs \u2014 it cannot affect the window. Neither change alters mark/sweep control flow. Re-audited 2026-09-06 after #9831 touched `gc/policy.rs`. Its hunks are (a) the tiny-parse pressure guard's pricing (`tiny_parse_pressure_headroom_bytes`, `tiny_parse_pressure_due*`, a `Cell` byte-count base) consulted from JSON.parse's mutator-side boundaries (`gc_bump_malloc_trigger`, `gc_collect_pending_suppressed_parse`, `gc_schedule_parse_boundary_collection_if_pressure`), none of which is reachable from inside a cycle, and (b) one extra `Cell` store in `note_collection_finished_arena_occupancy`, which runs from `publish_reclaim_outcome` in the Publish subphase \u2014 after `step_sweep` has already consumed the snapshot. Mark/sweep control flow between `census_pass1_if_armed` and `census_take_if_armed_at_full_sweep_start` is untouched. Re-audited 2026-09-05 (train126) after #9755 restructured `gc/cycle.rs`. Its hunks are all root-scan machinery (`RootScanSubphase`, `RootScanCycleState`, the mutable-scanner iteration state), which runs BEFORE mark propagation completes; `gc/mod.rs` gains only a `mod young_log;` declaration. The bracketing is unchanged \u2014 `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep` \u2014 and a synchronous full mark-sweep still moves nothing between them. Re-pinned 2026-09-05 for the #9740 hot-TLS conversion of this file: the sole change is `thread_local!` \u2192 `crate::perry_thread_local!`, a macro-name swap with identical declaration syntax and `.with()` call sites. No control flow, no phase boundary, and no storage semantics change. Re-audited 2026-09-06 (train128) after #9794's GC diagnostics touched `gc/mod.rs` and `gc/policy.rs`: both gain diagnostic module declarations and counters only \u2014 no mark/sweep control flow, and the census bracketing in `step_mark_propagation` / `step_sweep` is unchanged. Re-audited for #9794's GC diagnostics: `gc/mod.rs` gains `mod diag_sites;` / `mod survival_diag;`, a re-export, a `diag_sites::full_started(...)` call at TRIGGER time (before mark propagation begins), and exit-time reporting. Nothing executes between mark-complete and sweep-entry, so the window is unchanged. Re-audited 2026-09-06 for the retained array-growth verifier fix: the cycle.rs change passes the existing non-copying evacuation verifier an explicit all-forwarded policy. That call remains in minor finalization, outside the synchronous full-cycle census window; its root and heap reads do not allocate GC objects, move objects, or invoke JS callbacks. The mark-complete and sweep-entry boundaries are unchanged. Re-audited 2026-09-05 after #9830 touched `gc/policy.rs`. That change is (a) six `thread_local! {` blocks rewritten as `crate::perry_thread_local! {` and (b) one `#[cfg(test)]` accessor listing the trigger path's hot-slot indices. The macro keeps the same storage, the same `.with()` at every read and write, and the same destructor registration (the teardown guard exists exactly when `needs_drop` holds, which is what `std::thread_local!` already decided); no value, predicate or branch in the file changes, so no mark or sweep control flow does. The one new behaviour is on a declaration's FIRST read: `HotKey::resolve_and_cache` takes a mutex and allocates a key through the GLOBAL allocator. Even if a first read landed inside this window it would be sound \u2014 the window's contract is that nothing relocates and no JS callback runs, and a mimalloc allocation does neither. `census_pass1_if_armed` is still inside `step_mark_propagation` and `census_take_if_armed_at_full_sweep_start` inside `step_sweep`; the bracketing is untouched. Re-audited 2026-09-06 (train132) after #9860 and #9845 touched `gc/mod.rs`. Both hunks are re-export lists and nothing else: #9860 adds `idle_reclaim_elapsed_starts` / `IDLE_RECLAIM_REARM_MS`, and #9845 adds `owner_is_dead_copied_minor_from_space_of_type`. No mark or sweep control flow changes. #9845's substantive work sits in `gc/oldgen.rs` and `gc/copying.rs`, neither pinned: the copying-minor arm (`finalize_dead_copied_minor_from_space_regexps`) runs on a MINOR, which skips both census boundaries; the full-cycle arm (`collect_dead_registered_regexps_post_trace`, from `with_dead_collection_finalize`) walks the RegExp registry building a Vec of addresses \u2014 no GC allocation, no JS callback, so it cannot relocate the snapshot's subjects \u2014 and it is reached from the sweep body, i.e. AFTER `census_take_if_armed_at_full_sweep_start` has already `take()`n the snapshot out of the thread-local. The mark-complete -> sweep-entry window is unchanged. Re-audited 2026-09-07 for the tenuring price counters: `gc/mod.rs` only loads diagnostic atomics in the process-exit reporting path, after collection has returned; it does not alter mark/sweep control flow or execute inside the mark-complete -> sweep-entry window.", "window": { "start": { "file": "crates/perry-runtime/src/gc/census.rs", @@ -293,7 +317,7 @@ "sources": { "crates/perry-runtime/src/gc/census.rs": "388414f9629f196e84673e91bebd04bdcdcabdaa180252d2dfe4b82d1b49ca5a", "crates/perry-runtime/src/gc/cycle.rs": "2e2f5adca2229f74409e01a1cb571e2147cd8a33f58d0976711fce98d4777309", - "crates/perry-runtime/src/gc/mod.rs": "43523b66595c61516ef6fcd4139d3ec5b4768a13c46ae1470c1d45481eacfdd9", + "crates/perry-runtime/src/gc/mod.rs": "2f8fa58d50aa95cd0156e68e01197ebb556d821a0e748ef238d1c5c407e8e77c", "crates/perry-runtime/src/gc/policy.rs": "dc9242ed40c0aa9c411d1ec0235c0219c6716dd82d56eb4d46578f7e889825d2", "crates/perry-runtime/src/gc/progress.rs": "a5ad3971bbe4047229ca57325234780daa85921dbc778e1c08dff4ad07ccfb96" }