perf(gc): whole-block in-place promotion — kill the ~243 ns/object promotion tax on a fully-live nursery

[proggeramlug]

retain (3M records, none ever dropped) and retain_wide are the only two
benchmarks still far off Node — 0.80 s vs 0.12 and 1.33 s vs 0.15 on the pinned
M1 mini. Everything else is within 1.2×–3.6×.

The cause is that every survivor gets promoted, object by object. The GC
trace shows 2,097,155 promotions across four copying minors at 509.7 ms of
pause — 243 ns per object — spent on arena_alloc_gc_old, the memcpy,
layout_transfer, old_page_account_promoted_object, the GcMoveHookKind
hooks, and the forwarding/rewrite that follows. The measured young-survival
ratio on those cycles is 1.000: not one of those objects had a reason to
move.

The structural answer is V8-style whole-page promotion: when a block is
(near-)entirely live, relabel the block as old-gen instead of evacuating it.
Nothing moves, so there is no copy, no old-gen allocation, no layout transfer,
no move hook, no forwarding pointer, and no slot anywhere in the heap or in any
address-keyed side table needs rewriting.

Things this has to get right:

• The promotion decision. Block liveness is not knowable before the trace
  and Eden blocks are recycled at offset 0, so per-block history means nothing.
  A per-cycle decision from the previous cycle's measured survival ratio works
  only if a promoting cycle still measures.
• Page-generation metadata and the remembered set when a page changes
  generation in place — this is where correctness lives. In particular
  Old ⟹ TENURED (perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) and the old-gen page-object index, which is what the
  remembered-set dirty scan uses to reach an object at all.
• Pacing. A copying minor recycles Eden's blocks; promotion hands them away.
  The arena-bytes trigger fires on arena_total_bytes(), so the allocation
  runway collapses unless the capacity is given back.
• Footprint. Dead objects on a promoted page are retained until the next
  full. The per-cycle cost is (1 − liveness) × young bytes; the steady state
  needs its own bound.

[proggeramlug]

Measurement of the residue named in #7744's "Not closed", on main @ 0a2bf15 (perry-dev, M1 Max), retain.ts under PERRY_GC_TRACE=1.

The one full mark-sweep is 36% of all GC pause, and its premise does not hold on this workload.

6 cycles, 504.4 ms total pause. Five minors; one full at 183.7 ms.

What that full did:

arena in-use before → after	67.0 MB → 59.0 MB
reclaimed	8.37 MB = 11.9%
retained_forwarded_stub_bytes	0
retained_forwarded_stub_objects	0

Where its time went: trace_worklist 93.1 ms (55%), atomic_finalize 32.9 ms (19%), remembered_set_marking 28.9 ms (17%), sweep 9.5 ms (6%).

So more than half of it is re-tracing the 3M-object live worklist to free 8 MB, and it freed zero forwarding stubs — which is the entire documented justification for the arena-growth escalation ("a non-moving minor sweep cannot free array-growth forwarding stubs … only a FULL mark-sweep reclaims them"). Being under the 20% productive threshold, it is also what pushes GC_MAJOR_PACING_BACKOFF_SHIFT 0 → 1, i.e. the yield-adaptive backoff (#7726) only learns the bet was bad after paying 183.7 ms for it, once per run.

The signal is available before escalating. set_forwarding_address has exactly two production callers: array/push_pop.rs:162, which leaves the retained array-growth stub, and gc/copying.rs:744, whose forwarding is transient and cleared by the same cycle. So retained stub debt is countable with one add on the growth path and a reset when a full sweeps them, rather than the object walk evacuation_policy uses today. arena_growth_full_escalation_due could then require that there is stub debt to reclaim, instead of discovering there was none by doing the work — and old-gen garbage is unaffected either way, since old_reclaim_pressure_due is a separate trigger.

Note this got cheaper to justify after #7744: whole-block in-place promotion does not copy, so it does not leave forwarding stubs at all on exactly the high-survival workloads where the escalation is most expensive.

I have not implemented this — it is a cross-cutting collector change that needs the bench corpus and a gc_ratchet pass to land safely, and I did not want to start it inside an unrelated PR. Posting the measurement so whoever picks it up starts from numbers.

(The other half of the residue #7744 names — the remembered-set scan over the array's dirty element slots — is untouched by the above.)

[proggeramlug]

Closing: the promotion tax this issue was opened for is gone, measured three ways

243 ns/object → 8.9 ns/object. Not an estimate — PERRY_GC_TRACE=1 per-cycle
records on retain.ts: cycles 1–4 promote 2,109,820 objects in 18.7 ms of
in_place_promotion (gc-handoff/RETAIN4-NOTES.md). The intermediate step was
105 ns/object when #7744 landed whole-block promotion; #7888 removed the trace on
fully-live promoting cycles, which is the rest of it.

End to end, on the pinned quiet M1 mini (gc-handoff/MEASUREMENT-PROTOCOL.md,
baseline round 2026-08-12 @ 14e291f3a), the two benchmarks this issue names:

bench	at filing	now	node	now vs node
retain	0.80 s	0.153 s	0.137	1.12×
retain_wide	1.33 s	0.197 s	0.157	1.25×

The opening premise — "the only two benchmarks still far off Node" at 6.7× and
8.9× — no longer holds; both are inside the 1.3× band that 15 of the 19 corpus
programs sit in.

Each of the four "things this has to get right" has a named mechanism, and each
was measured rather than argued:

• The promotion decision — gc/promote_in_place.rs, per cycle from the
  previous cycle's measured young-survival ratio. Sound rather than
  optimistic because a promoting cycle still traces, so it re-measures; the
  observed population is bimodal with a three-order-of-magnitude gap (0.999–1.000
  vs 0.000–0.004), so the 95% threshold is justified by footprint, not by
  classification.
• Page-generation metadata and the remembered set — arena/promote.rs retags
  the young blocks to Old/PromotedYoung before the trace, so the barrier
  predicates read old-gen semantics while the trace runs and record the right
  edges; Old ⟹ TENURED (perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) is restored by the finish walk's header stamp,
  and the old-gen page index is registered in per-page bulk runs (now as a
  described run rather than a stored list, perf(gc): describe a promoted block's page-object list instead of storing it #7914).
• Pacing — the exact hazard the issue predicted did fire: the first working
  version collapsed Eden's high-water 57 → 18 MB and turned a 0.81 s program into
  1.52 s. Fixed by returning the freed capacity as one-shot headroom to the
  next trigger rather than re-reserving blocks.
• Footprint — PROMOTED_DEAD_BUDGET_BYTES (32 MB of promoted dead bytes since
  the last full) plus the untraced arm's max(128 MB, old-gen at last measurement) budget, both reset by a full collection. Measured
  in_place_dead_bytes and in_place_sparse_blocks are zero across the
  benchmark set.

The residue, named rather than left attached to this issue

None of it is promotion cost, and three of the four already have their own
trackers:

• memory(gc): untraced promotion can retain an old-heap-sized phase-change cohort as live #7902 — untraced promotion can retain a phase-change cohort as live.
• perf(gc): the nursery/promotion budget is denominated in BYTES but the collector's cost is per-OBJECT — compaction is taxed back #7929 — the nursery/promotion budget is denominated in bytes while the
  collector's cost is per-object.
• perf(gc): time-budgeted steps still contain unbounded atomic work #7903 — time-budgeted steps still contain unbounded atomic work.
• Unfiled until now: cycle 0 cannot promote in place, because nothing has
  been measured yet — so the first copying minor evacuates object-by-object and
  is 58% of retain's GC pause, 81% of retain1's, 71% of retain_wide1's,
  and cycle 1 then promotes those very blocks in place. The obvious fix (a
  smaller first nursery) is measured unsafe as a global change: at a 2 MB cap
  asyncpipe goes from 0 collections to 4, costing 385 ms on a 127 ms program.
  Filed as perf(gc): the first copying minor cannot promote in place — 58-81% of remaining GC pause on the live-set cluster #7937.

Also worth recording for whoever picks up the next round: on this cluster the
mutator is now the binding constraint, not the collector — retain_wide and
shapes lose to scriptc with a collector that costs zero, and retain needs it
to cost exactly that. A further round aimed here that works on the collector is
working on the smaller half.