Skip to content

coordinator: one host headroom accounting, and a park holds until a queued session has no room - #1782

Merged
nikhilunni merged 11 commits into
mainfrom
fix/one-host-resource-accounting
Nov 10, 2026
Merged

nikhilunni merged 11 commits into
mainfrom
fix/one-host-resource-accounting

Conversation

@nikhilunni

@nikhilunni nikhilunni commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Symptom. A host whose parked VMs had reserved its RAM held every park while a resume queued. The park reaper divided the host's allocatable_mib by its total RAM and ignored the budgets reserved on it; the host adds a resident VM's memory back into allocatable_mib, so the host read as roomy to the reaper and as full to placement. The queued resume never placed and no park descended.

A park holds until a queued session has no room. A parked VM keeps its RAM and its reservation, so a park costs nothing until a session needs that room. The gate (ParkView in idle_evictor.rs) reads demand, not a percentage:

  • A session parks, and a park holds, while every queued session fits on some host placement ranks.
  • When a queued session fits nowhere, one park per host yields per reaper pass: the oldest park, on the host with the most unreserved RAM (the nearest to a fit), and none on a host where an eviction already runs. One running eviction counts for one waiting session, so the reaper does not descend more parks than the queue needs.
  • A host reserved beyond the RAM it measures yields a park.
  • A session that goes idle while a queued session has no room does not park.
  • An unmeasured host does not park.

ENGRAM_PARK_HEADROOM_FLOOR_PCT is gone. A percentage of unreserved RAM refuses every park on a packed host while nothing waits for the RAM.

The reaper reads the fleet once per pass, not once per parked session. A failed read holds the parks: a descent is a full capture that cannot be undone.

One definition. engram_core::types::host::HostHeadroom { allocatable_mib, cpu_budget, reserved } with free_mib, free_vcpus and fits. Free RAM is allocatable less reserved; a parked VM keeps its reservation until it descends, so it is not free. Every decision that asks "is there room on this host" reads it: placement (create, resume, queue, capture, teleport), the host view, the fleet autoscaling signal, and the park gate.

Deleted copies.

  • engram-postgres: HostFit; three copies of the reservation SQL (pick_host_2d, per_host_reserved, placement_no_fit_details) → reserved_by_host_sql; two fit-map loops → headroom_map.
  • engram-sim: the fit arithmetic in pick_host_2d and placement_no_fit_details.
  • engram-coordinator: the arithmetic in pick_from, placement_preview, fleet_snapshot, HostView::from_row, and the reaper's raw-RAM percent.

Sim drift fixed. The simulator gated placement on CPU when a host reported no cores; Postgres does not. Both now read HostHeadroom::fits, and a conformance scenario pins it.

Behaviour changes.

  • The park gate, as above. The reaper's raw mem_total − mem_used fallback for a host without a RAM ledger is gone.
  • A deployment that set ENGRAM_PARK_HEADROOM_FLOOR_PCT (for example to turn parking off) loses that switch.
  • The placement fit rule is unchanged: unmeasured RAM or an unreported core count does not gate placement.

ADR 0074's divergence log records the change.

Test plan

  • cargo clippy -p engram-core -p engram-coordinator --all-targets -- -D warnings: clean
  • New tests: headroom_tests in engram-core; in the coordinator, a park yields to a queued session with no room, a full host holds its parks while nothing waits, a session does not park while a queued session has no room, an unmeasured host does not park, a failed read holds the park, and three unit tests of ParkView::yields; the conformance scenario a_host_without_reported_cores_has_no_cpu_gate
  • CI Gate

🤖 Generated with Claude Code

https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e

nikhilunni and others added 6 commits November 9, 2026 09:11
Add `HostHeadroom` to engram-core: a host's allocatable RAM, its CPU
budget and the budgets reserved on it, with `free_mib`, `free_vcpus`,
`fits` and `free_pct`. Every decision that asks "is there room on this
host" now reads it.

Replace the copies of that arithmetic:

- engram-postgres: delete `HostFit`. `reserved_by_host_sql` is the one
  reservation query (it was copied into `pick_host_2d`,
  `per_host_reserved` and `placement_no_fit_details`) and `headroom_map`
  the one fit-map builder (two loops before).
- engram-sim: `pick_host_2d` and `placement_no_fit_details` read
  `HostHeadroom`. This fixes a drift: the simulator gated placement on
  CPU when a host reported no cores; Postgres does not.
- engram-coordinator: `pick_from`, `placement_preview`, `fleet_snapshot`
  and `HostView` read `HostHeadroom`. `fleet_snapshot` feeds the
  operator's autoscaling signal, so the operator sees the same free
  figures placement reserves against.

The fit rule is unchanged: a host with unmeasured RAM or no reported
cores does not gate placement.

Review pass: tightened the `HostHeadroom` doc to the fact the arithmetic
relies on (a VM's RAM counts once, in `reserved`); the other comments
state what each item reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
`host_has_memory_headroom` divided the host's `allocatable_mib` by its
total RAM and ignored the budgets reserved on the host. The host adds a
resident VM's memory back into `allocatable_mib`, so a host whose parks
had reserved its RAM read as roomy, held every park, and a queued resume
never placed on it while placement saw the host as full.

The gate now reads `HostHeadroom::free_pct`: the unreserved RAM, the
same accounting placement reserves against, against
`ENGRAM_PARK_HEADROOM_FLOOR_PCT`. A parked VM keeps its reservation, so
a host full of parks has no headroom and the reaper descends a park,
which frees the RAM the resume waits for.

Behaviour change: the raw `mem_total - mem_used` fallback for a host
without a RAM ledger is gone. An unmeasured host has no headroom and
does not park; it takes the full eviction.

`MiniMeta` gains a `reserved` map for `per_host_reserved`, and the new
test `parked_reservations_without_headroom_descend_the_park` covers the
production case. ADR 0074's divergence log records the change.

Review pass: rewrote two test comments that described the deleted
fallback, and gave the new test its own doc comment above the ADR 0101
test's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
The park gate compared a host's unreserved RAM with a percentage of its
total. A parked VM keeps its reservation, so on a packed host the gate
refused every park and descended the existing ones although nothing
waited for the RAM, and the fast un-pause was lost for no gain.

The gate now reads demand. A session parks, and a park holds, while
every queued session fits on some host placement ranks. When one does
not fit, one park per host yields per reaper pass: the oldest, on the
host nearest to a fit, and none where an eviction already runs. A host
reserved beyond the RAM it measures yields a park too. A session that
goes idle while a queued session has no room does not park.

`ENGRAM_PARK_HEADROOM_FLOOR_PCT` and `HostHeadroom::free_pct` are gone.
The reaper reads the fleet once per pass, not once per parked session,
and a failed read holds the parks: a descent is a full capture that
cannot be undone.

Also from the review of this branch:
- the placement preview keeps the unmeasured-host rule of the placement
  transaction (no CPU check), which the refactor had changed;
- a conformance scenario pins that both stores place on a host that
  reports no cores;
- the RAM ledger's module doc states that no backend sets the `parked`
  flag and why it must not be set alone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
A post-copy restore whose uffd handler exits before the load failed with
"uffd handler exited before the load (exit status: 1)". The reason is
only in the handler's log in the jail directory, which a failed restore
removes. The two-host teleport tests fail this way now and then, and the
CI log cannot say why.

The error now ends with the tail of `uffd-handler.log`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
The helper sent `allow_park: true` and reached the capture only because the old headroom gate failed closed on the simulated host, which reports no total RAM. With the demand gate that host parks. The callers exercise the capture and the finalize, so the helper sends `allow_park: false`, the payload of a park's descent.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni nikhilunni changed the title coordinator: one host headroom accounting, and the park reaper reads it coordinator: one host headroom accounting, and a park holds until a queued session has no room Nov 10, 2026
nikhilunni and others added 5 commits November 10, 2026 07:42
The kernel's partition scan reads the start of a new ublk device. On newer kernels it runs in a work item, so the read can arrive after the test's first guest read, and the source records it as read cold. The test then saw chunks [3, 0] where it expected [3]. The test now leaves the first chunk out of the comparison; chunks 1 and up still prove that the copy's stream is not recorded.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni
nikhilunni merged commit 1f9266d into main Nov 10, 2026
28 checks passed
@nikhilunni
nikhilunni deleted the fix/one-host-resource-accounting branch November 10, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant