Repository navigation
coordinator: one host headroom accounting, and a park holds until a queued session has no room - #1782
Merged
Merged
Conversation
Add `HostHeadroom` to engram-core: a host's allocatable RAM, its CPU budget and the budgets reserved on it, with `free_mib`, `free_vcpus`, `fits` and `free_pct`. Every decision that asks "is there room on this host" now reads it. Replace the copies of that arithmetic: - engram-postgres: delete `HostFit`. `reserved_by_host_sql` is the one reservation query (it was copied into `pick_host_2d`, `per_host_reserved` and `placement_no_fit_details`) and `headroom_map` the one fit-map builder (two loops before). - engram-sim: `pick_host_2d` and `placement_no_fit_details` read `HostHeadroom`. This fixes a drift: the simulator gated placement on CPU when a host reported no cores; Postgres does not. - engram-coordinator: `pick_from`, `placement_preview`, `fleet_snapshot` and `HostView` read `HostHeadroom`. `fleet_snapshot` feeds the operator's autoscaling signal, so the operator sees the same free figures placement reserves against. The fit rule is unchanged: a host with unmeasured RAM or no reported cores does not gate placement. Review pass: tightened the `HostHeadroom` doc to the fact the arithmetic relies on (a VM's RAM counts once, in `reserved`); the other comments state what each item reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
`host_has_memory_headroom` divided the host's `allocatable_mib` by its total RAM and ignored the budgets reserved on the host. The host adds a resident VM's memory back into `allocatable_mib`, so a host whose parks had reserved its RAM read as roomy, held every park, and a queued resume never placed on it while placement saw the host as full. The gate now reads `HostHeadroom::free_pct`: the unreserved RAM, the same accounting placement reserves against, against `ENGRAM_PARK_HEADROOM_FLOOR_PCT`. A parked VM keeps its reservation, so a host full of parks has no headroom and the reaper descends a park, which frees the RAM the resume waits for. Behaviour change: the raw `mem_total - mem_used` fallback for a host without a RAM ledger is gone. An unmeasured host has no headroom and does not park; it takes the full eviction. `MiniMeta` gains a `reserved` map for `per_host_reserved`, and the new test `parked_reservations_without_headroom_descend_the_park` covers the production case. ADR 0074's divergence log records the change. Review pass: rewrote two test comments that described the deleted fallback, and gave the new test its own doc comment above the ADR 0101 test's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
The park gate compared a host's unreserved RAM with a percentage of its total. A parked VM keeps its reservation, so on a packed host the gate refused every park and descended the existing ones although nothing waited for the RAM, and the fast un-pause was lost for no gain. The gate now reads demand. A session parks, and a park holds, while every queued session fits on some host placement ranks. When one does not fit, one park per host yields per reaper pass: the oldest, on the host nearest to a fit, and none where an eviction already runs. A host reserved beyond the RAM it measures yields a park too. A session that goes idle while a queued session has no room does not park. `ENGRAM_PARK_HEADROOM_FLOOR_PCT` and `HostHeadroom::free_pct` are gone. The reaper reads the fleet once per pass, not once per parked session, and a failed read holds the parks: a descent is a full capture that cannot be undone. Also from the review of this branch: - the placement preview keeps the unmeasured-host rule of the placement transaction (no CPU check), which the refactor had changed; - a conformance scenario pins that both stores place on a host that reports no cores; - the RAM ledger's module doc states that no backend sets the `parked` flag and why it must not be set alone. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
A post-copy restore whose uffd handler exits before the load failed with "uffd handler exited before the load (exit status: 1)". The reason is only in the handler's log in the jail directory, which a failed restore removes. The two-host teleport tests fail this way now and then, and the CI log cannot say why. The error now ends with the tail of `uffd-handler.log`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
The helper sent `allow_park: true` and reached the capture only because the old headroom gate failed closed on the simulated host, which reports no total RAM. With the demand gate that host parks. The callers exercise the capture and the finalize, so the helper sends `allow_park: false`, the payload of a park's descent. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
The kernel's partition scan reads the start of a new ublk device. On newer kernels it runs in a work item, so the read can arrive after the test's first guest read, and the source records it as read cold. The test then saw chunks [3, 0] where it expected [3]. The test now leaves the first chunk out of the comparison; chunks 1 and up still prove that the copy's stream is not recorded. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Symptom. A host whose parked VMs had reserved its RAM held every park while a resume queued. The park reaper divided the host's
allocatable_mibby its total RAM and ignored the budgets reserved on it; the host adds a resident VM's memory back intoallocatable_mib, so the host read as roomy to the reaper and as full to placement. The queued resume never placed and no park descended.A park holds until a queued session has no room. A parked VM keeps its RAM and its reservation, so a park costs nothing until a session needs that room. The gate (
ParkViewinidle_evictor.rs) reads demand, not a percentage:ENGRAM_PARK_HEADROOM_FLOOR_PCTis gone. A percentage of unreserved RAM refuses every park on a packed host while nothing waits for the RAM.The reaper reads the fleet once per pass, not once per parked session. A failed read holds the parks: a descent is a full capture that cannot be undone.
One definition.
engram_core::types::host::HostHeadroom { allocatable_mib, cpu_budget, reserved }withfree_mib,free_vcpusandfits. Free RAM is allocatable less reserved; a parked VM keeps its reservation until it descends, so it is not free. Every decision that asks "is there room on this host" reads it: placement (create, resume, queue, capture, teleport), the host view, the fleet autoscaling signal, and the park gate.Deleted copies.
HostFit; three copies of the reservation SQL (pick_host_2d,per_host_reserved,placement_no_fit_details) →reserved_by_host_sql; two fit-map loops →headroom_map.pick_host_2dandplacement_no_fit_details.pick_from,placement_preview,fleet_snapshot,HostView::from_row, and the reaper's raw-RAM percent.Sim drift fixed. The simulator gated placement on CPU when a host reported no cores; Postgres does not. Both now read
HostHeadroom::fits, and a conformance scenario pins it.Behaviour changes.
mem_total − mem_usedfallback for a host without a RAM ledger is gone.ENGRAM_PARK_HEADROOM_FLOOR_PCT(for example to turn parking off) loses that switch.ADR 0074's divergence log records the change.
Test plan
cargo clippy -p engram-core -p engram-coordinator --all-targets -- -D warnings: cleanheadroom_testsin engram-core; in the coordinator, a park yields to a queued session with no room, a full host holds its parks while nothing waits, a session does not park while a queued session has no room, an unmeasured host does not park, a failed read holds the park, and three unit tests ofParkView::yields; the conformance scenarioa_host_without_reported_cores_has_no_cpu_gate🤖 Generated with Claude Code
https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e