Skip to content

feat(disk): a resume boots before its disk is copied in (ADR 0125 amended) - #1783

Merged
nikhilunni merged 35 commits into
mainfrom
disk/lazy-through-fill
Nov 10, 2026
Merged

feat(disk): a resume boots before its disk is copied in (ADR 0125 amended)#1783
nikhilunni merged 35 commits into
mainfrom
disk/lazy-through-fill

Conversation

@nikhilunni

@nikhilunni nikhilunni commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

A cold resume wrote the whole difference between the session's disk and the nearest volume on the host into a thin volume before the VM started (ADR 0125, "nothing is faulted lazily"). An image whose tag moves often shares little with a session that is a few hours old, so the difference is most of the disk. On the dev cluster, a session with a 37 GiB root disk woken on fresh hosts that staged a newer image took 34.7 s from prompt to run_started, 29.6 s of it the disk write (2030 chunks of 16 MiB).

Change

The resume boots the VM before its disk is copied in. ADR 0125 is amended in place (Decision 3, the resume lifecycle, a dated divergence entry); DESIGN.md "Session disk" states the facts and the tests that guard them.

  • engram-disk-thin: the live volume of a resume is a dm-clone device. Its destination is a thin snapshot of the nearest base, its source a read-only block device, its metadata a small thin in the pool. A discard marks every region the base already holds, or the manifest leaves out, as copied without a copy; the kernel copies the rest in the background. When every region is in, the settle reloads the device as a thin of its own under the same path. Suspend, operation and resume run as one task that a dropped caller cannot cancel, and opening the pool resumes a device left suspended.
  • engram-host-agent blk_source.rs: the source is a ublk device the host-agent serves in process from the fill engine (libublk, user recovery with reissue). The kernel's ascending copy gets whole chunks at Hydrate from a window fetched ahead; any other read is the guest's and gets a ranged read at Demand. A store error that can pass stalls a read; it does not fail it. A read that meets a host-agent shutdown stays queued for the next process. A host without /dev/ublk-control logs an error at startup and writes disks in full as before.
  • engram-host-core: one lazy stage path. The base is the kept checkpoint volume or staged image version that shares the most chunks with the manifest at the same offset. A settle driver per sandbox serves the source again after a host-agent restart (from the pool's names; no new heartbeat field, no sandbox.json field), polls the copy, and cuts over. A volume whose source nothing serves is never deleted.
  • Checkpoints in the window read nothing through the source and lose no write: a block whose mapping changed since the base is the snapshot's, any other is the manifest's. Only a snapshot taken after the settle becomes a checkpoint volume. The periodic checkpoint waits for the settle, for at most 10 minutes, because a checkpoint in the window reads every copied block back and nearly stops the copy. An eviction or a snapshot does not wait.
  • Settings: the host-fleet chart sets the source's window and the kernel's in-flight copy limit (thinPool.hydration.windowChunks 12, thinPool.hydration.inFlightRegions 512).
  • No hash on a volume read: FillEngine::read no longer hashes a fetched chunk (9 to 13 percent of host-agent CPU in the copy). This also applies to the eager stage and to the substrate server's whole-chunk disk reads. A chunk is checked when it is written to the store or to the cache.
  • Disk first-touch: the source records the chunks the guest read before the copy reached them; the snapshot publishes them, and the next resume fetches those at Prefix.
  • Teleport: a source with no checkpoint volume (hydrating, or settled before its first checkpoint) refuses a live move and the coordinator moves it by snapshot. The live move's three block tiers stay.
  • Rollout: REATTACH_GENERATION goes to 3, so the pre-upgrade drain evicts every session before the roll. Nodes need ublk_drv; node-prep loads it. A rollback past this change needs the hosts drained by hand first, as for generations 1 and 2 (CHANGELOG upgrade note).
  • CI: a new lane, tests (ublk source, root), runs the blk_source and lazy_restore root tests on a GitHub-hosted runner, because the Firecracker runners' kernel is built without ublk. It is in the CI Gate. The dm-clone test stays in the Firecracker root lane.
  • Dev shell: the flake pins nixos-26.05; nixos-unstable no longer builds sqlx-cli.

Known limits, not changed here:

  • A source holds up to 14 chunks (224 MiB) in memory for each hydrating disk. There is no budget per node.
  • dm-clone starts no background copy while guest I/O is in flight on the device.
  • Serving a chunk the host holds at another offset is not built. The moved image shared 354 chunks with the session in place and 651 anywhere, so the gain is small.

Test

Dev cluster (two c3 hosts, a 37 GiB root / 13 GiB memory session three days older than the staged image, fresh hosts for each wake):

Build Prompt to run_started Disk before boot Disk fully local First token
main (eager) 34.7 s 29.6 s at boot 45.1 s
this branch, window 4 / in-flight 128 4.1 s 0.7 s 123 s after 10.8 s
this branch head (window 12 / in-flight 512) 2.75 s 0.7 s 53 s after 9.0 s

On the head build (b864319ce):

  • The copy averages 612 MiB/s and runs at 850 MiB/s once the memory fill is done (peak 1118); the pool's disk is the limit. Host-agent memory peaked at 810 MiB. The longest gap in a random-read probe in the guest was 0.13 s. Guest file checksums matched with the page cache dropped.

  • SHA-256 is 3.4 percent of host-agent CPU in the copy, down from 9.1 percent. Heap-profile sampling (not changed here) is 58 percent.

  • The first periodic checkpoint after the settle made the checkpoint volume and left no source, base or metadata volume. The settle ended before the first periodic checkpoint was due, so the checkpoint hold did not act in this run; its unit test covers the rule.

  • SIGTERM to the host-agent at 91 percent copied: no I/O error on the host or in the guest, the next process served the disk again, the settle completed, checksums matched, and the guest ran at normal speed after. Guest reads of regions not yet copied waited 13.6 s, the length of the restart (about 10 s of it the kubelet's restart delay in this test).

  • Root tests on a cluster node (kernel 6.12): clone 2/2, blk_source 3/3, lazy_restore 3/3.

  • CI: the ublk lane passes 6/6 on the GitHub runner's kernel (7.0). If a ublk test hangs, the lane prints each test thread's kernel stack and the kernel log after 10 minutes.

Follow-up, not in this PR: the host kernel scans a source device for partitions. UBLK_F_NO_AUTO_PART_SCAN turns that off, but libublk 0.4.8 refuses the flag.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D

nikhilunni and others added 21 commits November 9, 2026 10:29
@nikhilunni @claude
disk-thin: dm-clone tables, skip discards, hydration control, cutover
c2bb9d2 
`SessionDisk` gains three methods with a "not on this platform" default:
`hydrate` makes a device over a destination volume that a guest can open
before the volume holds its data, `hydration` reports how far the kernel
has copied it in, and `settle` maps the device straight onto the
destination once every region is in.

`ThinPoolDisk` implements them with a `clone` target: destination = the
thin volume, source = any read-only block device, metadata = an 8 MiB thin
named `<device>-md`, one region per chunk. The device is created with
hydration off; a `BLKDISCARD` of every chunk the destination already holds
(`no_discard_passdown`) marks those regions copied without a write, so the
blocks the volume shares with its origin stay shared and only the delta is
ever copied. Then hydration is turned on with a 1 MiB batch and 8 MiB in
flight. `settle` suspends, loads a `linear` table over the destination and
resumes in one owned task, so the guest's held I/O is released whatever
happens to the caller; the metadata thin goes after. `delete` dispatches on
the table kind so a hydrating or settled device takes its metadata with it.

The review pass folded `delete`'s device removal into `remove_device`,
shared by the thin and clone paths, and `thin_of` onto `target_of`.

Root tests in `tests/clone.rs` run over the CI pool with a loop device as
the source: the delta is copied, a guest write through the device lands,
the settled destination reads as source plus write, `dirty_blocks` against
the origin lists only the copied and written chunks (the skipped chunk is
still shared), a settle before the copy is done is refused, and a delete
mid-copy leaves nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
disk-thin: suspend, operation and resume as one task; resume on open
1bf0f9d 
`snapshot` and `settle` each spawned an owned suspend-op-resume task.
`while_suspended` is that task once, and hands both outcomes back so a
caller decides what a failed resume costs it (`snapshot` deletes the
unactivated thin). `Dm::suspended` reads the kernel's `DM_SUSPEND_FLAG`,
and `open` resumes every device of its pool that a host-agent dying
inside that task left suspended: the device holds every I/O until then,
and its inactive table, if any, is the one the task meant to make live.

Root test: a device suspended behind the disk's back reads again after
the next `open`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-agent: BlkSource, a ublk device served by the fill engine
c7bcefa 
A hydrating session volume copies its cold regions from a read-only
block device. BlkSource is that device: a ublk device the host-agent
serves in process, one per role, with the bytes coming from the fill
engine directly. There is no helper process, no second fetch path and
no second cache copy; the engine's classes, hedging, peer racing and
admission are the whole policy.

The source cannot be told which reads are the kernel's background copy
and which are the guest waiting, so it tells them apart by position.
The copy walks the delta chunks in ascending order; the source keeps a
window of whole chunks around that stream, fetched at `Hydrate` and
four chunks ahead, and hands them out region by region, so the store
sees one GET per chunk. A read anywhere else is a guest read: a ranged
GET at `Demand`, and its chunk is recorded as touched (the first-touch
prefix of a later restore).

A host-agent that dies leaves the device to the kernel: with
`UBLK_F_USER_RECOVERY` the reads it left queued wait, and `recover`
takes the device over from the next host-agent and answers them. A stop
is refused while a device still maps onto the source: a ublk device
with reads queued and no server holds the kernel until reboot.

The queue thread sleeps in `io_uring_enter`; finished reads cross back
from tokio through an eventfd read queued on the same ring. libublk
sends control commands over a thread-local ring that only the thread
which built the `UblkCtrl` has, so the server thread and every blocking
task that issues one make their own first.

The root lane loads `ublk_drv` and runs the two tests: a served
manifest answers a demand read and a region-by-region stream with only
the demand read recorded, and a server killed with a read queued is
recovered by a new one that completes the read and deletes the device.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core: one lazy stage path
ea0b6e8 
A role whose manifest is not on the host used to take one of three
eager paths (`OntoKept`, `FromImage`, `Whole`): the delta was written
in full before the guest started, 27-29 s of a 39 s cold resume. They
are now one path, `bring_up`: the nearest base this host holds (a kept
checkpoint of the lineage, the staged image, or nothing) is snapshotted
and the difference comes in behind the running guest, through
`SessionDisk::hydrate` and a source the fill engine serves
(`SourceServer`, the trait `BlkSource` implements). On a platform
without a hydrating device the same path writes the difference first
(`Staged`).

The names stay the only record. A hydrating role has no checkpoint
volume until it settles; two names carry what a checkpoint volume
carried:

  sb-<sandbox>-<role>-dest-<manifest>   the live device's destination,
                                        which becomes that manifest
  sb-<sandbox>-<role>-base-<manifest>   what it was snapshotted from

From them and the kernel's own state (the clone table names the source
device) a restarted host-agent rebuilds everything: `reattach` finds
the hydrating roles, recomputes the plan from the two manifests, and
serves the source again. No caller tracks a manifest ref or a device
id between calls.

`hydration_plan` is the pure core: chunks the base holds in final form
are skipped (the kernel marks them hydrated through a discard, so the
destination keeps sharing their blocks), chunks the manifest leaves out
are zeroed before the copy, and the rest is the delta the source
serves. `SessionVolumes::settle` is one step of the way to a settled
volume, for a driver loop; it cuts the device over and stops the source.
A release or purge deletes the live device before the volumes under it
and stops its source after, which is the order the ublk rule needs. A
hydrating volume whose source this process does not serve is never
deleted: the kernel removes a device only after its reads complete, and
nothing would answer them. Its destination and base stay with it, so a
later `reattach` can rebuild the plan.

`SessionDisk::hydration` reports `Settled` for a volume that never
hydrated, so one question serves every volume.

The thin-pool disk lists and opens hydrating and settled devices, and
reports the source device of a hydrating one.

Tests drive the lazy path through a fake hydrating disk (hard link for
the device, one chunk copied per poll) and a file-backed source server
that refuses to stop while a device still copies from it, so the order
rules are asserted, not assumed. The eager fallback keeps every
existing test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-agent: the restore boots before the disk hydrates
e16160a 
The pooled backend takes the lazy path the previous commit opened. The
host's `SourceServer` is `UblkSources`: a `BlkSource` per hydrating
role, served in process. A restore that comes up on a hydrating volume
gets a settle driver, a thin loop over `SessionVolumes::settle` that
polls the kernel's progress, cuts the device over when every region is
in, stops the source and logs `disk settled role=… regions=…
elapsed_ms=…`. Destroy drops the driver before the VM goes, so no
settle runs against a volume being released; the release itself ends
the device before its source, the order the ublk rule needs.

At startup, before the volume reconcile, `reattach_all` serves again
every source the previous process left to the kernel: a hydrating
volume whose source nothing serves cannot be deleted (the kernel waits
for reads nothing answers), so the reconcile would otherwise refuse the
dead sandboxes' volumes. Live sandboxes' roles get a settle driver; dead
sandboxes' volumes are deleted as before; a ublk device nothing maps
onto and nothing serves is reaped.

`REATTACH_GENERATION` goes to 3: a build before this one cannot serve
a source again after a restart, so the upgrade drain evicts every
session in both directions. The CHANGELOG carries the upgrade note;
node-prep loads `ublk_drv` on every start.

The root test `lazy_restore` runs the whole path over the pool: a cold
read through the source returns the manifest's bytes while the kernel
copies, a write to a hole survives the settle, the sources stop, and a
process killed mid-copy leaves a volume the next process must serve
again before it can delete it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core, disk-thin: checkpoints of a hydrating volume
470e906 
A checkpoint never reads through the source and never loses a write
made during the window. The snapshot is taken as before, under the
guest pause; the thin-pool disk now snapshots a hydrating or settled
device by the thin under it (the clone or linear table's destination),
with the device itself suspended so the guest's writes and the kernel's
copy both rest.

The upload follows one rule, `upload_hydrating`: the volume logically
holds the destination's manifest, and a block whose mapping still
equals the base's is one the copy has not reached, holding the base's
bytes where the manifest's belong. So a block whose mapping changed
(hydrated or written) is read from the snapshot and any other is the
manifest's: a chunk with no changed block keeps its entry unread, one
changed throughout is read whole, and a mixed one is the manifest's
chunk with the changed blocks laid over it. Mapping, not content, is
the test, so a guest write of bytes equal to the base's is a change.
`upload` and `upload_hydrating` share the fan-out that puts chunks.

The names stay the record. While the device hydrates, the destination
takes the published manifest's name and the snapshot goes, so a
restarted host-agent rebuilds the plan from the manifest the disk now
holds and no checkpoint volume exists before the settle. Once settled,
the snapshot becomes the checkpoint volume and the destination and base
names go: the normal layout. `checkpoint_of` answers with the
destination's manifest while there is no checkpoint volume, so `take`,
`pending` and the publish redrive need no new case.

Tests: the composition rule on a mixed chunk over a half-chunk block
(the file disk's content diff stands in for the mapping diff), a
checkpoint during the window and one after the settle through the fake
hydrating disk, and in the root lane a snapshot of the hydrating device
that diffs against the base.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core, host-agent: disk first-touch prefix
6f77201 
The kernel copies a hydrating volume in ascending order, so the chunks
the guest reads first in its window are cold until the copy reaches
them, and each read waits on a demand fetch. The source already records
the chunk indices of those reads in order. A capture now publishes them
as the session's disk first-touch record (`disk-first-touch.v1`, the
memory record's encoding, the same cap), beside the memory record.

The next restore of the session reads the record with its trace
lineage and hands the indices to the root volume's source. The ones the
copy must bring in (`prefix_hashes`: the record restricted to the
delta, as the wanted manifest's hashes; chunks the base holds and
chunks the manifest leaves out need no fetch) are filled at `Prefix`
without waiting, so they are resident when the guest asks for them and
when the copy reaches them. `Source::Continue` carries the list; a
create and a swap pass none.

`SessionVolumes::first_touch` answers from the serving source, and
after the source stops from what it reported, until the live volume
goes: a capture after the settle still publishes what the window read.
`Source` is no longer `Copy`.

Tests: the prefix restricted to the delta in order; the record
through the window, after the settle and after the release on the fake
disk; the disk record round trip beside the memory record's publish.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core, host-agent: a hydrating source moves by snapshot
302cf07 
A live move carries the disk in three tiers: the destination continues
the advertised checkpoint and pulls the blocks changed since it, before
and under the pause. The destination's volume now hydrates while those
blocks land, which is correct as it stands: a block the pull writes
changes the mapping the way a guest write does, and the destination's
first checkpoint lays it over the advertised disk by the same rule.

The source side has no such reference while it hydrates, or after it
settles before its first checkpoint: there is no checkpoint volume, so
"changed since the checkpoint" would be the whole copy plus the guest's
writes, not a live move's small difference. `take_precopy` refuses with
`Unsupported`, the presetup maps that to the coordinator's
snapshot-move fallback, and the session moves by checkpoint and resume
until a checkpoint makes the normal layout. Before this the diff failed
on a missing volume and the coordinator rolled the move back.

Tests: the refusal while hydrating and after the settle, and the live
move possible again after a checkpoint.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
docs: ADR 0125 amended for the hydrating resume; DESIGN.md session disk
11d72e6 
ADR 0125 said the dirty chunks are written before the VM starts and
nothing is faulted lazily on the disk path. Decision 3 and the Resume
lifecycle now describe the dm-clone device, its ublk source served
from the fill engine, the discard that marks held regions copied, the
settle, the checkpoint rule in the window and the restart that serves
a volume again from the pool's names. The staging tradeoff no longer
claims there is no lazy fallback; a new tradeoff names the userspace
process on the cold-region path and the ublk stop order. A dated
divergence entry carries the dev-cluster numbers (the tuned hydration
rate against the kernel's defaults, the copy's start lag, udev's open
after a create) and the decisions the plan left open: the reattach
generation, the source that moves by snapshot, the block tiers that
stay. The facts list and the DESIGN.md fact page name the tests that
guard each new fact. Node-prep's `modprobe ublk_drv` landed with the
restore.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core: reattach serves a role once; one failed sandbox does not s…
9164184 
…top the pass

A second `reattach` of a role this process already serves asked the
kernel to recover a device with a live server, which failed after
libublk's 30 s recovery retry. The role is skipped while its source is
in `serving`. `reattach_all` logs a sandbox whose reattach fails and
goes on to the rest: its volumes stay, unserved and undeletable, and
the other sandboxes come up. Found on the dev cluster by the root test
that kills a process mid-window.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
disk-thin, host-agent: root tests read off the runtime and name the m…
23cc864 
…etadata thin

The `blk_source` serve test read the device on the runtime that
dispatches the source's requests; on one thread the read waits for
itself, and the device keeps the read queued under user recovery, so
the test process could not be killed. The reads run on a blocking
thread now. The clone test expected the metadata thin as
`t-clone-io-md-<pid>`; it is `<device>-md`. Both found on the dev
cluster.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
disk-thin, host-core: a settled volume is a thin of its own
dc35978 
The settle reloaded the hydrated device as `linear` over its
destination, so the live device mapped the destination for the rest of
its life. The first checkpoint after the settle publishes, then deletes
the destination and base names to reach the normal layout, and that
delete failed with `Resource busy`: every first post-settle checkpoint
failed once, and a teleport that needed one aborted. Found on the dev
cluster with real guests; no data was lost, the retry recovered.

The settle now snapshots the destination's thin under the device's
suspend (the destination is suspended for its snapshot, as the pool
requires of an active origin) and reloads the device as that snapshot.
The live volume is then a thin like any other, shares only blocks with
the destination, and the destination deletes cleanly. A failed reload
resumes the clone table and deletes the snapshot; a crash between the
two leaves a thin with no device, which the pool open already deletes.

The `linear` paths go: `thin_under`, `open`, `delete`, `list`,
`hydration` and `parse::linear_device` know thin and clone only. The
snapshot and the settle share `create_snap`.

The fake hydrating disk stops sharing the destination's file at the
settle, so a delete of a destination the live volume depends on fails
in the unit tests too. The root test writes to the settled device,
reads the destination unchanged, and deletes the destination under the
open device. The settle log's `regions` is what the copy still had to
bring in when the driver first looked, not the device's total.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
nix: the dev shell tracks nixos-26.05, which has Node 26
ddbbab1 
#1736 raised the shell's Node to `nodejs_26`, but the locked nixpkgs
(nixos-unstable, 2026-04-22) has no such package, so `nix develop`
failed to evaluate in every fresh checkout. Today's nixos-unstable has
it but ships `sqlx-cli` 0.9.0, which has no cached macOS build and
fails to link from source. nixos-26.05 has Node 26.13 and `sqlx-cli`
0.8.6, the workspace's sqlx version, from the binary cache.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core: a restore picks its base by what it shares, not by name
060d2db 
A restore that did not find the session's kept checkpoint started from
the image version the snapshot named, and from nothing when the host
did not stage that exact version. With an image whose tag moves every
hour that is the normal case: the host holds a newer version that
shares most of its chunks with the session's disk, and the whole disk
became the copy.

`nearest_base` compares the wanted manifest with the lineage's kept
checkpoint and every staged image version and takes the volume that
holds the most chunks in place, the same hash at the same offset
(`shared_chunks`; the kept checkpoint wins a tie). No volume of the
manifest's shape that shares a chunk means no base. The base hint goes
from `Source::Fork` and `Source::Continue`; the restore no longer reads
the snapshot's `base_disk_manifest` for the disk. The eager path on a
host without a hydrating device takes the same base.

Tests: a session that began on one image version resumes from a moved
one and copies only what that version lacks; `shared_chunks` counts by
offset and refuses a different shape. The "without a base" test now
runs on a host that holds no volume of the image.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
dst-cosim: the simulated host forks without a base hint
0c78bb3 
`Source::Fork` lost its `base` field with the base chosen by content;
the co-simulation host still passed one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core: only a snapshot taken after the settle becomes a checkpoin…
c41df2e 
…t volume

A checkpoint taken while the volume hydrated and published after the
settle became the role's checkpoint volume: `publish` read the live
device's state, saw it settled, and renamed the earlier snapshot. That
snapshot holds base bytes in every region the copy had not reached. Its
manifest was right, but a resume of the lineage on the same host boots
the kept volume without a fetch, and `nearest_base` can pick it.

`take` now records whether the live volume was settled before the
snapshot, and `publish` decides by that. A partial snapshot is deleted
and the destination takes the published name, as during the window; the
next checkpoint makes the checkpoint volume. A pending checkpoint found
by name after a crash counts as partial.

Test: a snapshot taken with one of two chunks copied, a settle, then
the publish; no checkpoint volume appears until the next checkpoint,
whose volume holds the manifest's bytes.

Found by two independent reviews of the branch.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
disk-thin, host-core: a failed hydrate and an interrupted settle leav…
2b69363 
…e no metadata volume

Three ways a clone's metadata thin stayed in the pool, under a name no
sweep parses:

- `hydrate` kept its handle of the metadata device open across its own
  cleanup, so the delete after a failed create, discard or enable met a
  busy device; a failed device-number lookup skipped the cleanup.
- A process that died between the cutover and the metadata delete left
  it: the next `settle` saw a settled device and did nothing.
- A metadata delete that failed after the cutover was never tried
  again, for the same reason.

`hydrate` keeps only the path and builds the table inside the guarded
block. `SessionVolumes::settle` runs the disk's settle, which is the
cleanup on a settled device, while the role still has its destination;
`reattach` does the same for a role it finds settled.

The skip discard also refused a volume whose last chunk is short; it
now clips the discard to the volume. The metadata size comment gave the
bitset of a 2 million region volume; a 300 GiB root has 4.9 million.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-agent: the settle driver serves a hydrating disk first, for ever…
5efba72 
…y live sandbox

A reattach that failed at startup (a store error on either manifest)
was logged once and never tried again: the guest's reads of cold
regions and the kernel's copy stayed queued until the next restart. A
create from an explicit manifest hydrates like a resume and had no
settle driver at all, so its clone, source and base never went.

The driver now starts with `reattach`, retried like its other steps,
and takes no role list: it runs after a restore, after a create, and
for every live sandbox at startup.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-agent: the source stalls on a store error, and a guest read does…
4e99bce 
… not wait at Hydrate

- A fetch error became an I/O error on the guest's root disk. The
  source now tries again, from 100 ms doubling to 5 s, until the device
  stops; only a missing chunk, a hash mismatch and an out-of-range read
  fail.
- A read inside the stream window waited on the window's fetch at
  `Hydrate`, behind every other session's background work. A read that
  is not whole regions is the guest's: it goes to a ranged `Demand`
  read unless the chunk is already in. A window fetch that has waited
  1 s is asked again at `Demand`.
- The host-agent set the ublk source on every Linux host, so a node
  without `ublk_drv` failed every cold resume. It probes
  `/dev/ublk-control` at startup; without it, it logs an error and
  writes each volume in full before the guest starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
host-core: disk.hydrate says where its time goes
1062145 
`disk.hydrate` took 0.7 s of a cold resume on the dev cluster with no
span under it. The zeroing, the source's start and the device's create
each get one, and the first two run together: neither touches what the
other does.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
docs: the lazy resume's upgrade note, rollback rule and review facts
e044932 
- CHANGELOG: the upgrade note for the hydrating resume (`ublk_drv`,
  generation 3, drain before a rollback).
- ADR 0125 said no node carries a hydrating volume across the roll "in
  either direction". The drain acts only on hosts below the target's
  generation, so a rollback needs a drain by hand, as for generations
  1 and 2. The ADR and DESIGN.md also gain the facts the review fixes
  established and the memory a source holds.
- The root test's comments claimed a cold read and a death mid-copy;
  they say what the test proves.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@github-actions

github-actions Bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow CI / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed⏩ skipped⏩ skipped✅ passedNov 10, 2026, 5:14 AM

nikhilunni and others added 8 commits November 9, 2026 18:18
@nikhilunni @claude
ci: the root lane accepts a built-in ublk driver and says what the ke…
bdce025 
…rnel has

The runner's kernel has no `ublk_drv` module, so `modprobe` failed the
lane before any test. A kernel can also have the driver built in: the
step now needs `/dev/ublk-control`, not the module, and prints the
kernel's ublk and dm-clone configuration when the device is missing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
@nikhilunni @claude
ci: the ublk source's tests run on a kernel that has ublk
6815718 
The Blacksmith runner's kernel is built with `CONFIG_DM_CLONE=y` and
without `CONFIG_BLK_DEV_UBLK`, and has no module tree, so no step can
load the driver there. `blk_source` and `lazy_restore` move to a new
lane, `test-ublk-root`, on a GitHub-hosted Ubuntu runner, which ships
`ublk_drv` in `linux-modules-extra`. They boot no guest and need no
KVM. The dm-clone test stays in `test-fc-root`. The new lane is in the
gate's `needs` and runs under the Firecracker lanes' detector flag.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent: a read that meets a shutting-down host-agent waits for th…
ba93d5d 
…e next one

On SIGTERM the runtime drops the source's dispatch task, and the queue
thread, still alive, answered each new read with EIO. On the dev
cluster a restart during the copy gave the guest I/O errors on its root
disk and made dm-clone fail the copies in flight.

The queue thread now leaves such a read unanswered. The device has
`UBLK_F_USER_RECOVERY_REISSUE`, so the kernel gives it to the server
the next host-agent starts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent: only a read outside the window is a first touch
87253ad 
A read inside the stream window that is not whole regions goes to
`Demand` since 4e99bce, and was also recorded as touched. The host's
own probe of a new block device reads sector 0, which is in the window
of a new stream, so every resume recorded chunk 0, and the root test
`a_source_serves_the_manifest_and_records_the_chunks_read_cold` failed
on a real kernel. A chunk in the window arrives without a prefix
fetch; only a read outside it is recorded.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent, disk-thin: the copy is limited by the pool's disk, not by…
3c9c547 
… its own window

Measured on the dev cluster, a 31.7 GiB difference beside a 13 GiB
memory fill: with 4 chunks fetched ahead and 8 MiB of kernel copies in
flight the copy averaged 264 to 376 MiB/s and settled in 86 to 123 s;
the samples show the kernel's copies waiting on the source or on their
own in-flight limit while the host had idle CPU and link. With 12
chunks ahead and 32 MiB in flight it ran at 760 to 1017 MiB/s steady
(608 MiB/s average, settled in 53 s) with the pool's backing disk at 85
to 100% utilisation. The host-agent's peak RSS did not grow, and the
guest's read gaps and prompt timings were not worse.

The ADR records the numbers, the 224 MiB a source can hold per
hydrating volume, and two limits that stay: dm-clone starts no
background copy while guest I/O is in flight on the device, and a
checkpoint in the window reads every copied block back from the same
disk, which nearly stops the copy while it runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
ci: the ublk lane loads the module before it looks for a package
bee8b64 
The GitHub runner's kernel (7.0, azure) has no
`linux-modules-extra` package, so the install failed before the lane
tried the module. The lane loads `ublk_drv` first and installs the
extra package only when that fails.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent, disk-thin, helm: the two hydration limits are settings
fd710d4 
The source's window (whole chunks fetched ahead of the kernel's copy)
and dm-clone's `hydration_threshold` (regions the kernel copies at one
time) decide how fast a resumed disk copies in and how much memory and
pool I/O it takes. The right values depend on the node's disk, so the
host-fleet chart sets them (`thinPool.hydration.windowChunks`,
`thinPool.hydration.inFlightRegions`) through
`ENGRAM_DISK_HYDRATION_WINDOW_CHUNKS` and
`ENGRAM_DISK_HYDRATION_IN_FLIGHT_REGIONS`. Unset, the host-agent uses
12 and 512 as before. The batch size stays a constant.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
fill: a whole chunk read for a volume is not hashed
9f8c125 
`FillEngine::read` hashed every chunk it fetched from the store. In a
profile of a resume that copies 31.7 GiB, SHA-256 was 9 to 13 percent
of the host-agent's CPU. A chunk is checked when it is written to the
store or to the chunk cache; a read does not check it again.

This changes every caller of `read`: the hydration stream, the eager
stage of a volume, the checkpoint of a partly changed chunk of a
hydrating volume and the substrate server's whole-chunk disk reads. `get` (a cache fill) checks as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
nikhilunni and others added 2 commits November 9, 2026 19:32
@nikhilunni @claude
host-agent: the periodic checkpoint waits for the disks to settle
6f0eacc 
A checkpoint taken while the kernel copies a disk in reads every copied
block back from the pool's disk, because a copied block and a written
block look the same in the pool's mapping. On the dev cluster two such
checkpoints dropped the copy from about 260 MiB/s to 7.6 MiB/s for 28 s.

The periodic driver skips a sandbox whose settle driver still runs. The
hold ends 10 minutes after the driver started: a driver that cannot
settle runs for ever, and the session still needs checkpoints. An
eviction or a snapshot does not wait.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
docs: the hydration settings, the unhashed read and the checkpoint hold
ef3b469 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni
nikhilunni marked this pull request as ready for review November 10, 2026 03:41
nikhilunni and others added 4 commits November 9, 2026 19:44
@nikhilunni @claude
disk-thin: the dm-clone table names its arguments
b864319 
`fd710d469` put the in-flight limit first in the positional list, so
the table named the limit as the metadata device and the kernel refused
it (ENOENT). The `clone` and `lazy_restore` root tests caught it. The
format string names each argument.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
ci: the ublk lane shows where a hung test waits
67f05f9 
On the GitHub runner's kernel (7.0)
`a_source_serves_the_manifest_and_records_the_chunks_read_cold` did not
end, the signal from nextest's slow timeout did not end it either, and
the job ran to its 30-minute limit with no output. A thread blocked in
the kernel on a ublk device cannot be killed. After 10 minutes the lane
prints each test thread's kernel stack, the ublk devices' in-flight
counts and the kernel log, then fails.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent: a source device gets no partition scan
357b39f 
Kernel 7.0 scans a new ublk device for partitions in a work item, and
an open of the device waits for that scan. The scan reads the device,
so the source must answer while the open waits. The `blk_source` test
opened the device on its one runtime thread, which is the thread that
answers, and never ended on the CI runner (kernel log: the test blocked
on a mutex owned by `ublk_partition_scan_work`, one read in flight).

The source sets `UBLK_F_NO_AUTO_PART_SCAN`. Its bytes are a guest's
disk, and the host has no use for a partition table a guest wrote. An
older kernel does not know the flag and scans as before. The test opens
the device off the runtime, as it already reads it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni @claude
host-agent: the source does not set the no-partition-scan flag
4317b0c 
libublk 0.4.8 checks the control flags against its own list, which does
not hold `UBLK_F_NO_AUTO_PART_SCAN`, and refuses the device before the
kernel sees it. Every serve failed with "Invalid input". The flag goes;
the test's open off the runtime thread, which is what the hang on
kernel 7.0 needed, stays.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
@nikhilunni
nikhilunni merged commit 8cec598 into main Nov 10, 2026
28 checks passed
@nikhilunni
nikhilunni deleted the disk/lazy-through-fill branch November 10, 2026 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nikhilunni