Repository navigation
disk: lazy session disk, steps 1 and 2 (ADR 0127) - #1747
Closed
nikhilunni wants to merge 38 commits into
Closed
nikhilunni wants to merge 38 commits into
nikhilunni wants to merge 38 commits into
Conversation
A restore that cannot finish in about two seconds boots over dm-clone at once: reads of uncopied regions go to a read-only source that serves the manifest, writes go into the thin volume, and a background copy fills the rest. When the copy ends, a cutover to a linear table leaves an ordinary kernel-owned volume. The source holds no writes and lives only while its sandbox copies, in a process outside the host-agent, so ADR 0125's ownership rule holds. A source failure stalls reads and never ends a session. This amends ADR 0125's resume contract, which wrote the whole disk before boot when the host had no staged base for the session's image version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
…arned 0126 is taken on main. The ADR now states the production node image facts (dm-clone built in, ublk_drv a module that node-prep loads), the first source's recovery result, the single-core bound in today's chunk fetch paths, and two rules: a read that fails for lost data returns an I/O error after one fresh fetch while every other failure is retried, and a ublk device is never stopped while it holds a request that nothing will finish, because that blocks in the kernel until the node reboots. Decision 1's rates are measured per host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
`engram-disk-source` serves one disk manifest as /dev/ublkbN. A read takes each chunk from the first tier that holds it: a staged volume with a chunk of the same hash, the host chunk cache (read-only), then the chunk store. Offsets with no chunk read as zeros. One fetch serves every concurrent read of a chunk, and each fetch runs in its own task so the work spreads across cores. The device is read-only and created with user recovery and reissue: when the process dies the kernel keeps the device and its queued reads, and `serve --recover <id>` attaches a new process. State is a function of the arguments only. A read that fails for lost data (the store does not hold the chunk, or its bytes fail the hash check) is refused after one fresh fetch; every other failure is retried with backoff. A stop first ends reads still retrying, because the kernel does not finish stopping a device while a request is in flight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The tests read a served manifest through /dev/ublkbN and compare every byte, check that writes fail, kill the server mid-read and recover it with no failed read, and stop a server that holds a read it cannot serve. They run on the Firecracker root lane with ENGRAM_TEST_UBLK=1; the lane loads ublk_drv, from the kernel's extra modules if needed. The lane detector runs that lane when the crate's closure changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
A chunk the store does not hold is lost data, so its read now fails with EIO after one fresh fetch instead of waiting. The stop test used a deleted chunk to make a read wait, which no longer waits. It now puts a directory where the chunk's blob should be: the store fails without reporting the chunk absent, the read retries, and the stop must end it. A new test reads a deleted chunk and expects EIO while other reads still succeed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The memo evicts only a cell that no reader holds. A cell whose fetch is in flight stays, so a second read of that chunk waits on the same fetch and does not start another. When more chunks are in flight than the memo keeps, the memo holds more cells until those reads end. A chunk read from the base volume must hash to the chunk the manifest names. Bytes that do not are logged and the next tier serves the chunk. The reader builds its tiers before it shares them, so adding a tier cannot panic. The parallel-fetch test states what it shows (the fetches overlap) and asserts a bound well under the serial time, not one near one fetch. Review pass: the memo comment about the touched entry being newest no longer held and is replaced by the hold-count rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The policy that tries a failed read again, fetches lost data once more and ends a read when the server stops is in `policy`, which builds on every platform. The ublk server calls it. Unit tests drive it against a faulty store with a paused clock: a transient error is tried again until the read succeeds, a chunk the store does not hold is fetched twice and then refused, and a stop ends a read that is trying again. The select between a read and a stop is biased to the read, so a stop never refuses a read that is ready. A transient failure logs a warning on attempts 1, 2, 4, 8 and so on, and at debug level otherwise, so an outage does not log an error for each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
…nother shape A read runs on the tokio runtime, and the tag's task waits on an eventfd that the runtime side signals. The signal is now written when the runtime side ends in any way, after its result is sent or its sender is dropped. A read that panics thus gives the kernel EIO, and does not leave a request that blocks the stop and delete of every device on the node. The eventfd poll is armed again when it ends with EINTR, EAGAIN or ECANCELED while the read still runs. Other ring errors still end the request. libublk sizes its per-queue state from the queue count it is given, but starts one thread per queue of the device. A recovery whose queue count or depth differs from the device's is refused with an error before recovery starts; the device stays for a recovery of the correct shape. The `--queues` and `--depth` help says that they must match with `--recover`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
A source that gives no ready line in time is killed and reaped. A test that fails between a crash and its recovery recovers the device and stops it, because a device that no process serves holds its I/O, and a read blocked on it keeps the test process from exiting. A nextest override ends a test of this binary after 2 minutes. A new test recovers a device with another queue count: the source exits with an error (not a panic) and the device then recovers with the correct shape. The CI step updates the package index before it installs the extra kernel modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
`delete` held the bounded wait for the last holder and the node removal inline, and `activate` held the cleanup after a failed open. Both move into helpers (`remove_device`, `open_created`, `node`), so every device the disk makes is opened and removed the same way. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
A cloned drive is the guest's disk before its bytes are local: reads of regions not yet copied go to a read-only source, writes land in a destination volume, and a bitmap volume records the copied regions. `SessionDisk` gains the operations a lazy restore and a checkpoint during the copy need: - `clone_drive`: a drive over a destination, a bitmap and a source, with hydration on or off and its threshold and batch size. - `open_clone`, `clone_state` (copied and total regions, hydration on or off, or cut over) and `enable_hydration`. - `cut_over`: refused with `ResourceBusy` until every region is copied; idempotent after. - `snapshot_clone`: the destination and the bitmap snapshotted with no copy between them. - `clone_view`: a read-only drive over such a pair and the source that reads what the guest saw. - `remove_clone`: a drive or a view; its volumes stay. `ThinPoolDisk` makes the drive a dm-clone device under the node directory, named like a volume, region 64 KiB. The cutover suspends the device, loads `linear` onto the destination and resumes it, so the node and every open handle stay valid. `snapshot_clone` suspends the drive while it takes both snapshots: the background copy does not stop for a guest pause, and without the suspend the bitmap snapshot could record a region the destination snapshot does not hold. A view is a second clone table, loaded read-only with hydration off. The device calls reuse the existing ioctl layer, which gains a read-only create, a table load into the inactive slot and a blocking resume for a drop guard. The table and status formats are pinned by tests against strings recorded on kernel 6.8. `FileDisk` (macOS and the simulator) has no dm-clone and hydrates eagerly: `clone_drive` copies the source into the destination, the drive is a hard link under `.drives/` that `list` does not read, and `cut_over` records the state only. Like `ThinPoolDisk`, it refuses `enable_hydration` and `snapshot_clone` after the cutover. The simulator's `CrashDisk` was deleted with engram-dst-host, so the crash test carries a test-local wrapper with the same contract: a lazy restore and its cutover, cut at each of its seven disk operations, converge on the next pass to the source plus every guest write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
Three tests over the pool, with a read-only loop device as the source: - Reads of uncopied regions equal the source and copy nothing; a partial and a whole-region write both land; the cutover is refused while regions are uncopied; after hydration the cutover keeps the node, its device number and a handle opened before it, and the destination equals the source plus every write. - A checkpoint view: with the source behind dm-delay, the destination and bitmap are snapshotted while the copy runs and after guest writes. The view over the snapshots reads exactly the guest-visible image (the destination snapshot alone reads zeros where nothing was copied), copies nothing, refuses writes, and still reads the same after the drive cuts over and its bitmap is deleted. - What a drive refuses: a source smaller than the destination, a taken name, a snapshot pair that is not the drive's, `remove_clone` on a thin volume or while open, hydration after the cutover. Each test ends with no device of its own left, and its drop guard removes the devices a failed test leaves. The dm-thin step of the root lane runs the new target and loads dm_clone and dm_delay, from the extra modules when the runner kernel does not ship them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
The background copy does not stop for the guest pause. The checkpoint syncs the drive, which commits the bitmap with every region the guest wrote, then suspends the drive while it snapshots the bitmap and the destination, so the bitmap snapshot never records a region that the destination snapshot does not hold. The view is loaded read-only. The root lane proves the view on kernel 6.8. The Design section names the drive operations on `SessionDisk` and states that `FileDisk` hydrates eagerly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
…h left A drive operation suspended the device through an ioctl on a blocking thread. When the caller was dropped during that await, the device stayed suspended and nothing resumed it. A failed resume cleared the guard, so the drop did not retry it. A process death between the suspend and the resume left the device suspended until a person resumed it. Two operations on one drive could resume each other's suspend. - `cut_over`, `snapshot_clone` and `snapshot` run the suspend, the operation and the resume in one task of their own, so a dropped caller cannot stop the sequence. - `Suspended` stays armed until a resume succeeds. It retries a failed resume five times, and the drop tries once more and logs the device as left suspended. - A per-drive lock serializes `clone_drive`, `enable_hydration`, `cut_over`, `snapshot_clone`, `clone_view` and `remove_clone` on one name. - `ThinPoolDisk::open` resumes every suspended device of its pool, thins before the drives and views on them. - A failed `snapshot_clone` resumes the drive first, then deletes only the snapshots it made. `FileDisk` cleans up the same way, so a retry does not find `AlreadyExists` on one disk only. Root tests: a drive and its destination left suspended resume when the pool is opened again, the waiting write completes and the copy runs to the cutover; a snapshot name that is taken fails the call, keeps the volume of that name, and leaves the drive running. The test rig resumes leftover devices before it removes them. Review pass: `snapshot` had its own unguarded suspend with no resume retry; it now uses `Suspended` in a detached task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
…k refuses `SessionDisk` now states what a drive owns and what each disk refuses, and `FileDisk` keeps that contract with files: - A drive takes a bitmap that no drive has used. Its first 4 KiB, where the kernel writes the dm-clone superblock, must read as zeros (`check_fresh_bitmap`, read past the page cache with `Volume::read_direct`); else `InvalidInput`. A destination or bitmap that another drive or view uses is `ResourceBusy`. - Volumes, drives and views share one name space: a new name that is taken is `AlreadyExists` on both disks. `delete` of a volume that a drive or view uses is `ResourceBusy`. A drive or view is not renamed (`InvalidInput`). - A view opens read-only, and `cut_over`, `enable_hydration` and `snapshot_clone` refuse it. - `CloneState::Incomplete` names a `clone_drive` that did not finish: the caller removes the drive and makes it again on a new bitmap. - `remove_clone` states the one difference that stays: `FileDisk` cannot see open handles. `FileDisk` keeps each drive or view `d` as files under `.drives/`: `d` links the destination and exists only once the rest is made, `d.bitmap` links the bitmap while the table would read it, and a `d.cut-over` or `d.view` marker. A volume with more than one link is in use. One helper, `FileDisk::drive`, decides what a name is. `clone_drive` writes a header into the bitmap, as the kernel does. The crash-convergence test writes different bytes on each pass and checks that the second pass keeps the first pass's writes, so a pass that copied the source over a live destination fails it. The restore it drives handles `Incomplete`. New `FileDisk` tests cover the fresh bitmap, the volumes in use, the shared name space, views, and the cleanup of a failed `snapshot_clone`. Review pass: `remove_clone` and `open_clone` of a volume's name gave `NotFound` on `FileDisk` and `InvalidInput` on `ThinPoolDisk`; both give `InvalidInput` now. The `CLONE_REGION_BYTES` doc claimed the pool's block size; it now says it equals the default and is part of the on-disk format. A long doc line in `lib.rs` is wrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
`ThinPoolDisk` keeps the `SessionDisk` contract of the previous change: - One helper, `drive_kind`, decides what a name is from its live table and the device's read-only flag: a read-write `clone` copies, a `linear` is cut over, a read-only `clone` is a view, and no table is `Incomplete`. It replaces three separate decisions (`is_drive`, the `clone_state` match, `remove_clone`). - `clone_drive` refuses a taken name first, then a destination or bitmap with a holder in sysfs (`ResourceBusy`), then a bitmap whose superblock area is not zero (`InvalidInput`). - A view opens read-only and refuses `cut_over`, `enable_hydration`, `snapshot_clone` and `rename`; a drive refuses `rename`. - `clone_view` reads the region size from the bitmap snapshot's dm-clone superblock (`parse::clone_superblock_region_sectors`; the magic at byte 32 and the region size at byte 172, checked against a bitmap on kernel 6.8), so a view composes with the size its drive used. - `dm::Dm::info` reports the suspended and read-only flags, and `dm::holders` counts the devices stacked on a volume. Root tests: a half-made drive, with no table and with its table loaded inactive (which writes the bitmap header), is reported `Incomplete`, removed and made again on a new bitmap, and converges; the refusal test adds a small or used bitmap, a destination or bitmap in use, a delete of a volume in use, and a rename of a drive; the view test adds a read-only reopen and the refused drive operations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
… region format - Decision 8: with the bitmap snapshot taken first and the drive synced, the pair is consistent, because the kernel records a region in the bitmap only after its copy is in the destination. The suspend makes the pair consistent in either order and costs milliseconds inside the existing pause. - Design: a drive takes a bitmap no drive has used and refuses volumes another drive or view uses; operations on one drive are serialized. The region size is part of the on-disk format, so a change of `CLONE_REGION_BYTES` is a format change; a view reads the size from its bitmap snapshot's superblock. - Host-agent restart: opening the pool resumes devices a death left suspended, and a drive with no live table is removed and made again. - Terms and Testing match the code and the new root tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpn1PPfqUmqAHEAPeQVF
With one task per chunk fetch, the network is no longer the limit. The disk diff waits on direct writes into the thin volume, and the memory prefetch shares that disk through the chunk cache. Decision 1 rates are host write rates under a concurrent memory restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
|
The latest Buf updates on your PR. Results from workflow CI / buf (pull_request).
|
The root lane's runner kernel has no dm-clone module and no package of extra modules. A built-in target has no module either, so the steps now check the kernel's target list and the ublk control device, and say which feature is missing. A report step prints the kernel version, its targets and its config. The ublk step runs even when the dm-thin step fails, so one run reports both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
Planning step 3 against the code found places the ADR was wrong or silent: - The source record is a durable record of its own. sandbox.json belongs to the Firecracker backend and the orphan reap deletes it. - A dead source blocks a suspend, so each suspend checks the source. - The cutover waits for checkpoints in flight, and the source outlives the VM while an eviction's upload reads through a view. - The upload diffs the destination against its origin; with no staged base, a checkpoint hashes the whole disk. - A live move during hydration is refused; a release keeps nothing. - The volume names carry the lineage while a role hydrates. - A flag turns the lazy path on, after a build that knows the names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
The Blacksmith runner kernel has dm-clone and dm-delay built in but no ublk driver, and has no package of extra modules. The hosted 24.04 image's kernel fails on a ublk device add. The new lane runs on the hosted 22.04 image (kernel 6.8, the driver in its extra modules), with its own detector flag and a place in the CI Gate. The Firecracker root lane no longer runs the disk source tests, which it cannot load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
This is the expand step for the lazy session disk. This build makes no cloned drive and no lazy volume. It recognizes and removes them, so that a later build that makes them can roll back to this one. Names for role r of sandbox s: - sb-<s>-<r>-dest-<manifest>-v<N>: the destination (Kind::Dest). It holds the lineage like a checkpoint volume. - sb-<s>-<r>-bitmap: the copy bitmap (Kind::Bitmap). - tmp-sb-<s>-<r>-bm: the bitmap snapshot of a checkpoint in flight (Kind::BitmapSnapshot). - tmp-sb-<s>-<r>-view: the read-only view, a drive (Kind::View). - sb-<s>-<r>: the live name, which is a drive for a cloned role. The longest name is still 100 bytes. Existing names parse as before. Behaviour for a cloned role (a role with a dest, bitmap, bm or view): - roles() counts it. list() does not show drives, but a drive holds its destination, so the role always has a listed volume. - path() gives the drive's node through open_clone. - live() stays a volume and returns Unsupported for a cloned role. - take, take_precopy and checkpoint return Unsupported before they touch a volume. A destination reads zeros where the drive has not copied, so a capture of it would publish a wrong disk manifest. - pending() returns Unsupported when a cloned role has a pair in flight, and NotFound when it has none. - checkpoint_of() gives the newest of the checkpoint and dest versions. Thus the eviction-finalize redrive accepts a publish that renamed the destination. - release, purge, reap and reconcile remove the view and the drive first, then delete the dest, bitmap and transient volumes. A checkpoint volume older than the destination is deleted, so a release during hydration keeps no checkpoint volume. A checkpoint newer than the destination (after the cutover) is kept. The happy paths of live() and path() do not list the disk again. They list only after open fails. Teardown calls remove_clone only for roles that have a lazy volume. Thus the cost for today's sandboxes does not change. The review pass found and fixed these items: - The four name builders repeated one format and one expect. Now one role_name builds every name. - parse removed a suffix from the role in a nested match. Now it splits the role from its suffix one time. - "Which volumes hold the lineage" was coded in checkpoint_of and in sweep. Now Kind::lineage is the one definition, and checkpoint_of, take, pending and sweep use it. Tests: round trips for every name, and that malformed names do not parse. On FileDisk, with a cloned role made by hand: roles, path and checkpoint_of; every capture refuses and leaves the names as they were; release, reconcile and purge leave no drive and no volume. Also a test for the eviction-finalize redrive over a destination. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
A lazy source makes a session role a cloned drive. The sandbox boots at once, reads the regions that are not yet local from a read-only source that serves the manifest, and the drive copies the rest in the background. - DiskSources (new module disk_sources): the source of one role. start, ensure_alive, stop and list. start takes the manifest and an optional staged base volume with its manifest. It returns the device path that the drive reads. FileSources stages the manifest into a file for FileDisk, which copies a source in full before clone_drive returns. - Source::LazyFork and Source::LazyContinue. Each has the same lineage rule as Fork and Continue. They are new variants, so the eager callers and the eager arms do not change. Every arm of provision_role now gives the live path and the lineage. - clone_role makes the bitmap volume, then the destination (a snapshot of the staged volume of the manifest or of the base when this host has one, else a new volume, named dest_name after the lineage), starts the source, and makes the drive with hydration on. A role without disk sources gets Unsupported, not an eager restore. On a failure, the purge in provision removes the drive, the source and the volumes. - clone_bitmap_bytes is the one size for a bitmap volume: two bits for each region, 1 MiB more, and at least 4 MiB. The root tests in clone.rs use it in place of their fixed 4 MiB. - Teardown (release, purge, reap, reconcile) does these steps for each cloned role and each role that has a source: recover the source, remove the view and the drive, stop the source. Only after that are the volumes deleted. A step that fails stops the steps after it for that role. Thus no drive is removed over a dead source, and no source stops under a drive. The review pass found these items: - Fork, LazyFork and Empty each wrote a new lineage in three copies of the same code. Now new_lineage does it for all three. - The snapshot of the live volume came after the match, which a lazy arm cannot share. Now live_over is that step for each eager arm. Tests: on FileDisk with FileSources, a lazy Fork and a lazy Continue over a staged base read back the manifest at the live path. A provision that fails after the swap source started leaves no drive, no source and no volume. A lazy request without sources is refused and leaves nothing. A release recovers the source, removes the drive, then stops the source, and leaves no volume. Also a root lane test in clone.rs: SessionVolumes over ThinPoolDisk, with the source a read-only loop device over the staged file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
A cloned role now checkpoints in each of its states. A checkpoint is no longer refused. A live move is still refused. take, for a cloned role: - While the drive copies: recover the source (a suspend waits for the source's reads in flight), sync the drive, then snapshot_clone the pair (tmp-sb-<s>-<r> and tmp-sb-<s>-<r>-bm). - After the cutover, with the destination as the lineage holder: sync the drive and snapshot the destination. - After the first -ckpt- that is newer than the destination: the same snapshot, which publishes as an ordinary difference from that -ckpt-. PendingCheckpoint now records its shape (Volume, Hydrating with its bitmap snapshot, or CutOver). pending() makes the shape again from the names alone. publish: - Hydrating: make a view over the pair and the role's source. The manifest is the source's manifest, with each chunk that the destination snapshot can have written since its origin replaced by what the view reads there. A chunk equal to the source's chunk is not uploaded. The destination then takes the new version's name, and the view and pair are deleted. A crash at any step is redriven from the names: the view is made again. A crash after the rename publishes the pair again under the next version, with the same bytes. - CutOver: hash every chunk and publish an ordinary -ckpt- volume. The destination keeps its name, so the -ckpt- is the newest holder, and release keeps it. Supporting changes: - SessionDisk::written_chunks lists the chunks that a write can have changed since a snapshot was made from its origin. The list does not depend on content. The content compare of dirty_blocks is not correct here: a guest write of the origin's bytes over a source chunk with different bytes leaves the content equal to the origin. The default lists every chunk, which FileDisk uses. ThinPoolDisk uses thin_delta. With no origin, it compares against an empty thin that it makes for the read and then deletes. - DiskSources::served gives the source's device, its manifest and the manifest of its base. After the first publish, the destination's name no longer gives the manifest that the source serves. The origin is the staged volume of that base. FileSources keeps these in a "<file>.served" file next to the source file. - checkpoint_split::upload_over is the upload over a given base and chunk list. upload now uses it, and does not upload a chunk equal to the base chunk at the same offset. The review pass found these items: - ThinPoolDisk's create and its delta read repeated the code of dirty_blocks. Now create_thin and delta do these jobs. - take branched two times on one flag. Now one match gives the shape. - Kind::lineage and the new holder kind were two definitions of the same fact. Now Kind::holder is the one definition. Tests on FileDisk with FileSources: - A checkpoint while the drive copies, with a partial write and a whole-chunk write, stages into exactly what the drive reads. - A take cut short, and a publish that left a view, publish from pending(). - After the cutover, the first checkpoint is a full -ckpt-, the second one diffs (changed_blocks gives one chunk), and release keeps the newest -ckpt-. Root lane (clone.rs): the same "the staged manifest equals the drive" check while hydration runs over a loop source behind dm-delay. The loop source helper is now a SessionRig shared with the earlier lazy test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
Fixes for the findings of an independent review of the lazy checkpoint, and for a root lane test that failed in CI. - A destination snapshot alone is never published as a whole disk. A failed snapshot_clone now deletes the destination snapshot first. pending() takes the CutOver shape only when the kernel reports the drive as cut over. A destination snapshot with no bitmap snapshot while the drive copies is InvalidData, and the next take replaces it. - One place removes a view or a drive: remove_clone_of. It recovers the role's source first, so take, discard, publish and teardown all get that step. - A role whose take-down fails keeps its volumes. take_down returns the failed roles, and purge and sweep do not delete their names. - The first checkpoint after the cutover hashes every chunk against the durable previous manifest. It uploads only the chunks that differ, and does not send a store HEAD for each chunk. - upload_over checks that the volume and the manifest have the same shape. clone_role checks the destination against the wanted manifest. - After the cutover and a checkpoint volume newer than the destination, a role opens and moves as a plain role does. Only the kernel knows whether a drive still copies, so written() is the one place that asks it: live writes go to the live volume, to a cut-over destination, or to a destination that is still copying. live() opens a cut-over drive as a volume. take_precopy snapshots the destination once a -ckpt- volume holds the lineage, and refuses before that. - An upload error is no longer replaced by the error of the view removal that follows it. - Two lineages for one role are now an error (InvalidData) in checkpoint_of, take and pending, not a silent pick of one. sweep reports such a role and keeps its volumes. - DiskSources::stop waits a bounded time for a short hold on the device, such as a udev probe, then fails with ResourceBusy. host-core does not assume that the kernel has released the device: a failed stop keeps the role's volumes for the next sweep. Root lane (clone.rs): - A new test makes the destination a snapshot of a staged image volume. It uses a LazyContinue of a manifest that differs from the base in regions not yet copied, and a guest write of the base's own bytes. The staged manifest equals the drive. This test exercises the origin branch of written_chunks. - The loop source removes its dm-delay device with `dmsetup remove --retry` in a bounded loop. It no longer holds the device lock while it runs commands that can fail. Cleanup now goes on after a poisoned lock. The review pass found these items: - take and take_precopy each decided the drive state in their own way. Now written() is the one decision. - The holder rule was in lineage_in and sweep, each with its own version compare. Now holders() is the one rule, and it is the place that finds two lineages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
A review of the lazy checkpoint found two conditions the ADR did not state: a rollback to a build without source support while sandboxes hydrate can meet a dead source, and the chunk selection depends on a guest that cannot discard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
ProcessSources is the production DiskSources (ADR 0127 Decision 4 and
5). Each source is an engram-disk-source serve process:
- It is spawned without kill_on_drop. Its stderr goes to a log file in
<work_dir>/disk-sources, never to a pipe. The host-agent reads one
ready line ("<device path> <device id>") from stdout, with a timeout.
- It moves to the node cgroup leaf <parent>/<sandbox>-disk-<role>. It
does not share the Firecracker leaf, because a source can outlive
its VM. escape_to_node_cgroup and remove_node_cgroup are now public
and take a leaf name.
- A durable record <work_dir>/disk-sources/<sandbox>-<role>.json holds
the pid, the process start time (so that a reused pid is not taken
for the source), the device id and path, the queues and depth, and
the serve arguments. It is written once after the spawn and again
after the ready line, so a host-agent that dies at any point leaves a
record of the process.
ensure_alive starts a dead source again with --recover and the
recorded arguments, queues and depth. A record whose device is gone,
or whose device id another record holds (the kernel reuses the id of
a deleted device), is removed and gives NotFound. A start that did not
report its device is stopped. stop recovers a dead source first, then
sends SIGTERM (the server fails the reads it cannot serve and deletes
the device), then SIGKILL after a timeout. If the device stays, stop
keeps the record and fails, so that the next stop recovers the device
before it stops it again. A failed recovery ends with SIGKILL, which
keeps the device for the next recovery.
The host-agent calls recover_all after the pool opens and before the
volume reconcile. The flag --lazy-restore (ENGRAM_LAZY_RESTORE:
off|force|auto) is off by default. When it is off, the host starts no
source and behaves as before. Any other mode gives SessionVolumes
ProcessSources over a thin pool, and FileSources over sparse files.
This change does not change the restore path. A host whose
/dev/ublk-control is absent logs why at start and refuses each start.
durable_record gets load (one record, an error when the record is
torn) and delete. The residue sweep test now shows that the sweep
keeps disk-source records and logs.
Shipping: the host-agent image, the musl host-binaries artifact and
the detector closures include engram-disk-source. Node prep loads
ublk_drv before the pool check, so a node whose pool is already active
loads it too. A node without the driver continues; the host-agent then
refuses to start a source.
Tests: unit tests with a fake probe and a shell script in place of the
binary, for the record round trip, a record torn at every byte offset,
a missing record, garbage siblings, and the restart decision (a gone
pid, a reused pid, a gone device, a device id that another record
holds). A root test, disk_source_ublk, kills a source while a read
waits on its device, calls ensure_alive, checks that the read gives
the manifest's bytes, and checks that stop leaves no process and no
device. It runs in the test-ublk lane, which now also builds the
host-agent; engram-host-agent is in that lane's detector closure.
Review pass:
- durable_record: load and load_all now share one decode step instead
of two copies of the envelope and body checks.
- The detector's host-agent image closure was a second list of the
FC_BINS binaries; it now uses FC_BINS.
- The cgroup helper is shared with a leaf name, not copied.
- start used a mutable result chained through flags; one async block
now carries the steps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
The hosted 22.04 image's protoc (3.12) refuses proto3 optional fields, so the host-agent disk-source test did not build. The lane now installs the protoc release that the image bake pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
SessionVolumes::cut_over(sandbox, role) ends the hydration of a cloned role (ADR 0127 Decision 7). When every region is copied and no checkpoint of the role is in flight, it makes the source serve, cuts the drive over to a linear table onto the destination, deletes the bitmap volume and stops the source. The linear table maps the destination only (engram-disk-thin cut_over_locked), so after it nothing reads the bitmap or the source. A cutover that cannot go ahead is not an error. It returns Cutover::Waiting with the reason: regions still to copy, or a checkpoint in flight. A checkpoint is in flight when the snapshot pair (tmp-sb-<s>-<r>, -bm) or the view exists, also when a crash left it: pending() publishes such a pair as Hydrating, which reads through a view over the source. A lone tmp volume also makes the cutover wait: before the cutover pending() refuses it, after it pending() would publish it as a complete disk. The next take replaces it. The driver treats a Waiting as "try again later" and logs only errors, so the expected states do not hide behind an io::ErrorKind that real failures also use (ResourceBusy). The caller serializes cut_over with the take and publish of the role. The host-agent holds the capture lock of the sandbox across both. Each step is safe to cut. After the kernel cutover, the drive state is CutOver, written() and take_down already treat a cut-over drive with its bitmap and source left as a cut-over role, and the next call deletes the bitmap and stops the source. hydration(sandbox, role) reports what is left: Copying, Ready (hydrated, or cut over with the bitmap or source left), or None when nothing is left. While the drive copies, it also makes the source serve, so a periodic driver restarts a dead source (Decision 5). Tests: on FileDisk and FileSources, a cutover waits while a pair or a view exists and goes ahead after the publish; a cutover cut after the kernel step, and one cut after the bitmap delete, are finished by the next call, and the role checkpoints and releases between; after the cutover the first checkpoint is a -ckpt- volume. On the root lane, a role hydrated to the end cuts over with a handle open, reads the manifest and the guest write, and leaves no bitmap, source or loop device. Review pass: the error context "source of the <role> disk of sandbox <s>" was written in two places; it is now one helper, source_err, which the new calls also use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
A sandbox whose disks restore over a source gets a hydration task (ADR 0127 Decision 6). The task lives in the new hydration module and belongs to the sandbox like its hint poller: PooledBackend holds one per sandbox, destroy stops it, and the task ends when no disk is left to cut over. Each tick (1 s, HostAgentConfig::hydration_interval) the task reads SessionVolumes::hydration for each disk, which also restarts a dead source while the drive copies. When a disk is ready, the task takes the capture lock of the sandbox and calls SessionVolumes::cut_over, so a cutover never runs between the take and the publish of a checkpoint. A failure is logged when it changes and tried again on the next tick; no path kills, evicts or rewinds the VM because a source failed (Decision 5). Nothing restores lazily yet. PooledBackend::arm_hydration is the entry point for the lazy path in provision_for_restore, which comes with the restore gate. At startup, rearm_hydrations runs after the source recovery and the volume reconcile, and arms the task for each live sandbox with a disk that still copies or whose cutover has steps left. The heartbeat carries the sandboxes with a task as `hydrating` (Decision 9). The heartbeat is JSON and the field is serde(default) in the host request and in the protocol Heartbeat, so an older payload decodes as none and the wire version stays. The coordinator does not read the field yet; serde drops it there. Metrics: engram_disk_hydrating_sandboxes (gauge), engram_disk_hydration_seconds (histogram from arm to the last cutover, on the capture buckets), engram_disk_cutovers_total by outcome. Tests: a re-armed task finishes a cutover cut short after the kernel step and ends, and a sandbox with no cloned disk needs no task; destroy stops a task that waits on a checkpoint pair; the protocol heartbeat round-trips `hydrating` and decodes an older payload without it. Review pass: the capture-lock lookup was written inline in capture_lock; it is now capture_lock_in, which the task shares. The AbortTaskOnDrop doc comment named its first two users instead of the fact; it now states the fact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
provision_for_restore now picks, for each disk, an eager source
(Fork/Continue) or its lazy counterpart (LazyFork/LazyContinue), from
the --lazy-restore mode (ADR 0127 Decision 1):
- off: eager, as before. Nothing new is read.
- force: lazy for each disk with a manifest. A disk of zeros stays
zeros.
- auto: lazy when the estimate of the eager writes is more than the
budget. When the estimate cannot be read, the disk is eager.
A host without disk sources is always eager. force and auto record the
estimate.
The estimate is one pure function, eager_estimate::estimate in
host-core. Its inputs are the wanted manifest, the manifest of the
volume the eager path starts from, a "chunk is local" predicate (the
chunk cache holds it on disk) and the two write rates. Its result is
local bytes / local rate + remote bytes / remote rate. The rates are
inputs, so a measured rate can replace the configured one later.
The eager path and the estimate share one definition of where an eager
provision starts. SessionVolumes::eager_origin is the choice
provision_role made inline: snapshot a staged image, adopt a kept
checkpoint, stage onto a kept checkpoint of the lineage, stage onto the
staged image of the base, or stage in full. provision_role and the new
eager_plan both use it, and stage_into is gone. eager_plan reads
manifests and volume names only, no data. It fetches the manifests once
more than the provision does. A 26 GiB disk at 16 MiB chunks has a
manifest of about 1700 entries, so this is small.
New flags, each with an env variable:
- --lazy-restore-budget-secs (ENGRAM_LAZY_RESTORE_BUDGET_SECS, 2)
- --lazy-restore-remote-mbps (ENGRAM_LAZY_RESTORE_REMOTE_MBPS, 200)
- --lazy-restore-local-mbps (ENGRAM_LAZY_RESTORE_LOCAL_MBPS, 800)
The defaults come from config::EagerBudget::default(). The flags refuse
a rate of zero or less and a negative budget.
Each restore records the path, the estimate and the provision time:
- the restore.provision_disks span gets the fields root_path,
root_estimate_seconds, swap_path, swap_estimate_seconds and
provision_seconds;
- a "disk restore path" log line gives the byte split;
- metrics: engram_disk_restore_total{path,role},
engram_disk_restore_estimate_seconds{role},
engram_disk_restore_provision_seconds{path}.
When any disk restores lazily, the detached restore task arms the
hydration of the sandbox after the VM restore succeeds. On a failed
restore, the release that already follows takes the cloned drive
down.
The background memory prefetch of a restore with lazy memory now goes
with its sandbox (Decision 6). Its task is held in memory_prefetches;
destroy aborts it, and so do a failed provision and a failed restore.
Tests:
- the estimate with no base, over a base, with local chunks, and at
the budget edge;
- eager_plan names the volume the provision would start from: a staged
image, a kept checkpoint, a kept older version of the lineage, the
base image, or nothing;
- the four mode cases of provision_for_restore on FileDisk and
FileSources (off, force, auto under and over the budget);
- on the Firecracker root lane, session_disk restores with force over
loop-device sources, reads the checkpoint, cuts over and leaves no
source.
Review pass: the restore_path doc said a disk is eager when its
estimate cannot be read, which is true only in auto; it now says so.
The CLI defaults repeated the numbers in EagerBudget::default(); they
now read them from it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
An eager restore writes its disks one after the other, so the budget now applies to the sum of their estimates, not to each disk alone. The new pure function eager_estimate::lazy_disks makes the choice. When the sum is within the budget, every disk is eager. Otherwise disks go lazy, largest estimate first, until the sum of the eager ones is within the budget. A disk with no estimate stays eager and counts as nothing, as before. force now restores a disk eagerly when its eager provision writes nothing (Estimate::writes is false). This is the adopt of a checkpoint kept at exactly that manifest, or a snapshot of a staged volume of that manifest. force restores lazily only when bytes would be written, or when the estimate cannot be read. PooledBackend::restore_paths takes the place of restore_path. It estimates every disk first, then decides for all of them, and records the path and the estimate of each disk as before. Estimate::exceeds is gone; lazy_disks is now the only rule that compares an estimate with the budget. Tests: - lazy_disks with a sum within the budget, the largest disk going lazy first, all disks over the budget, and disks with no estimate; - Estimate::writes for an empty plan and for a base equal to the wanted manifest; - provision_for_restore under force restores an adopted checkpoint eagerly. Review pass: no other findings. The old per-disk rule and its exceeds helper are removed, not kept beside the new rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
Fixes from a review of the lazy restore: - A host that ran with lazy restore on and restarts with it off kept live sources that nothing owned: no recovery, no cutover, no stop, and a hydration task that failed on each tick. The host-agent now always builds its sources (processes over a thin pool, files over sparse files), recovers them and re-arms their hydration. The mode gates only new lazy restores. - A restore picks the lazy path only when a source can start. DiskSources::can_start is new: a ProcessSources without the ublk driver cannot start; FileSources always can. Without the driver, force and auto restore eagerly, and the host-agent logs that at start. It also logs an error if records exist that cannot serve. - Each start, recovery and stop of a source runs in a task of its own. A caller that is dropped (a client deadline, a disarmed hydration) can no longer stop it between the spawn and the record. - A startup sweep logs each engram-disk-source process and each ublk device that no record names, and deletes each such device that no process serves. A device whose server died is deleted only when every record file reads, because a torn record can name a device that a drive reads. - engram-disk-source has a new command, delete. It deletes a device in state DEAD, or, with --quiesced, in state QUIESCED. It refuses a LIVE device and a device whose server process still runs. The kernel rule (drivers/block/ublk_drv.c): a stop of a LIVE device quiesces it and waits for the requests in flight, which with no server never ends. A stop of a QUIESCED device makes each request it holds fail, and it does not wait. A DEAD device has no disk left. START_USER_RECOVERY needs a QUIESCED device, so a device that its server stopped and did not delete can never be recovered. The stop of a source now deletes such a device as its last step, instead of failing on each attempt. - The cutover no longer holds the capture lock across a source recovery or stop. hydration() recovers a dead source before the lock is taken. cut_over now only cuts the drive over and deletes the bitmap, under the lock. The new SessionVolumes::stop_source stops the source after the lock is released, and refuses while the drive copies or a view exists. A crash between the two steps leaves the role Ready, and the next tick finishes it. - A disk whose hydration step fails waits twice as long after each failure, up to 60 s, so a recovery that exits at once does not start a process on each tick. The startup recovery runs four sources at a time. - A hydration task removes its own entry only when the generation that armed it is still the entry. A disarmed task that runs to its end no longer removes a later task's entry. - Fields of SourceRecord that a later build can omit are #[serde(default)]. A test reads a record without them and with a field this build does not know. Not changed: auto mode still reads each manifest twice (once for the estimate, once for the provision). The provision API would need to accept the estimate's manifest; that is a separate change. Tests: the fake-probe unit tests now cover the stop's delete of a device its server left, the delete of a device that cannot be recovered, a refused delete, a dropped start, the sweep, the sweep with a torn record, and records across one release. New tests also cover the backoff, the eager restore on a host that cannot start a source, and stop_source refusals. New root tests: delete refuses a served device and deletes one whose server died (ublk_device), and the sweep deletes a device that no record names (disk_source_ublk). Both run in the ublk lane. Review pass: - The two lazy restore tests in pooled_backend repeated one snapshot metadata block; a helper holds it now. - The process-name checks compared a prefix, so a short name matched; they now compare the 15 bytes that the kernel keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
dm-clone hydration copies every region of the device from the source to the destination and does not detect zeros. The destination is a thin volume: a new one, or a snapshot of a staged base. A lazy restore of a large disk therefore wrote the whole disk into the pool: zero regions became allocated blocks, and regions equal to the base stopped sharing with it. A clone drive now marks those regions copied before the guest starts. dm-clone reads a discard to a region it has not copied as a hint to skip that region: it sets the region in its bitmap and copies nothing. With `no_discard_passdown` the discard does not reach the destination, so the destination keeps its blocks. Rules relied on, from drivers/md/dm-clone-target.c in 6.8 and 6.12: - process_deferred_discards (L1175) sets the covered regions with dm_clone_cond_set_range (L1202 / L1201); complete_discard_bio (L455) completes the bio without remapping it when passdown is off. - bio_region_range (L288) rounds the start up and the end down, so a region covered in part is not marked. - process_discard_bio (L474) and the deferred path do not check DM_CLONE_HYDRATION_ENABLED. With the metadata read-only the discard completes without an error and marks nothing (L509, L1191 / L1190). - The marks commit with the next flush or suspend, as copied regions do (commit_metadata, L1129). - Without passdown the drive sets discard_granularity to the region and a region-aligned max discard (set_discard_limits, L2045 / L2041); blkdev_issue_discard splits at granularity boundaries. Changes: - engram-disk-thin: every clone table carries `no_discard_passdown` (a view too: it is read-only and must never pass a discard to its destination snapshot). `ThinPoolDisk::skip_hydration` issues one BLKDISCARD per joined range on the drive, then a flush. - engram-host-core: `SessionDisk::skip_hydration`, with `disk::skip_ranges` (checks alignment and order, joins adjacent ranges) and `disk::discard` (the BLKDISCARD that `punch_hole` also uses). `FileDisk` copies the whole source at `clone_drive`, so its skip only checks the ranges. - session_volumes: `hydration_skips` computes the skip set from the manifests only. A chunk is skipped when the wanted manifest and the manifest the destination holds (the staged volume's, or none for a new volume, which reads as zeros) list the same hash or neither lists it (`Manifest::delta_from`). Each run of such chunks is rounded inward to whole regions, so a chunk size that is not a multiple of the region (the 4 KiB test chunks) is handled without a special case. `clone_role` makes the drive with hydration off, marks the skips, then enables hydration. Why a checkpoint stays correct: hydration never writes a skipped region, so `written_chunks` (thin_delta against the destination's origin) does not list it, and the upload takes the source manifest's chunk there. By construction that chunk is what the destination holds: zeros on a new volume, the base's equal chunk on a snapshot. If a crash loses the marks, hydration copies the region again with the same bytes; the chunk is then listed and read through the view, which is also correct. The guest cannot discard (Firecracker gives it no discard), so only these host discards ever skip a region. Tests: unit tests for the skip set (new destination, base equal and different, holes in both, a short last chunk, a chunk smaller than a region), for skip_ranges, and for the FileDisk skip. Root lane: a lazy role over a mostly-zero manifest and one over a staged base mark the skipped regions at once, the destination maps only the copied chunks (thin_delta), the pool grows by far less than the disk, and the drive reads the wanted bytes. The staged-base checkpoint test now expects the 40 regions equal to the base to be copied at once. Review pass: the trait doc said "the kernel" on a platform-neutral method, and two comments described a crash re-copy and the checkpoint rule less exactly than the code behaves; all three are reworded. The skip first called `punch_hole`, whose contract is "reads as zeros", which is false on a clone drive; the ioctl moved into `disk::discard`, which states that the device gives the discard its meaning. Range checks and joins live in one helper that both disks use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
… does The cutover no longer stops the source: the host-agent stops it after it releases the capture lock. The root test now calls stop_source before it checks that no source is left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
Profiling a production-shaped guest showed that dm-clone copies every region, with no zero detection, so a lazy restore wrote the whole volume into the pool. The drive now loads with hydration off and no discard passdown, the host discards the regions the destination already holds, then turns hydration on. A lazy restore therefore uses the pool as an eager restore does, and the tradeoff that said otherwise is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
A live move's destination provisions its disks through provision_for_restore. Under `force` (and under `auto` when the estimate passes the budget) that chose a lazy restore. The move then writes the source's blocks after the checkpoint into the destination's live volumes (the post-copy fetch poller, through SessionVolumes::live), and live() refuses a drive that still copies. Every live move to such a host failed. restore_paths now takes `move_dest` and keeps each disk of a move destination on its eager source in every mode, until the lazy teleport target exists. One helper, is_post_copy_dest, gives the fact to restore_with and to provision_for_restore: the staged post-copy entry, or the peer file that the staging writes in the snapshot folder. The other restores may be lazy, because only the guest writes their disks: a resume (also with peer hints, which fill the chunk cache, not a volume), a move by snapshot (an ordinary resume at the destination), a fresh create, a base restore for a session, and a cold-base capture hit. Review pass: the provision_for_restore doc linked a function that does not exist (restore_path); it now links restore_paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
… the disk maps The first checkpoint after a cutover (Shape::CutOver) hashed every chunk of the destination snapshot. On a large disk that is mostly zeros this took minutes. The host-agent holds the capture lock across the publish, so the checkpoint also blocked start_agent for that time. The publish now reads only written_chunks(None, snapshot), the chunks the thin snapshot maps. On FileDisk that is every chunk, so the behaviour there does not change. The new checkpoint_split::upload_only makes every chunk outside the list a hole: it removes those offsets from the previous manifest before upload_over. Thus an unlisted chunk never takes the previous manifest's chunk, and a mapped chunk equal to the previous manifest's chunk is still not uploaded. Why this is the same manifest: after the cutover the drive is linear over the destination, so the snapshot is what the guest reads. A chunk with no mapped block reads as zeros, and the old code made it a hole after it read it. The hydration that came before also agrees. (a) Over a new destination, an unmapped chunk was not hydrated (the discard seeding skipped it because the source manifest has no chunk there) and the guest did not write it, so the source has zeros there. (b) Over a staged base, an unmapped chunk of the snapshot is also not mapped in the base, so the base has zeros there. The seeding skips a region only when the source has the base's bytes there, so the source has zeros too. A chunk that the base maps and the source has as zeros is not skipped. dm-clone writes zeros into it, which maps it, so it is listed. A guest write maps its blocks, also a write of zeros. The guest cannot discard. Tests: a disk_flows unit test (unlisted chunks are holes, also where the volume holds data and the known manifest lists a chunk). The FileDisk session test of a checkpoint after the cutover is unchanged and passes. A root-lane test in tests/clone.rs: a 64 MiB lazy role with four listed chunks hydrates and cuts over, the guest writes one chunk, and the first checkpoint maps five chunks, reads less than an eighth of the disk (from the device's sysfs read counter), and stages into the drive's bytes. Review pass: the Shape::CutOver doc said the upload hashes every chunk; it now says every chunk the snapshot maps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The first two steps of ADR 0127 (Proposed): a session disk that boots before its data is local.
engram-disk-source(new crate): a read-only ublk server over a chunk manifest. It reads from the staged base, then the chunk cache, then the store. It uses user recovery, so a restart of the server does not fail a guest read. Lost data reads as EIO; a transient store failure is retried.engram-disk-thin: dm-clone drives in theSessionDiskseam (clone_drive,enable_hydration,cut_over,snapshot_clone,clone_view,remove_clone). Suspend, operation and resume run as one task that a caller cannot cancel. Opening the pool resumes a device left suspended.Nothing calls these paths yet. Prod behaviour does not change. Step 3 wires the lazy restore into the host-agent.
Test plan
engram-disk-sourceunit tests (14)thin_pool+clone(16),ublk_device(6), no devices leftublk_drv,dm_cloneanddm_delaysession_diskwith a real guest🤖 Generated with Claude Code
https://claude.ai/code/session_01HNhpmyZHg2DXK1MNF4tnR7