os: add experimental reproducible Debian/mkosi backend

[kvinwang]

Summary

• add an experimental Debian trixie/mkosi guest OS backend
• pin Debian snapshot, Linux 7.1.4 source checksum, and EDK2 stable-202502 revision
• reuse the TDX DMA and ACPI BadAML/SystemMemory hardening patches from meta-dstack
• align explicit kernel features with the Yocto guest configuration
• export artifact-manifest v1 and use the common os/image/assemble.sh release path
• produce the Yocto-compatible bare-metal and UKI layouts, dm-verity rootfs, and TDX/GCP/AWS measurement material

Validation

• ./os/mkosi/build.sh lint
• clean Linux 7.1.4 kernel build with both dstack patches
• pinned Intel TDX OVMF build
• complete prod rootfs and release assembly
• artifact schema, checksum, dm-verity, GPT label/GUID and tar layout checks
• generated dstack-0.6.0.tar.gz and dstack-0.6.0-uki.tar.gz
• QEMU UKI smoke test reached verified-root mount, switch_root, and Debian systemd

Notes

This remains experimental. Package contents are Debian rather than Yocto, while the external release contract and file/partition layout are shared with the Yocto backend.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[kvinwang]

Reproducibility/clean-build follow-up pushed as c83948eb3.

Verified with the exact full two-clean-build gate:

JOBS=16 ./os/mkosi/build.sh repro-check os/mkosi/build-repro-final
...
Yocto functional parity paths accepted       # work-a
Yocto-compatible release format accepted     # work-a
Yocto functional parity paths accepted       # work-b
Yocto-compatible release format accepted     # work-b
reproducibility check passed
Elapsed: 35:28.42, exit status 0

The two dstack-0.6.0.tar.gz outputs are byte-identical. The fixes cover deterministic kernel/in-tree BTF, NVIDIA and ZFS external modules, nvattest Rust/C++, dstack Rust binaries, NVIDIA container tooling, fixed dm-verity UUID/salt, removal of the traversal-dependent ldconfig cache, stable absolute mkosi paths, and clean-build Sysbox/OVMF path handling.

Additional gates rerun successfully:

• mkosi static acceptance / functional parity
• full output format and dm-verity artifact checks in both clean builds
• prek run --all-files
• exact PR range prek command
• shellcheck
• REUSE is clean for the tracked tree; the only local reuse lint complaint remains the user's unrelated untracked docs/security/advisories/luks2_tamper.py, which was not touched or added.

[kvinwang]

Post-push QEMU smoke test also passed the intended platform-independent portion on the final reproducible disk.raw: Linux 7.1.4-dstack, dm-verity mapping and verified-root mount, var-volatile.mount, volatile bind setup, and dstack-firstboot.service all succeeded. Boot then stopped at the expected bare-QEMU boundary: no dstack-data partition and no /dev/vdb (dstack-prepare reported exactly that). Log: /tmp/mkosi-qemu-repro-final.log.

[kvinwang]

Added the missing SEV-SNP artifact chain in 986e487af (+ shellcheck annotation a158ce33e):

• builds pinned OvmfPkg/AmdSev/AmdSevX64.dsc with the same Yocto embedded-GRUB removal patch
• exports ovmf-sev.fd through artifact-manifest schema v1
• emits metadata bios-sev: ovmf-sev.fd
• generates and checksums measurement.snp.cbor
• makes both files mandatory in mkosi output acceptance

Verification:

• independent work-a/work-b TDX OVMF hashes match byte-for-byte
• independent work-a/work-b SEV OVMF hashes match byte-for-byte
• both complete release tarballs including measurement.snp.cbor match byte-for-byte (90fabd5d...)
• measurement.snp.cbor is 680 bytes, matching the Yocto artifact shape
• output checks, static acceptance, shellcheck and exact-range prek pass

Rootfs size attribution versus /home/kvin/src/dstack-v060-artifacts/image-store/dstack-0.6.0 is complete; detailed numbers follow in the report.

[kvinwang]

Added the explicitly opt-in, development-only mkosi component cache in e9a8ffcf0.

Security/audit boundary:

• only the new dev-image action enables it;
• normal image unconditionally unsets the internal enable flag;
• repro-check also unconditionally disables it, even if a caller injects the flag;
• Debian rootfs, dm-verity, UKI, measurements and release archives are never cached;
• corrupt entries fail their SHA-256 check and rebuild;
• the implementation is one small helper with local tar.zst entries and no remote cache/database.

Cached component outputs: dstack Rust, container stack, Sysbox, nvattest, kernel source/build/stage, NVIDIA, ZFS, and TDX+SNP OVMF.

Measured verification (JOBS=16, prod):

• cold dev-image, all 8 misses: 20:19 (includes populating 2.0 GiB cache and full output/parity validation)
• hot dev-image, all 8 hits: 3:06
• second hot build in a different output directory: 2:52
• hot output tarballs compared byte-for-byte: identical
• inherited-cache-flag probes for both image and repro-check: no cache directory created and no cache access
• unit tests cover hit, input-key invalidation, corrupt archive rejection/rebuild, and disabled bypass
• static acceptance, shellcheck, and exact PR-range prek: passed

Usage:

DSTACK_DEV_CACHE_DIR="$HOME/.cache/dstack/mkosi-dev" \
  ./os/mkosi/build.sh dev-image "$PWD/os/mkosi/build-dev"

[kvinwang]

Follow-up 911b41ad1 makes the development cache key conservative over the complete installed Debian package/version set in addition to compiler/build-tool versions, so host headers and libraries cannot change underneath a cache hit. Revalidated current HEAD: after a transient GitHub OpenSSL submodule timeout, retry reused the seven completed component entries, completed OVMF, and produced a valid image; the subsequent all-eight-hit build completed in 3:02 and its release tarball was byte-identical to the prior build.

[kvinwang]

Refactored the cache implementation in b6dfefd1d in response to the cohesion/readability concern.

Structure now has three explicit layers:

• build.sh: release orchestration only; it passes --dev-cache exclusively for dev-image and otherwise knows nothing about cache internals.
• scripts/build-components.sh: one cohesive place for the eight non-Debian component build/staging/merge boundaries.
• scripts/dev-cache.sh: one cohesive place for fingerprinting, package/tool dependency capture, locking, checksum validation, restore and atomic store.

In particular, build.sh lost roughly 90 lines of fingerprint/cache/component detail and now invokes component assembly once. The old output-count/key plumbing is gone; the cache API is now simply dev_cache_init plus dev_cache_run component base outputs... -- command. A subshell owns each lock, so lock cleanup is structural rather than duplicated.

Revalidated refactored HEAD:

• full cold v2 cache population and complete image/parity/output validation: passed
• all eight component cache hits: passed
• hot build: 3:12
• cold and hot release tarballs from different output trees: byte-identical
• image and repro-check probes: no cache directory or cache access
• corrupt cache/input invalidation/disabled-path unit tests: passed
• exact PR-range prek, shellcheck and static acceptance: passed

[kvinwang]

Rootfs size parity is now pushed through 1fdf33b21, developed and verified in the isolated worktree /home/kvin/src/dstack.worktrees/mkosi-rootfs-size so the other agent's main-worktree changes were never touched.

Measured against the supplied Yocto 0.6.0 artifact:

Yocto rootfs.img.parted.verity: 594,542,592 bytes
mkosi rootfs.img.parted.verity:  589,299,712 bytes
mkosi is smaller by:              5,242,880 bytes (0.88%)

Retained changes:

• strip production module DWARF while preserving BTF, modinfo and loadable ELF data
• remove non-runtime headers/static archives/libtool metadata, docs/man/info/locales, redundant distro OVMF, and package-manager state hidden by volatile /var binds
• remove the redundant Debian ovmf guest package (the pinned TDX/SEV firmware remains in release artifacts)
• regenerate modules.dep and related indexes after cacheable kernel/NVIDIA/ZFS stage trees are merged

Verification evidence:

• exact two-clean-build repro-check: passed in 33:58; tarballs byte-identical (ca8189d...)
• parity and Yocto-compatible output checks passed in both builds
• ovmf-sev.fd and 680-byte measurement.snp.cbor retained
• nf_tables, ZFS, and NVIDIA resolve through the regenerated module indexes
• QEMU: dm-verity root mounted, nftables succeeded, volatile state and firstboot succeeded; it stopped only at the expected missing /dev/vdb boundary
• exact-range prek, shellcheck, static acceptance, and REUSE all pass

[kvinwang]

Final mkosi-native audit and reproducibility verification is complete.

Key changes in the final series:

• moved image-tool compilation (dstack-mr from this repository and pinned nitro-tpm-pcr-compute) into mkosi.build;
• moved artifact generation/assembly/cleanup into native finalize, post-output and clean hooks;
• replaced the ineffective Format=none setup (which skips mkosi's post-build lifecycle) with a native tar carrier that mkosi.postoutput replaces with the Yocto-compatible archive;
• moved flavor policy to mkosi profiles, package/file cleanup to native content settings, and service/directory policy to systemd presets/tmpfiles/skeleton trees;
• minimized the tools tree to the native misc profile plus explicit assembly packages (571 MiB locally instead of 1.6 GiB);
• made production bypass component cache-key calculation completely;
• made linked-worktree development keys use a host-generated source inventory without mounting Git metadata;
• normalized privilege-dependent empty-directory modes and canonicalized squashfs through a sorted, owner/mode-normalized tar stream before mksquashfs.

The last item came from a real cross-host failure: local and tdxlab trees had identical paths and file bytes, but privileged GitHub construction differed in one setgid directory bit and backing-filesystem inode order. The canonical stream now removes both inputs. A mechanical rebuild from the formerly different local/GitHub root trees produced identical squashfs bytes.

Final verified source revision: 2611402ad7cadded2a044777f9542b5ef6579393

bare: 630921781a37210ddc21104390d1b9c9da5d725541fa48b5c0e9f1e1b1a10af1
UKI:  0b40d5095589235c465dea3dd6c294f00bd90f1c1ccb5cf9fd8c8ff475df85f0

Verification:

• two clean, component-cache-disabled local builds: byte-identical;
• clean direct-SSH tdxlab build: byte-identical to local for both archives;
• standard ubuntu-latest, 4-job manual workflow: passed byte-for-byte in 46m16s: https://github.com/Dstack-TEE/dstack/actions/runs/30146117209
• exact PR-range prek, static mkosi acceptance, component cache/merge tests, shellcheck, REUSE and all repository PR checks: passed;
• final branch is pushed and clean.

Custom logic retained is documented in os/mkosi/README.md and is limited to the Yocto image/archive protocol, independently keyed development component outputs, strict component rootfs collision detection, and portable skeleton-mode normalization.