From ac6b24052f8b833444c46cf468bee6b1aff40bdb Mon Sep 17 00:00:00 2001 From: Chris Hodapp Date: Sun, 25 Oct 2026 00:42:29 -0700 Subject: [PATCH] feat: integrate microvm.nix guests microvm.nix runs no module evaluation itself: a guest is a NixOS evaluation with its guest module, and its host module declares microvm.vms. and reads the guest's runner, shares and toplevel. Two integrations bring guests into the tree, both evaluated by caisson's code. caisson-microvm, the native integration, is a configuration declared inside a NixOS host that holds similarly configured guests. Its options are what they share: hypervisor, vcpu, mem, interfaces and shares, typed as upstream types them, and the start options autostart (false), restartIfChanged (true) and runtimeMaxSec. Its default gives the nearest NixOS configurations beneath it caisson/microvm-guest, which imports upstream's guest module and defines its options from the configuration. slots. declares the guests -1 to -N from the registered configuration of the kind, each with its vsock CID and its shares under a directory of the slot, and passes the kind's moduleImports to them. microvm, the upstream-compatible integration, is the guest itself: a NixOS evaluation with upstream's guest module, whose modules set the microvm options as microvm.nix users write them. caisson/microvm-host, a NixOS module, defines microvm.vms. with evaluatedConfig for every guest that belongs to the host, sets the start options and RuntimeMaxSec, refuses upstream's config and flake forms for those names, and turns caisson.microvm-host.operators. into a polkit rule that lets those users start and stop exactly their guests' units. Guests are published as nixosConfigurations. and are no colmena nodes. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 + docs/development/repository-structure.md | 11 +- lib-overlays/caisson-microvm/default.nix | 533 ++++++++++++++++++ lib-overlays/microvm/default.nix | 202 +++++++ modules/caisson-microvm/core | 1 + modules/caisson-microvm/default/default.nix | 27 + modules/generic/core/default.nix | 2 + modules/nixos/microvm-guest/default.nix | 51 ++ modules/nixos/microvm-host/default.nix | 212 +++++++ tests/dependencies/flake.lock | 38 ++ tests/dependencies/flake.nix | 2 + .../configs/nixos/microvm-reset/default.nix | 10 + .../configs/nixos/microvm-slot/default.nix | 12 + tests/pinned-world/default.nix | 376 +++++++++++- tests/unit/lib-overlays.nix | 5 + 15 files changed, 1478 insertions(+), 6 deletions(-) create mode 100644 lib-overlays/caisson-microvm/default.nix create mode 100644 lib-overlays/microvm/default.nix create mode 120000 modules/caisson-microvm/core create mode 100644 modules/caisson-microvm/default/default.nix create mode 100644 modules/nixos/microvm-guest/default.nix create mode 100644 modules/nixos/microvm-host/default.nix create mode 100644 tests/pinned-world/configs/nixos/microvm-reset/default.nix create mode 100644 tests/pinned-world/configs/nixos/microvm-slot/default.nix diff --git a/README.md b/README.md index 433f4f5e..89a8c95c 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,8 @@ source by. | `disko` | `disko` | a standalone disk layout, written as disko reads it and published as the `diskoConfigurations.` that disko's command formats and mounts; a machine that carries its layout, as `disko-install` and disko's image builder expect, is a NixOS configuration that selects caisson's `disko` NixOS module | | `caisson-facter` | none | the hardware of a machine, a nixos-facter report checked against the report versions caisson knows and extended with a project's options, with the NixOS configurations that run on it declared inside it; a machine gets the report through nixpkgs' `hardware.facter` | | `facter` | none | a `facter.json` as nixos-facter writes it, with the NixOS configurations that run on it declared inside it; a machine gets it as `hardware.facter.reportPath` | +| `caisson-microvm` | `microvm` | microvm.nix guests that share their hypervisor, CPUs, memory, interfaces and shares, declared inside the NixOS host that runs them, each guest a NixOS configuration declared inside it, with pools of slots that differ in their vsock CID and shares; the host selects caisson's `microvm-host` NixOS module to run them; the microvm integration we recommend for new configurations | +| `microvm` | `microvm` | a microvm.nix guest written as microvm.nix reads it, a NixOS configuration that sets its `microvm.*` options itself, declared inside the NixOS host that runs it; for porting an existing guest | | `treefmt` | `treefmt-nix` | the formatters of a tree, as treefmt-nix modules; the configuration named `default` is published as the `formatter.` that `nix fmt` runs | | `nvf` | `nvf` | a Neovim written as nvf modules, published as `nvfConfigurations..`, in the shape `neovimConfiguration` returns, kept in a separate Nix profile that the machine, home-manager configuration or hjem configuration it is declared inside activates | | `nixvim` | `nixvim` | a Neovim written as nixvim modules, published as `nixvimConfigurations..`, kept in a profile as an nvf configuration is | diff --git a/docs/development/repository-structure.md b/docs/development/repository-structure.md index 07e439ee..551724d7 100644 --- a/docs/development/repository-structure.md +++ b/docs/development/repository-structure.md @@ -27,14 +27,15 @@ without nixpkgs can depend on it directly. The layer users reach for: the integrations (`structural`, `flake-parts`, `nixpkgs`, `nixos`, `nixos-minimal`, `home-manager`, `home-manager-minimal`, `darwin`, `hjem`, `caisson-colmena`, `colmena`, -`caisson-disko`, `disko`, `caisson-facter`, `facter`, `terranix`, -`system-manager`, `treefmt`, `nvf`, `nixvim`, `topology`, `kubenix`, -`nixidy`), each +`caisson-disko`, `disko`, `caisson-facter`, `facter`, +`caisson-microvm`, `microvm`, `terranix`, `system-manager`, `treefmt`, +`nvf`, `nixvim`, `topology`, `kubenix`, `nixidy`), each a library overlay contributing its `lib.caisson` namespace, registering its module class where it has a class (`nixos-minimal` and `home-manager-minimal` each evaluate a class another integration -owns and carry constructors only), and taking its ecosystem as an `ecosystemSrc` argument +owns and carry constructors only, and `microvm` evaluates the `nixos` +class as configurations of its own type), and taking its ecosystem as an `ecosystemSrc` argument resolved from the declarations of the mkLib call that composes it (including flake-parts: the integration calls the consumer's flake-parts source with the composed library as its `nixpkgs-lib`, and carries the @@ -52,7 +53,7 @@ inject the same names beside `self`. ## The pinned world The upstream world caisson is tested against (nixpkgs, home-manager, -colmena, disko, nix-darwin, hjem, terranix, system-manager, and the test +colmena, disko, nix-darwin, hjem, terranix, system-manager, microvm.nix, and the test tooling) is pinned in `tests/dependencies/flake.lock`, a lock only the checks and formatter partitions read, so caisson's `flake.nix` carries no churning pin. The diff --git a/lib-overlays/caisson-microvm/default.nix b/lib-overlays/caisson-microvm/default.nix new file mode 100644 index 00000000..d016d896 --- /dev/null +++ b/lib-overlays/caisson-microvm/default.nix @@ -0,0 +1,533 @@ +# SPDX-License-Identifier: MIT +# +# The caisson-microvm integration, the native microvm.nix integration: +# it owns the `caisson-microvm` class. A caisson-microvm configuration +# is a set of similarly configured guests that a NixOS host runs as +# microvm.nix virtual machines. It is declared inside the host as +# `caisson.caisson-microvm.configurations.`, and it declares its +# guests as NixOS configurations, under `caisson.nixos.configurations`. +# +# The evaluation is caisson's own `evalModules` over the options the +# guests share, the framework modules, the selection and the +# configuration's module. The shared options are those of microvm.nix +# that every guest has the same: `hypervisor`, `vcpu`, `mem`, +# `interfaces` and `shares`, typed as microvm.nix types them. The +# start options, `autostart`, `restartIfChanged` and `runtimeMaxSec`, +# say how the host starts and stops the guests. `slots.` +# declares a pool of guests of one kind, `-1` to `-`, +# each given the NixOS configuration registered as +# `lib.caisson.configs.nixos.` and its own vsock CID and shares. +# +# The default module of the class adds caisson's `microvm-guest` NixOS +# module to the nearest NixOS configurations declared inside it, the +# guests. That module imports microvm.nix's guest module and defines +# the guest's microvm options from this configuration. caisson's +# `microvm-host` NixOS module is what a host selects to run the guests: +# it defines microvm.nix's `microvm.vms.` from them. +# +# A caisson-microvm configuration is not published itself. Its guests +# are published as `nixosConfigurations.`. +{ + mkLibOverlay, + ... +}: +{ + + imports = [ + # The library of nixpkgs, which this overlay calls through the + # composed library, imported by key so it is composed wherever + # this is. + ((mkLibOverlay ../nixpkgs-lib) // { key = "nixpkgs-lib"; }) + # What an integration is written from, imported by key so it is + # composed wherever this integration is. + ((mkLibOverlay ../integrations) // { key = "integrations"; }) + ]; + + overlay = + final: prev: + let + + resolveEcosystemSrc = final.caisson.integrations.resolveEcosystemSrc ecosystem; + ecosystem = { + name = "microvm"; + context = "caisson.caisson-microvm"; + }; + + # A microvm.nix source: a microvm.nix flake or source tree, either + # of which holds the guest module at `nixos-modules/microvm` and + # the host module at `nixos-modules/host`. + assertMicrovmEcosystemSrc = + context: ecosystemSrc: + let + path = if builtins.isAttrs ecosystemSrc then ecosystemSrc.outPath or null else ecosystemSrc; + in + if + path != null + && builtins.pathExists "${path}/nixos-modules/microvm/default.nix" + && builtins.pathExists "${path}/nixos-modules/host/default.nix" + then + ecosystemSrc + else + throw "${context} requires `ecosystemSrc` to be a microvm.nix flake or source tree (holding nixos-modules/microvm and nixos-modules/host)."; + + # The microvm.nix source of a configuration: `explicit` when + # given, else what the composition declares or pins under the + # name `microvm`, as recorded on `manifest`. + ecosystemSrcOf = + { explicit, manifest }: + assertMicrovmEcosystemSrc "lib.caisson.caisson-microvm" (resolveEcosystemSrc { + inherit explicit manifest; + }); + + # Where that microvm.nix source came from, from the same + # arguments, as a manifest records it under `ecosystems.microvm`. + ecosystemAccountOf = + { explicit, manifest }: + final.caisson.integrations.ecosystemAccountOf ecosystem { inherit explicit manifest; }; + + # The guest options microvm.nix declares, from its options module + # alone, over `lib`. Only the declarations are read, for their + # types and defaults, so nothing of a NixOS evaluation is needed. + # The options module also defines options of NixOS, such as + # `assertions`, which this evaluation does not declare, so the + # definitions are not checked against the declarations. + upstreamOptionsOf = + { lib, src }: + (lib.evalModules { + modules = [ + "${src}/nixos-modules/microvm/options.nix" + { _module.check = false; } + ]; + }).options.microvm; + + # The options of the class: what the guests share, how the host + # starts them, and the pools of slots. + sharedOptions = + upstream: + { lib, ... }: + { + options = { + hypervisor = lib.mkOption { + inherit (upstream.hypervisor) type default; + description = '' + The hypervisor every guest of this configuration runs on, as + microvm.nix's `microvm.hypervisor` takes it. Each guest gets it + at `mkDefault` priority, so a guest's own definition replaces it. + ''; + }; + vcpu = lib.mkOption { + inherit (upstream.vcpu) type default; + description = '' + The number of virtual CPUs of every guest, as microvm.nix's + `microvm.vcpu` takes it. Each guest gets it at `mkDefault` + priority. + ''; + }; + mem = lib.mkOption { + inherit (upstream.mem) type default; + description = '' + The memory of every guest in megabytes, as microvm.nix's + `microvm.mem` takes it. Each guest gets it at `mkDefault` + priority. + ''; + }; + interfaces = lib.mkOption { + inherit (upstream.interfaces) type; + default = [ ]; + description = '' + The network interfaces every guest has, as microvm.nix's + `microvm.interfaces` takes them. Each guest gets these + definitions beside its own. + ''; + }; + shares = lib.mkOption { + inherit (upstream.shares) type; + default = [ ]; + description = '' + The directories every guest shares with the host, as + microvm.nix's `microvm.shares` takes them. Each guest gets + these definitions beside its own and the shares of its slot. + ''; + }; + autostart = lib.mkOption { + type = lib.types.bool; + default = false; + description = '' + Whether the host starts the guests when it boots. When false, + a guest runs only after its unit `microvm@.service` is + started. + ''; + }; + restartIfChanged = lib.mkOption { + type = lib.types.bool; + default = true; + description = '' + Whether switching the host to a new generation restarts a + running guest whose configuration changed. + ''; + }; + runtimeMaxSec = lib.mkOption { + type = lib.types.nullOr lib.types.ints.positive; + default = null; + description = '' + The number of seconds a guest may run before systemd stops it, + set as `RuntimeMaxSec` of its unit `microvm@.service`. + When null, a guest runs until it is stopped. + ''; + }; + slots = lib.mkOption { + type = lib.types.attrsOf (lib.types.submodule slotOptions); + default = { }; + example = lib.literalExpression '' + { + rea = { + count = 2; + cidBase = 100; + shareRoot = "/var/lib/rea"; + vsockPort = 5000; + }; + } + ''; + description = '' + Pools of guests of one kind, by kind. A kind with `count` N + declares the NixOS configurations `-1` to `-N` + inside this configuration. Each of them takes the NixOS + configuration registered as `lib.caisson.configs.nixos.` + as its module, and gets the vsock CID and the shares of its + slot. + ''; + }; + }; + }; + + # The options of a pool of slots. + slotOptions = + { lib, ... }: + { + options = { + count = lib.mkOption { + type = lib.types.ints.unsigned; + description = "The number of guests of the kind."; + }; + cidBase = lib.mkOption { + type = lib.types.ints.between 2 4294967294; + description = '' + The number the vsock CIDs of the guests of the kind count from. + The guest `-` gets `cidBase + k` as `microvm.vsock.cid`. + The CIDs 0, 1 and 2 are reserved, so the base is 2 or more. + ''; + }; + shareRoot = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "/var/lib/rea"; + description = '' + The directory on the host under which each guest of the kind has + a directory of its own. When set, the guest `-` shares + `/-/in`, read-only, at `/work/in`, and + `/-/out`, writable, at `/work/out`, both + over virtiofs. When null, the guests of the kind get no shares + of their slot. + ''; + }; + vsockPort = lib.mkOption { + type = lib.types.nullOr lib.types.port; + default = null; + description = '' + The vsock port a guest of the kind serves on, which the programs + that control the slots read to wait for a started guest. The + guests do not read it. + ''; + }; + moduleImports = lib.mkOption { + type = lib.types.nullOr (lib.types.functionTo (lib.types.listOf lib.types.raw)); + default = null; + example = lib.literalExpression '' + modules: [ modules."caisson/microvm-guest" ] + ''; + description = '' + The selection over the nixos class of the registry that each + guest of the kind is constructed with, as the `moduleImports` + argument of `lib.caisson.nixos.mkConfiguration` takes it. It + replaces the default of the class, so a guest of the kind gets + neither the registry entries named `default` nor what the + levels above added, and then it lists caisson's `microvm-guest` + NixOS module itself. When null, each guest gets the default of + the class. + ''; + }; + extraModuleImports = lib.mkOption { + type = lib.types.nullOr (lib.types.functionTo (lib.types.listOf lib.types.raw)); + default = null; + description = '' + A selection over the nixos class of the registry added to the + selection of each guest of the kind, as the `extraModuleImports` + argument of `lib.caisson.nixos.mkConfiguration` takes it. When + null, nothing is added. + ''; + }; + }; + }; + + # The slots of an evaluated caisson-microvm configuration, by the + # name of the guest: its kind, its number, its vsock CID, the + # shares of its slot and the port its kind serves on. + slotsOf = + slots: + builtins.listToAttrs ( + builtins.concatMap ( + kind: + let + pool = slots.${kind}; + in + builtins.map ( + index: + let + name = "${kind}-${toString index}"; + directory = "${pool.shareRoot}/${name}"; + in + { + inherit name; + value = { + inherit kind index; + inherit (pool) vsockPort; + cid = pool.cidBase + index; + shares = + if pool.shareRoot == null then + [ ] + else + [ + { + tag = "work-in"; + source = "${directory}/in"; + mountPoint = "/work/in"; + proto = "virtiofs"; + readOnly = true; + } + { + tag = "work-out"; + source = "${directory}/out"; + mountPoint = "/work/out"; + proto = "virtiofs"; + } + ]; + }; + } + ) (final.range 1 pool.count) + ) (builtins.attrNames slots) + ); + + # The NixOS configurations the slots declare: for each slot, a + # NixOS configuration named after it, with the configuration + # registered under the name of its kind as its module and the + # selections of its kind. + slotsModule = + { config, lib, ... }: + { + caisson.nixos.configurations = builtins.mapAttrs ( + _name: slot: + let + pool = config.slots.${slot.kind}; + in + lib.caisson.nixos.mkConfiguration { + inherit (pool) moduleImports extraModuleImports; + configModule = + lib.caisson.configs.nixos.${slot.kind} or (throw '' + lib.caisson.caisson-microvm: the slots of the kind `${slot.kind}` take the + NixOS configuration registered as `lib.caisson.configs.nixos.${slot.kind}`, + and the composition registers none under that name. + ''); + } + ) (slotsOf config.slots); + }; + + # The values of the definitions of an option, as the module + # system kept them, for a guest to define again in its own + # evaluation, where the defaults of microvm.nix's submodules are + # computed against the guest. + definitionsOf = option: builtins.map (definition: definition.value) option.definitionsWithLocations; + + # The evaluator's call, on the lib of the view being evaluated: + # the options of the class, the module of the slots, the framework + # modules of the class, the selection and the configuration's + # module, with the composition's pinned sources as the `inputs` + # special argument. `ecosystemArgs`, which only the twin's pattern + # admits, is merged over the call last. + evaluate = + args: + { lib, manifest }: + let + selection = lib.caisson.integrations; + src = ecosystemSrcOf { + explicit = args.ecosystemSrc or null; + inherit manifest; + }; + upstream = upstreamOptionsOf { inherit lib src; }; + registry = lib.caisson.modules.caisson-microvm or { }; + frameworkModules = selection.frameworkModules "caisson-microvm" registry; + moduleImports = selection.moduleImportsOf "caisson-microvm" { inherit lib manifest; } args; + + # The configuration's module: the module passed, else the + # configuration registered under the configuration's name + # (`configs/caisson-microvm/`), else none. + name = manifest.name or null; + configModule = selection.configModuleOf { + inherit lib name; + class = "caisson-microvm"; + configModule = args.configModule or null; + }; + + moduleList = [ + (selection.caissonModule { + key = "caisson.caisson-microvm.sharedOptions"; + file = "caisson-caisson-microvm:sharedOptions"; + module = sharedOptions upstream; + }) + (selection.caissonModule { + key = "caisson.caisson-microvm.slots"; + file = "caisson-caisson-microvm:slots"; + module = slotsModule; + }) + ] + ++ frameworkModules + ++ moduleImports registry + ++ (if configModule == null then [ ] else [ configModule ]); + + evaluated = lib.evalModules ( + { + class = "caisson-microvm"; + specialArgs = { + inherit lib; + inputs = manifest.sources; + } + // (if (args.pkgSets or null) != null then { inherit (args) pkgSets; } else { }) + // (if (args.specialArgs or null) != null then args.specialArgs else { }); + modules = moduleList; + } + // (if (args.ecosystemArgs or null) != null then args.ecosystemArgs else { }) + ); + inherit (evaluated) config options; + in + { + value = evaluated; + # Where microvm.nix came from, as `src` was resolved. + ecosystems.microvm = ecosystemAccountOf { + explicit = args.ecosystemSrc or null; + inherit manifest; + }; + # What evalModules received: the options of the class, the + # module of the slots, the framework module of the class, the + # modules selected from the registry and the configuration's + # module. microvm.nix's options module gives the types of the + # shared options; no module list of microvm.nix is evaluated. + modules = moduleList; + # What the guests and the host read. `ecosystemSrc` is the + # microvm.nix source the guests import the guest module from. + # `guests` is what every guest gets: the hypervisor, the + # virtual CPUs and the memory, and the definitions of the + # interfaces and the shares. `start` is how the host starts and + # stops the guests. `slots` holds the slot of each guest the + # slots declare, by the name of the guest. + outputs = { + ecosystemSrc = src; + guests = { + inherit (config) hypervisor vcpu mem; + interfaces = definitionsOf options.interfaces; + shares = definitionsOf options.shares; + }; + start = { + inherit (config) autostart restartIfChanged runtimeMaxSec; + }; + slots = slotsOf config.slots; + }; + }; + + configuration = + args: + final.caisson.integrations.mkModuleConfiguration { + selection = { + class = "caisson-microvm"; + inherit args; + }; + type = "caisson-microvm"; + defaultPkgs = args.defaultPkgs or null; + evaluate = evaluate args; + }; + + integration = final.caisson.integrations.mkIntegration { + name = "caisson-microvm"; + class = "caisson-microvm"; + # What these return is a configuration, a function of + # `{ name, parent }`: the NixOS host that declares it under + # `caisson.caisson-microvm.configurations.` finalizes it. + mkConfiguration = + { + # The caisson-microvm configuration's module: the options + # the guests share, the slots, and the guests under + # `caisson.nixos.configurations`. When absent, the + # configuration registered under the configuration's name + # (`lib.caisson.configs.caisson-microvm.`), if any. + # Further modules of the class are selected with + # `moduleImports`, from the registry. + configModule ? null, + # The package sets, handed to the modules as the `pkgSets` + # special argument. + pkgSets ? null, + # The package set the guests run on: a function that + # receives the package sets available to each, by package + # config name, and returns the set to run on. + defaultPkgs ? null, + # The microvm.nix flake or source tree; resolved from the + # composition's declarations when absent. + ecosystemSrc ? null, + # The selection over the caisson-microvm class of the + # registry. It replaces the default of the class, which is + # every entry named `default` followed by what the + # configurations above added. + moduleImports ? null, + # A selection added to that selection, whichever it is. + extraModuleImports ? null, + # Extra module arguments, merged over those the framework + # supplies. + specialArgs ? null, + }@args: + configuration args; + # The same arguments and `ecosystemArgs`, the evaluator's + # arguments merged over the composed call last. + mkConfigurationWithEcosystemArgs = + { + configModule ? null, + pkgSets ? null, + defaultPkgs ? null, + ecosystemSrc ? null, + moduleImports ? null, + extraModuleImports ? null, + specialArgs ? null, + ecosystemArgs ? null, + }@args: + configuration args; + # What the upstream-compatible microvm integration and the + # host module build on, so all of them read microvm.nix the + # same way. + extra = { + # The microvm.nix source of a configuration: `explicit` when + # given, else what the composition declares or pins under the + # name `microvm`, as recorded on `manifest`; refused when it + # is no microvm.nix source. + inherit ecosystemSrcOf; + # Where that microvm.nix source came from, from the same + # arguments. + inherit ecosystemAccountOf; + # The check that a value is a microvm.nix source, with the + # name of the caller for the message. + inherit assertMicrovmEcosystemSrc; + }; + }; + in + { + caisson = (prev.caisson or { }) // { + caisson-microvm = integration.namespace; + }; + }; + +} diff --git a/lib-overlays/microvm/default.nix b/lib-overlays/microvm/default.nix new file mode 100644 index 00000000..14133535 --- /dev/null +++ b/lib-overlays/microvm/default.nix @@ -0,0 +1,202 @@ +# SPDX-License-Identifier: MIT +# +# The microvm integration, the upstream-compatible one. A microvm +# configuration is a microvm.nix guest written as microvm.nix's users +# write one: a NixOS configuration that imports microvm.nix's guest +# module and sets its `microvm.*` options itself. It is declared inside +# the NixOS host that runs it, as +# `caisson.microvm.configurations.`, and caisson's `microvm-host` +# NixOS module of the host defines `microvm.vms.` from it. The +# native integration, caisson-microvm, takes caisson's form instead: a +# set of guests that share their microvm options and slots. +# +# The evaluation is the NixOS evaluation of the nixos integration: +# eval-config over the composition the nixos integration publishes +# (`lib.caisson.nixos.compose`), with microvm.nix's guest module after +# the modules of that composition. So a microvm configuration gets what +# any caisson NixOS configuration gets, the framework modules of the +# nixos class and its default selection, and its modules are modules +# of the nixos class. +# +# A microvm configuration is published as `nixosConfigurations.`, +# as a NixOS configuration is. +{ + mkLibOverlay, + ... +}: +{ + + imports = [ + # The library of nixpkgs, which this overlay calls through the + # composed library, imported by key so it is composed wherever + # this is. + ((mkLibOverlay ../nixpkgs-lib) // { key = "nixpkgs-lib"; }) + # What an integration is written from, imported by key so it is + # composed wherever this integration is. + ((mkLibOverlay ../integrations) // { key = "integrations"; }) + # The nixos integration, whose composition and publishing this one + # builds on. + ((mkLibOverlay ../nixos) // { key = "nixos"; }) + # The native microvm integration, whose resolution of the + # microvm.nix source this one builds on. + ((mkLibOverlay ../caisson-microvm) // { key = "caisson-microvm"; }) + ]; + + overlay = + final: prev: + let + nixos = final.caisson.nixos; + native = final.caisson.caisson-microvm; + + # The evaluator's call, on the view being evaluated: eval-config + # over the composition of the nixos class, with microvm.nix's + # guest module, for the configuration's system. `ecosystemArgs`, + # which only the twin's pattern admits, is merged over the call + # last. + evaluate = + args: view: + let + common = nixos.compose { context = "lib.caisson.microvm.mkConfiguration"; } view args; + microvmSrc = native.ecosystemSrcOf { + explicit = args.microvmSrc or null; + manifest = view.lib.caisson-core.libManifest or { }; + }; + guestModule = "${microvmSrc}/nixos-modules/microvm"; + callArgs = { + modules = common.modules ++ [ guestModule ]; + inherit (common) specialArgs lib system; + } + // (if (args.ecosystemArgs or null) != null then args.ecosystemArgs else { }); + value = import "${common.src}/nixos/lib/eval-config.nix" callArgs; + in + { + inherit value; + # Where the module collection and microvm.nix came from. + ecosystems = { + nixos = common.srcAccount; + microvm = native.ecosystemAccountOf { + explicit = args.microvmSrc or null; + manifest = view.lib.caisson-core.libManifest or { }; + }; + }; + # What eval-config received: NixOS' module list, the modules + # of the composition, then microvm.nix's guest module. + modules = [ + (final.caisson.integrations.ecosystemUnit { + key = "nixos"; + project = "nixpkgs"; + file = "nixos/modules/module-list.nix"; + }) + ] + ++ common.modules + ++ [ + (final.caisson.integrations.ecosystemUnit { + key = "microvm"; + project = "microvm"; + file = "nixos-modules/microvm"; + }) + ]; + # What `nixos-rebuild` reads from a NixOS configuration, and + # the runner microvm.nix builds for the guest, which the host + # installs. + outputs = { + toplevel = value.config.system.build.toplevel; + runner = value.config.microvm.declaredRunner; + }; + }; + + configuration = + args: + final.caisson.integrations.mkModuleConfiguration { + # The modules of a microvm configuration are modules of the + # nixos class, selected as a NixOS configuration selects + # them. + selection = { + class = "nixos"; + inherit args; + }; + type = "microvm"; + # A guest is evaluated per system, as a NixOS configuration + # is: for the system of its host. + perSystem = true; + defaultPkgs = args.defaultPkgs or null; + inherit (nixos) exportsTo; + # A guest is a machine of its own; a configuration with an + # activation declared inside it checks its marker. + activation = final.caisson.integrations.machineActivation; + evaluate = evaluate args; + }; + + integration = final.caisson.integrations.mkIntegration { + name = "microvm"; + class = "nixos"; + # What these return is a configuration, a function of + # `{ name, parent }`: the NixOS host that declares it under + # `caisson.microvm.configurations.` finalizes it. + mkConfiguration = + { + # The guest's module, a module of the nixos class that sets + # the `microvm.*` options as microvm.nix's guest module + # declares them. When absent, the configuration registered + # under the configuration's name + # (`lib.caisson.configs.nixos.`), if any. Further + # modules of the class are selected with `moduleImports`, + # from the registry. + configModule ? null, + # The nixpkgs source tree; resolved from the composition's + # declarations when absent. + ecosystemSrc ? null, + # The microvm.nix flake or source tree, as a function of the + # tree's sources (`microvmSrc = sources: sources.;`); + # resolved from the composition's declarations under the + # name `microvm` when absent. + microvmSrc ? null, + # The package set the guest runs on: a function that + # receives the package sets available where it is + # declared, by package config name, and returns the set to + # run on. When absent, the selection of the nearest + # configuration above. + defaultPkgs ? null, + # The selection over the nixos class of the registry. It + # replaces the default of the class, which is every entry + # named `default` followed by what the configurations + # above added. + moduleImports ? null, + # A selection added to that selection, whichever it is. + extraModuleImports ? null, + # Extra module arguments, merged over those the framework + # supplies. + specialArgs ? null, + }@args: + configuration args; + # The same arguments and `ecosystemArgs`, the evaluator's + # arguments merged over the composed call last. + mkConfigurationWithEcosystemArgs = + { + configModule ? null, + ecosystemSrc ? null, + microvmSrc ? null, + defaultPkgs ? null, + moduleImports ? null, + extraModuleImports ? null, + specialArgs ? null, + ecosystemArgs ? null, + }@args: + configuration args; + }; + in + { + # The modules of a microvm configuration are registered with + # `lib.caisson.nixos.mkModule`, the registration form of the + # nixos class, and every configuration registered in that class + # is a NixOS configuration, so the namespace carries neither + # `mkModule` nor `mkConfigurations`. + caisson = (prev.caisson or { }) // { + microvm = builtins.removeAttrs integration.namespace [ + "mkModule" + "mkConfigurations" + ]; + }; + }; + +} diff --git a/modules/caisson-microvm/core b/modules/caisson-microvm/core new file mode 120000 index 00000000..3715354a --- /dev/null +++ b/modules/caisson-microvm/core @@ -0,0 +1 @@ +../generic/core \ No newline at end of file diff --git a/modules/caisson-microvm/default/default.nix b/modules/caisson-microvm/default/default.nix new file mode 100644 index 00000000..ba5f2358 --- /dev/null +++ b/modules/caisson-microvm/default/default.nix @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: MIT +# +# The default module of the caisson-microvm class. +# +# The nearest NixOS configurations beneath a caisson-microvm +# configuration are its guests. This module adds caisson's NixOS module +# `microvm-guest`, which imports microvm.nix's guest module and defines +# the guest's microvm options from the caisson-microvm configuration, +# to the modules that such a NixOS configuration imports by default. A +# NixOS configuration declared inside a guest is no guest and does not +# get it. A guest that passes `moduleImports` selects its modules +# itself, and then it lists `microvm-guest` itself, as +# `caisson/microvm-guest` in a flake that lists caisson in `projects`. +# +# `closure-lib` is the library of caisson, where `microvm-guest` is +# registered under that name. +{ closure-lib, ... }: +{ ... }: +{ + # The options under `caisson` are declared by the core module of the + # class. + imports = [ closure-lib.caisson.modules.caisson-microvm.core ]; + + caisson.forNearestChildren.extraDefaultModuleImports.nixos = _lib: [ + closure-lib.caisson.modules.nixos.microvm-guest + ]; +} diff --git a/modules/generic/core/default.nix b/modules/generic/core/default.nix index 2cf94b15..71c61c43 100644 --- a/modules/generic/core/default.nix +++ b/modules/generic/core/default.nix @@ -32,6 +32,7 @@ "caisson-colmena" "caisson-disko" "caisson-facter" + "caisson-microvm" "colmena" "darwin" "disko" @@ -40,6 +41,7 @@ "hjem" "home-manager" "kubenix" + "microvm" "nixidy" "nixos" "nixpkgs" diff --git a/modules/nixos/microvm-guest/default.nix b/modules/nixos/microvm-guest/default.nix new file mode 100644 index 00000000..937f665e --- /dev/null +++ b/modules/nixos/microvm-guest/default.nix @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: MIT +# +# A NixOS module for a guest of a caisson-microvm configuration: a +# NixOS configuration declared inside it. It imports microvm.nix's +# guest module, from the microvm.nix source the caisson-microvm +# configuration resolved, and defines the guest's microvm options from +# that configuration, read through +# `lib.caisson-core.evalManifest.nearest.caisson-microvm`: +# +# - `microvm.hypervisor`, `microvm.vcpu` and `microvm.mem` at +# `mkDefault` priority, so the guest's own definition replaces them; +# - the definitions of `microvm.interfaces` and `microvm.shares` that +# the caisson-microvm configuration holds, beside the guest's own; +# - for a guest that one of its slots declares, `microvm.vsock.cid` and +# the shares of the slot. +# +# The default module of the caisson-microvm class adds this module to +# the nearest NixOS configurations beneath a caisson-microvm +# configuration. A guest that passes `moduleImports` selects its +# modules itself, and then it lists this module itself. +{ ... }: +{ lib, ... }: +let + manifest = lib.caisson-core.evalManifest; + + machines = + manifest.nearest.caisson-microvm or (throw '' + caisson's `microvm-guest` NixOS module gives a guest the options of the + caisson-microvm configuration it is declared inside, and this NixOS + configuration is not declared inside a caisson-microvm configuration. + ''); + + inherit (machines.outputs) guests; + + # The slot of this guest, null for a guest no slot declares. + slot = machines.outputs.slots.${manifest.name or ""} or null; +in +{ + _file = "caisson:microvm-guest"; + + imports = [ "${machines.outputs.ecosystemSrc}/nixos-modules/microvm" ]; + + microvm = { + hypervisor = lib.mkDefault guests.hypervisor; + vcpu = lib.mkDefault guests.vcpu; + mem = lib.mkDefault guests.mem; + interfaces = lib.mkMerge guests.interfaces; + shares = lib.mkMerge (guests.shares ++ (if slot == null then [ ] else [ slot.shares ])); + } + // (if slot == null then { } else { vsock.cid = slot.cid; }); +} diff --git a/modules/nixos/microvm-host/default.nix b/modules/nixos/microvm-host/default.nix new file mode 100644 index 00000000..e3f09288 --- /dev/null +++ b/modules/nixos/microvm-host/default.nix @@ -0,0 +1,212 @@ +# SPDX-License-Identifier: MIT +# +# A NixOS module for a host that runs microvm.nix guests declared +# inside it. A host selects it by name, as `microvm-host`, or as +# `caisson/microvm-host` in a flake that lists caisson in `projects`. +# +# It imports microvm.nix's host module and defines +# `microvm.vms.` for every guest that belongs to the host. The +# guests are the NixOS configurations declared inside the host's +# caisson-microvm configurations and the host's microvm +# configurations. A guest belongs to the host when no other NixOS +# configuration or microvm configuration sits between the two +# (`lib.caisson.integrations.belongsHere`), so a guest declared inside +# a guest belongs to that guest. Each guest is read from the +# configurations the host passes up, so the host's +# `caisson..exported` selectors decide which guests it +# runs. A guest is named in `microvm.vms` after the name it is +# declared under. +# +# For each guest, `evaluatedConfig` is caisson's evaluation of the +# guest, the configuration published as `nixosConfigurations.`. +# A guest of a caisson-microvm configuration gets `autostart` and +# `restartIfChanged` from that configuration, and its unit +# `microvm@.service` gets `RuntimeMaxSec` from its +# `runtimeMaxSec` when that is set. A microvm configuration keeps the +# default `autostart` of microvm.nix and gets `restartIfChanged` +# true. These are at `mkDefault` priority, so a host module that sets +# them replaces them. The host refuses microvm.nix's `config` and +# `flake` forms for these names, since each evaluates the guest a +# second way. +# +# `caisson.microvm-host.operators` names the users that may start, stop +# and restart which guests, through a polkit rule. +# +# The guests are read in the evaluation of the host that holds them. +# In the evaluation without the configurations declared inside it, +# which the guests read, the host has no guests. +{ ... }: +{ config, lib, ... }: +let + manifest = lib.caisson-core.evalManifest; + integrations = lib.caisson.integrations; + + src = + let + found = lib.caisson-core.ecosystemSrc "microvm"; + in + if found == null then + throw '' + caisson's `microvm-host` NixOS module needs the microvm.nix source. + Declare `defaultEcosystemSrc.microvm` in the mkLib call, or pin a + source named `microvm` in the `sources` passed to mkLib. + '' + else + lib.caisson.caisson-microvm.assertMicrovmEcosystemSrc "caisson's `microvm-host` NixOS module" found; + + # Whether a configuration passed up through the host is a guest of + # the host, evaluated for the system of the host. + isGuest = + entry: + let + type = entry.manifest.type or null; + in + entry.attrset == "nixosConfigurations" + && (entry.system or null) == (manifest.system or null) + && integrations.belongsHere "nixos" entry + && integrations.belongsHere "microvm" entry + && ( + type == "microvm" + || ( + type == "nixos" && builtins.any (segment: segment.type == "caisson-microvm") (lib.init entry.path) + ) + ); + + # How the host starts and stops a guest. + startOf = + entry: + if entry.manifest.type == "microvm" then + { + autostart = null; + restartIfChanged = true; + runtimeMaxSec = null; + } + else + entry.manifest.nearest.caisson-microvm.outputs.start; + + entries = + if manifest == null || manifest.childless then + [ ] + else + builtins.filter isGuest config.caisson.exports.configurations; + + nameOf = entry: (lib.last entry.namePath).name; + + names = builtins.map nameOf entries; + + repeated = builtins.filter ( + name: builtins.length (builtins.filter (other: other == name) names) > 1 + ) (lib.unique names); + + # The guests of the host, by name. + guests = + if repeated != [ ] then + throw '' + caisson's `microvm-host` NixOS module names each guest after the name it is + declared under, and several guests of this host are declared as + ${lib.concatMapStringsSep ", " (name: "`${name}`") repeated}. + '' + else + builtins.listToAttrs ( + builtins.map (entry: { + name = nameOf entry; + value = { + inherit (entry.manifest) value; + start = startOf entry; + }; + }) entries + ); + + operators = config.caisson.microvm-host.operators; + + # The units each operator may start, stop and restart. + unitsByUser = builtins.mapAttrs (_user: builtins.map (guest: "microvm@${guest}.service")) operators; + + unknownOperated = lib.unique ( + builtins.filter (guest: !(guests ? ${guest})) (builtins.concatLists (builtins.attrValues operators)) + ); +in +{ + _file = "caisson:microvm-host"; + + imports = [ "${src}/nixos-modules/host" ]; + + options.caisson.microvm-host.operators = lib.mkOption { + type = lib.types.attrsOf (lib.types.listOf lib.types.str); + default = { }; + example = { + chris = [ + "rea-1" + "rea-2" + ]; + }; + description = '' + The users that may start, stop and restart guests of this host, by + user name, each with the names of the guests that user may control. + A polkit rule lets each user run `systemctl start`, `systemctl stop` + and `systemctl restart` on the unit `microvm@.service` of each + guest listed for that user, and on no other unit. Every name listed is + a guest of this host, and a name that is not is refused. + ''; + }; + + config = { + microvm.vms = builtins.mapAttrs ( + _name: guest: + { + evaluatedConfig = guest.value; + restartIfChanged = lib.mkDefault guest.start.restartIfChanged; + } + // ( + if guest.start.autostart == null then { } else { autostart = lib.mkDefault guest.start.autostart; } + ) + ) guests; + + systemd.services = lib.mapAttrs' (name: guest: { + name = "microvm@${name}"; + value.serviceConfig.RuntimeMaxSec = toString guest.start.runtimeMaxSec; + }) (lib.filterAttrs (_name: guest: guest.start.runtimeMaxSec != null) guests); + + assertions = + builtins.concatMap (name: [ + { + assertion = config.microvm.vms.${name}.config == null && config.microvm.vms.${name}.flake == null; + message = '' + The guest `${name}` of this host is a configuration declared inside it, and + caisson's `microvm-host` NixOS module gives microvm.nix its evaluation as + `microvm.vms.${name}.evaluatedConfig`. Leave `microvm.vms.${name}.config` + and `microvm.vms.${name}.flake` unset. + ''; + } + ]) (builtins.attrNames guests) + ++ lib.optional (manifest != null && !manifest.childless) { + assertion = unknownOperated == [ ]; + message = '' + `caisson.microvm-host.operators` lists ${ + lib.concatMapStringsSep ", " (name: "`${name}`") unknownOperated + }, which no guest of this host is declared as. The guests of this host are + ${ + if guests == { } then + "none" + else + lib.concatMapStringsSep ", " (name: "`${name}`") (builtins.attrNames guests) + }. + ''; + }; + + security.polkit.enable = lib.mkIf (operators != { }) true; + security.polkit.extraConfig = lib.mkIf (operators != { }) '' + // caisson's `microvm-host` NixOS module: the guests each user may start, + // stop and restart. + polkit.addRule(function (action, subject) { + var units = ${builtins.toJSON unitsByUser}; + if (action.id == "org.freedesktop.systemd1.manage-units" && units.hasOwnProperty(subject.user)) { + var verb = action.lookup("verb"); + if ((verb == "start" || verb == "stop" || verb == "restart") && units[subject.user].indexOf(action.lookup("unit")) >= 0) { + return polkit.Result.YES; + } + } + }); + ''; + }; +} diff --git a/tests/dependencies/flake.lock b/tests/dependencies/flake.lock index 3bb980fe..0cd19210 100644 --- a/tests/dependencies/flake.lock +++ b/tests/dependencies/flake.lock @@ -419,6 +419,27 @@ "type": "github" } }, + "microvm": { + "inputs": { + "nixpkgs": [ + "nixpkgs" + ], + "spectrum": "spectrum" + }, + "locked": { + "lastModified": 1792810338, + "narHash": "sha256-0DS+A3BsygKRpKe0qmUWVDSfKqpIHZCpHJagAklE0o4=", + "owner": "microvm-nix", + "repo": "microvm.nix", + "rev": "4876e2ae2de6ce225f4ccf664ae441128aa3f03a", + "type": "github" + }, + "original": { + "owner": "microvm-nix", + "repo": "microvm.nix", + "type": "github" + } + }, "mnw": { "locked": { "lastModified": 1784154424, @@ -738,6 +759,7 @@ "hjem": "hjem", "home-manager": "home-manager", "kubenix": "kubenix", + "microvm": "microvm", "nix-darwin": "nix-darwin", "nix-topology": "nix-topology", "nix-unit": "nix-unit", @@ -752,6 +774,22 @@ "treefmt-nix": "treefmt-nix_3" } }, + "spectrum": { + "flake": false, + "locked": { + "lastModified": 1786032750, + "narHash": "sha256-+Ba/kIn5BjvV7PN8bWOo7RoqgTY/Yv2M7qeMf3wS2Ag=", + "ref": "refs/heads/main", + "rev": "77d2389874bb092b8dbf05ecc5b1030372e80e98", + "revCount": 1417, + "type": "git", + "url": "https://spectrum-os.org/git/spectrum" + }, + "original": { + "type": "git", + "url": "https://spectrum-os.org/git/spectrum" + } + }, "stable": { "locked": { "lastModified": 1783625654, diff --git a/tests/dependencies/flake.nix b/tests/dependencies/flake.nix index 679f284c..7d8af0fb 100644 --- a/tests/dependencies/flake.nix +++ b/tests/dependencies/flake.nix @@ -35,6 +35,8 @@ inputs.hjem.inputs.nixpkgs.follows = "nixpkgs"; inputs.nix-darwin.url = "github:nix-darwin/nix-darwin"; inputs.nix-darwin.inputs.nixpkgs.follows = "nixpkgs"; + inputs.microvm.url = "github:microvm-nix/microvm.nix"; + inputs.microvm.inputs.nixpkgs.follows = "nixpkgs"; # The VMs of other distributions the system-manager VM test boots # (tests/system-manager). inputs.nix-vm-test.url = "github:numtide/nix-vm-test"; diff --git a/tests/pinned-world/configs/nixos/microvm-reset/default.nix b/tests/pinned-world/configs/nixos/microvm-reset/default.nix new file mode 100644 index 00000000..9acac007 --- /dev/null +++ b/tests/pinned-world/configs/nixos/microvm-reset/default.nix @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MIT +# +# A guest kind of the registered world whose slots select their +# modules themselves: the NixOS configuration each slot of the kind +# `microvm-reset` takes. +{ ... }: +{ ... }: +{ + system.stateVersion = "25.05"; +} diff --git a/tests/pinned-world/configs/nixos/microvm-slot/default.nix b/tests/pinned-world/configs/nixos/microvm-slot/default.nix new file mode 100644 index 00000000..41c808cc --- /dev/null +++ b/tests/pinned-world/configs/nixos/microvm-slot/default.nix @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MIT +# +# A guest kind of the registered world: the NixOS configuration each +# slot of the kind `microvm-slot` takes, and the configuration +# `microvm-reset` takes too. microvm.nix's guest module, which caisson's +# `microvm-guest` NixOS module imports, gives a guest its boot and file +# systems. +{ ... }: +{ ... }: +{ + system.stateVersion = "25.05"; +} diff --git a/tests/pinned-world/default.nix b/tests/pinned-world/default.nix index 2b06199a..7a9ac44b 100644 --- a/tests/pinned-world/default.nix +++ b/tests/pinned-world/default.nix @@ -3,7 +3,8 @@ # The pinned-world suite: caisson, the tree this file lives in, # composed with concrete pinned versions of the upstream world # (nixpkgs, home-manager, colmena, disko, nix-darwin, terranix, -# system-manager, read from tests/dependencies) and exercised end to end. Every upstream +# system-manager, microvm.nix, read from tests/dependencies) and +# exercised end to end. Every upstream # expectation caisson relies on is held by a probe here, so a pin # that moves and breaks an expectation fails at evaluation, in this # suite, rather than in a consumer. @@ -67,6 +68,8 @@ let registered."caisson/disko" registered."caisson/caisson-facter" registered."caisson/facter" + registered."caisson/caisson-microvm" + registered."caisson/microvm" registered."caisson/darwin" registered."caisson/terranix" registered."caisson/system-manager" @@ -90,6 +93,7 @@ let "caisson-colmena" "caisson-disko" "caisson-facter" + "caisson-microvm" "callConsumerFlake" "callFlake" "colmena" @@ -111,6 +115,7 @@ let "libManifest" "libOverlays" "manifestOf" + "microvm" "mkLib" "mkLibOverlay" "mkLibOverlays" @@ -223,6 +228,7 @@ let system-manager hjem terranix + microvm ; }; projects = { @@ -364,6 +370,103 @@ let }; colmenaTop = colmenaTopWith { }; + # What a microvm.nix guest needs beside what microvm.nix's guest + # module gives it, which boots it and gives it its file systems. + microvmGuestBase = { + system.stateVersion = "25.05"; + }; + + # A guest in microvm.nix's own form, as microvm.nix's examples write + # one: it runs on qemu and reads the store of its host over 9p. + upstreamMicrovmGuest = { + imports = [ microvmGuestBase ]; + microvm = { + hypervisor = "qemu"; + mem = 640; + shares = [ + { + tag = "ro-store"; + source = "/nix/store"; + mountPoint = "/nix/.ro-store"; + proto = "9p"; + } + ]; + }; + }; + + # A NixOS host named `host` that selects caisson's `microvm-host` + # module, inside the structural top of `registeredLib`, whose + # registry holds the NixOS configurations `microvm-slot` and + # `microvm-reset` the slots take. The host holds the caisson-microvm configuration `pool`, with + # the guests `a` and `b` and a pool of two slots of the kind + # `microvm-slot`, one slot of the kind `microvm-reset` that selects + # its modules itself, and the microvm configuration `upstream`. The + # user `probe` may control `a` and `microvm-slot-2`. + microvmHostTop = registeredLib.caisson.structural.mkTopConfiguration { + moduleImports = _modules: [ ]; + configModule = + { lib, ... }: + { + caisson.nixos.configurations.host = lib.caisson.nixos.mkConfiguration { + extraModuleImports = modules: [ modules."caisson/microvm-host" ]; + configModule = + { lib, ... }: + { + imports = [ minimalNixosBase ]; + users.users.probe.isNormalUser = true; + caisson.microvm-host.operators.probe = [ + "a" + "microvm-slot-2" + ]; + caisson.caisson-microvm.configurations.pool = lib.caisson.caisson-microvm.mkConfiguration { + configModule = + { lib, ... }: + { + hypervisor = "cloud-hypervisor"; + vcpu = 2; + mem = 768; + runtimeMaxSec = 3600; + shares = [ + { + tag = "ro-store"; + source = "/nix/store"; + mountPoint = "/nix/.ro-store"; + proto = "virtiofs"; + } + ]; + slots.microvm-slot = { + count = 2; + cidBase = 100; + shareRoot = "/var/lib/probe"; + }; + slots.microvm-reset = { + count = 1; + cidBase = 200; + moduleImports = modules: [ modules."caisson/microvm-guest" ]; + }; + caisson.nixos.configurations.a = lib.caisson.nixos.mkConfiguration { + configModule = { + imports = [ microvmGuestBase ]; + microvm.vsock.cid = 3; + }; + }; + caisson.nixos.configurations.b = lib.caisson.nixos.mkConfiguration { + configModule = { + imports = [ microvmGuestBase ]; + microvm.vsock.cid = 4; + microvm.mem = 1024; + }; + }; + }; + }; + caisson.microvm.configurations.upstream = lib.caisson.microvm.mkConfiguration { + configModule = upstreamMicrovmGuest; + }; + }; + }; + }; + }; + # A hive in the format colmena reads, with every feature the # upstream-compatible integration carries: `meta.nixpkgs`, shared and # per-node special arguments, `defaults` that read the node's name, a @@ -539,6 +642,8 @@ let "caisson/disko" "caisson/caisson-facter" "caisson/facter" + "caisson/caisson-microvm" + "caisson/microvm" "caisson/darwin" "caisson/terranix" "caisson/system-manager" @@ -644,6 +749,8 @@ let "disko" "caisson-facter" "facter" + "caisson-microvm" + "microvm" "darwin" "terranix" "system-manager" @@ -940,6 +1047,16 @@ let "system x86_64-linux / nixos probe-machine: inline/4" "system x86_64-linux / nixos probe-machine / system x86_64-linux / hjem probe: inline/7" ]; + # The slots take their registered configuration, so only the + # guests `a` and `b` given a module as a value record one. + microvm = [ + "top: inline/2" + "system x86_64-linux / nixos host: inline/5" + "system x86_64-linux / nixos host / caisson-microvm pool: inline/5" + "system x86_64-linux / nixos host / caisson-microvm pool / system x86_64-linux / nixos a: inline/5" + "system x86_64-linux / nixos host / caisson-microvm pool / system x86_64-linux / nixos b: inline/5" + "system x86_64-linux / nixos host / system x86_64-linux / microvm upstream: inline/4" + ]; }; recorded = { registered = inlineOf registeredWorld; @@ -959,6 +1076,7 @@ let }); darwin = inlineOf darwin.top; hjem = inlineOf hjem.top; + microvm = inlineOf microvmHostTop; }; # The registered configurations of the registered world, as # its `configs.` declaration events record them. @@ -2446,6 +2564,262 @@ let } ]; + # A host that selects caisson's `microvm-host` NixOS module runs + # the guests of its caisson-microvm configuration: the guests + # declared there, `a` and `b`, and the slots, `microvm-slot-1`, + # `microvm-slot-2` and `microvm-reset-1`. Each is a + # `microvm.vms.` whose `evaluatedConfig` is caisson's + # evaluation of the guest, the configuration the top publishes as + # `nixosConfigurations.`. The caisson-microvm configuration + # gives the guests its start options, its hypervisor, CPUs, memory + # and shares, and a guest that sets its memory keeps it. A guest + # sets its own CID, and a slot has the CID `cidBase + k` and the + # shares of its slot directory. The + # unit of each guest stops it after `runtimeMaxSec`. + caissonMicrovmGuestsRunOnTheirHost = + let + top = microvmHostTop; + host = top.nixosConfigurations.host.config; + vms = host.microvm.vms; + native = [ + "a" + "b" + "microvm-reset-1" + "microvm-slot-1" + "microvm-slot-2" + ]; + guest = name: vms.${name}.evaluatedConfig.config.microvm; + sharesOf = + name: + builtins.map (share: { + inherit (share) + tag + source + mountPoint + readOnly + proto + ; + }) (guest name).shares; + roStore = { + tag = "ro-store"; + source = "/nix/store"; + mountPoint = "/nix/.ro-store"; + readOnly = false; + proto = "virtiofs"; + }; + slotShares = slot: [ + roStore + { + tag = "work-in"; + source = "/var/lib/probe/${slot}/in"; + mountPoint = "/work/in"; + readOnly = true; + proto = "virtiofs"; + } + { + tag = "work-out"; + source = "/var/lib/probe/${slot}/out"; + mountPoint = "/work/out"; + readOnly = false; + proto = "virtiofs"; + } + ]; + in + builtins.attrNames top.nixosConfigurations == [ + "a" + "b" + "host" + "microvm-reset-1" + "microvm-slot-1" + "microvm-slot-2" + "upstream" + ] + && builtins.attrNames vms == native ++ [ "upstream" ] + && builtins.all ( + name: + vms.${name}.evaluatedConfig.config.system.build.toplevel.drvPath + == top.nixosConfigurations.${name}.config.system.build.toplevel.drvPath + ) (native ++ [ "upstream" ]) + && builtins.all (name: !vms.${name}.autostart && vms.${name}.restartIfChanged) native + && builtins.all ( + name: host.systemd.services."microvm@${name}".serviceConfig.RuntimeMaxSec == "3600" + ) native + && !(host.systemd.services."microvm@upstream".serviceConfig ? RuntimeMaxSec) + && + builtins.map (name: (guest name).vsock.cid) native == [ + 3 + 4 + 201 + 101 + 102 + ] + && + builtins.map (name: { inherit (guest name) hypervisor vcpu mem; }) native == builtins.map (name: { + hypervisor = "cloud-hypervisor"; + vcpu = 2; + mem = if name == "b" then 1024 else 768; + }) native + && sharesOf "a" == [ roStore ] + && sharesOf "microvm-reset-1" == [ roStore ] + && sharesOf "microvm-slot-1" == slotShares "microvm-slot-1" + && sharesOf "microvm-slot-2" == slotShares "microvm-slot-2"; + + # A microvm configuration declared inside a host is the guest + # itself: the host's `microvm.vms.upstream` has the evaluation of + # the microvm configuration as its `evaluatedConfig`, with the + # microvm options the guest's own module sets. The host starts it at + # boot, as microvm.nix does by default, and restarts it when it + # changes. The microvm configuration records microvm.nix's guest + # module as one declaration of the source `microvm`. + microvmConfigurationIsTheGuest = + let + top = microvmHostTop; + vm = top.nixosConfigurations.host.config.microvm.vms.upstream; + evaluated = vm.evaluatedConfig.config; + manifest = + top.caisson.manifest.children.system.x86_64-linux.children.nixos.host.children.system.x86_64-linux.children.microvm.upstream; + in + evaluated.system.build.toplevel.drvPath + == top.nixosConfigurations.upstream.config.system.build.toplevel.drvPath + && evaluated.microvm.hypervisor == "qemu" + && evaluated.microvm.mem == 640 + && builtins.map (share: share.proto) evaluated.microvm.shares == [ "9p" ] + && vm.autostart + && vm.restartIfChanged + && manifest.type == "microvm" + && manifest.outputs.runner.drvPath == evaluated.microvm.declaredRunner.drvPath + && + builtins.map (event: event.origin) ( + builtins.filter (event: event.operation == "declaration" && event.key == "microvm") manifest.history + ) == [ + { + project = "microvm"; + file = "nixos-modules/microvm"; + } + ]; + + # The polkit rule of a host lets each user its + # `caisson.microvm-host.operators` names start, stop and restart + # the units of the guests listed for that user and no other unit. + # A name that no guest of the host is declared as is refused. + microvmOperatorsGetExactlyTheirUnits = + let + text = microvmHostTop.nixosConfigurations.host.config.security.polkit.extraConfig; + units = builtins.filter builtins.isList (builtins.split "microvm@[a-z0-9-]*\\.service" text); + refused = microvmHostTop.nixosConfigurations.host.extendModules { + modules = [ { caisson.microvm-host.operators.probe = [ "missing" ]; } ]; + }; + in + pkgs.lib.hasInfix ''var units = {"probe":["microvm@a.service","microvm@microvm-slot-2.service"]};'' text + && builtins.length units == 2 + && (builtins.tryEval microvmHostTop.nixosConfigurations.host.config.system.build.toplevel.drvPath) + .success + && !(builtins.tryEval refused.config.system.build.toplevel.drvPath).success; + + # A slot kind that passes `moduleImports` replaces the default + # selection of the nixos class for its slots: `microvm-reset-1` + # selects caisson's `microvm-guest` NixOS module and nothing else, + # where `microvm-slot-1` takes the default of the class, caisson's + # `default` NixOS module among it, and the `microvm-guest` module + # the caisson-microvm configuration adds for its nearest NixOS + # configurations. + microvmSlotKindSelectsItsModules = + let + pool = + microvmHostTop.caisson.manifest.children.system.x86_64-linux.children.nixos.host.children.caisson-microvm.pool; + slot = name: pool.children.system.x86_64-linux.children.nixos.${name}; + reset = (slot "microvm-reset-1").moduleImports; + default = (slot "microvm-slot-1").moduleImports; + in + reset.act == "reset" + && reset.selected == [ "modules.nixos.caisson/microvm-guest" ] + && default.act == "default" + && builtins.elem "modules.nixos.caisson/default" default.selected + && + pkgs.lib.last default.default == { + key = "modules.nixos.microvm-guest"; + level = [ + { + type = "system"; + name = "x86_64-linux"; + } + { + type = "nixos"; + name = "host"; + } + { + type = "caisson-microvm"; + name = "pool"; + } + ]; + file = "modules/caisson-microvm/default"; + nearest = true; + }; + + # A guest belongs to the host it is declared inside. In a colmena + # configuration, a host on a disk layout is a node, and the guests + # inside it are none: neither a NixOS configuration of its + # caisson-microvm configuration nor its microvm configuration has + # colmena's deployment options or the file systems of the layout, + # which reach the nearest NixOS configurations beneath the colmena + # configuration and the layout, the host. The host still runs both + # guests. + microvmGuestsAreNoNodes = + let + top = hiveLib.caisson.structural.mkTopConfiguration { + moduleImports = _modules: [ ]; + configModule = + { lib, ... }: + { + caisson.caisson-colmena.configurations.default = lib.caisson.caisson-colmena.mkConfiguration { + configModule = + { lib, ... }: + { + caisson.caisson-disko.configurations.host = lib.caisson.caisson-disko.mkConfiguration { + configModule = + { lib, ... }: + { + imports = [ luksLvmLayout ]; + caisson.nixos.configurations.host = lib.caisson.nixos.mkConfiguration { + extraModuleImports = modules: [ modules."caisson/microvm-host" ]; + configModule = + { lib, ... }: + { + imports = [ diskoMachineBase ]; + caisson.caisson-microvm.configurations.pool = lib.caisson.caisson-microvm.mkConfiguration { + configModule = + { lib, ... }: + { + caisson.nixos.configurations.native = lib.caisson.nixos.mkConfiguration { + configModule = microvmGuestBase; + }; + }; + }; + caisson.microvm.configurations.upstream = lib.caisson.microvm.mkConfiguration { + configModule = upstreamMicrovmGuest; + }; + }; + }; + }; + }; + }; + }; + }; + }; + published = top.nixosConfigurations; + guests = [ + "native" + "upstream" + ]; + in + builtins.attrNames top.colmenaHive.nodes == [ "host" ] + && published.host.options ? deployment + && published.host.config.fileSystems ? "/boot" + && builtins.attrNames published.host.config.microvm.vms == guests + && builtins.all ( + name: !(published.${name}.options ? deployment) && !(published.${name}.config.fileSystems ? "/boot") + ) guests; + # The convention for a machine inside a layout: the layout declares # it under the layout's own name, read from # `lib.caisson-core.evalManifest.name`, and caisson's default NixOS diff --git a/tests/unit/lib-overlays.nix b/tests/unit/lib-overlays.nix index 62024817..5af7a317 100644 --- a/tests/unit/lib-overlays.nix +++ b/tests/unit/lib-overlays.nix @@ -124,6 +124,8 @@ let "disko" "caisson-facter" "facter" + "caisson-microvm" + "microvm" "terranix" "system-manager" "treefmt" @@ -5686,6 +5688,8 @@ in "system-manager" "caisson-facter" "facter" + "caisson-microvm" + "microvm" "treefmt" "nvf" "nixvim" @@ -5704,6 +5708,7 @@ in "system-manager" "caisson-facter" "facter" + "microvm" "treefmt" "nvf" "nixvim"