Skip to content

Latest commit

 

History

History
2085 lines (1840 loc) · 127 KB

usage.md

File metadata and controls

2085 lines (1840 loc) · 127 KB

Command-line usage

NVX has two command-line interfaces, which run from the repository root:

  • nvx runs NVX microVMs and sandboxes: run boots a microVM on the terminal, or restores a named snapshot once, sandbox runs or manages workloads over an EROFS image or a GPT disk and a private ext4 scratch or tmpfs upper, snapshot capture captures a named snapshot, from which snapshot restore starts managed sandboxes, and download installs the release of the checkout, which run and sandbox also download into a checkout that holds no artifacts.

  • scripts/nvx.py, with Python 3.10 or newer, builds, tests, benchmarks, and packages NVX:

    python3 scripts/nvx.py COMMAND [OPTIONS]
    

The examples on this page use the POSIX spelling. On Windows, use nvx\target\release\nvx.exe and python scripts\nvx.py instead. Every command accepts -h or --help, including nested commands:

nvx/target/release/nvx run --help
python3 scripts/nvx.py --help
python3 scripts/nvx.py performance gate --help

nvx

nvx is the Rust command-line interface that runs NVX's microVMs and sandboxes. Cargo builds it from the repository root at nvx/target/release/nvx, or nvx.exe on Windows, with the Rust release that rust-toolchain.toml pins; rustup installs Cargo. --target-dir keeps the executable there even where CARGO_TARGET_DIR names another directory. Build it again after changing nvx/:

cargo build --release --locked --features cli --manifest-path nvx/Cargo.toml --target-dir nvx/target

Without --root, nvx uses the artifacts of the checkout that holds it: the guest artifacts in build/ and OpenVMM in openvmm/target/release/. download installs those of a release there, and run and sandbox download them first into a Git checkout that holds none of them. The nvx of an extracted release package, bin/nvx, uses the release's artifacts; see Artifact root.

nvx keeps the state of its sandboxes and snapshots in the directories that their commands name, --state-dir and --store-dir, in formats of its own, which another version of NVX may not read and never migrates: upgrade with fresh state, as Upgrades says.

The test-microvm scenarios run nvx too. They run, in this order:

  1. the executable that the NVX environment variable names;
  2. nvx/target/release/nvx[.exe] in the checkout; or else
  3. that executable, after they build it with the command above, with Cargo's output on standard error, so that it never mixes with their output.

Without NVX, a built executable, or Cargo, they fail and name these choices. They rebuild nothing once the executable exists. The managed-lifecycle and vm-lifecycle scenarios also need Cargo, which runs their nvx tests.

The provenance of a checkout's nvx, build/nvx.provenance.json, binds the SHA-256 of its executable to a digest of the nvx sources that built it, the crate's manifest, lock file, and src/ tree and rust-toolchain.toml, and to the Rust release, target, and Cargo features of the build. nvx.py records it when it builds nvx for the scenarios, and record-nvx-provenance records it for a build made otherwise, such as with the command above. package and the adversarial executor refuse an nvx that it does not describe as built from the checkout's sources, so a stale build never ships or runs unnoticed.

Error codes

nvx reports each error on standard error as error[CODE]: MESSAGE, and exits with status 1, and each usage error as the usage of its parser and then PROG: error[CODE]: MESSAGE, such as nvx run: error[argument-invalid]: ..., and exits with status 2. Each error is one line: a line break in a value that its message names, such as a path, prints as \n or \r. CODE is a stable code, which tells errors apart where their messages may change: callers match codes, and stable fragments of messages, such as the names of options, fields, and files, never whole messages. A code keeps its meaning and is never reused, and a later release may add codes, which callers treat as failures of the command. The usage errors that nvx.py itself reports, as argparse does, carry no code, and neither does an interruption by SIGINT or, outside Windows, SIGTERM, which prints Interrupted and exits with status 130. The nvx README gives the ErrorKind of each code, which callers of the nvx library see in its place.

Code Status Meaning
argument-unrecognized 2 The command line holds an argument that the command does not take, such as an unknown option.
argument-missing 2 A required argument, or the value of an option, is missing.
argument-invalid 2 An argument has an invalid value, such as one that is not among its choices, not a number, not of its form, or not valid Unicode.
unsupported 1 NVX refuses the request: options that it cannot combine, a value that it does not accept, or a policy that it cannot represent or enforce. The message names the option or field.
unavailable 1 A dependency is missing or unusable, other than as a finer code says: install metadata, the hypervisor, a release, a distro layer or scratch image that the request names, or a host that NVX does not support.
artifact-missing 1 An artifact that the command runs, such as OpenVMM, the guest kernel, or an initramfs, does not exist, or no artifact root holds them.
sandbox-not-provisioned 1 No sandbox is provisioned at --state-dir, which does not exist, is not a plain directory, or holds no configuration.
sandbox-provisioned 1 provision found the sandbox provisioned already, or it was provisioned again while deprovision removed it.
sandbox-not-running 1 exec or stop found the VM of a provisioned sandbox not running.
sandbox-running 1 start or deprovision found the sandbox's VM, or a launch of it, running.
sandbox-exited 1 The sandbox's OpenVMM exited without a stop: the transition that finds it reports it and clears its runtime record, and start launches the sandbox again.
sandbox-busy 1 Another lifecycle transition held the sandbox's lock, or another command the lock of a snapshot or of a store's captures, until --timeout passed, or another exec held the sandbox's exec lock until --wait passed.
runtime-mismatch 1 The sandbox's VM runs under another runtime identity than the artifacts', whose differing parts the message names, so only stop, which terminates it, applies.
vmm-indeterminate 1 Whether the sandbox's OpenVMM runs cannot be determined, so its records remain for a later attempt.
format-unsupported 1 A record or report has a format that this build does not read, such as the state of a sandbox that another build provisioned or started.
foreign-files 1 deprovision found files that NVX does not own in the state directory, which the message names, and kept them with the directory.
outcome-report-exists 1 A file exists at the --outcome-report path.
scratch-claimed 1 Another sandbox claims the scratch image.
exited 1 OpenVMM exited, as when a start fails, whose message names the category and status of OpenVMM's outcome report when it has one.
still-running 1 A stop could not confirm the VM process's exit, so the caller keeps its verified process.
exec-refused 1 The guest agent refused the workload, as for a working directory that it cannot use.
endpoint-closed 1 OpenVMM closed the control endpoint, as it does when the VM shuts down.
protocol 1 The control session failed otherwise, as when the guest breaks the protocol or the session is reset.
timed-out 1 Another deadline passed, such as that of a stop or of a control request.
io 1 A host I/O operation failed, or state that NVX recorded is unreadable or malformed.
not-found 1 Other state that the command names does not exist on the host.
other 1 Any other failure.

run

nvx/target/release/nvx run
    [--guest {alpine,ubuntu,azurelinux}]
    [--hypervisor {auto,whp,kvm,mshv}]
    [--machine {microvm}]
    [--memory-mib MIB]
    [--memory-capacity-mib MIB]
    [--processors {1,2,4,8}]
    [--cpu-profile ID]
    [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]...
    [--mount-deny HOST_PATH]...
    [--mount-allow HOST_PATH]...
    [--mount-write HOST_PATH]...
    [--mount-owner {vmm,caller}]
    [--net IPV4/PREFIX]
    [--network-profile {portable}]
    [--network-egress {allow,deny}]
    [--network-ingress {allow,deny}]
    [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]...
    [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]...
    [--network-egress-policy-file PATH]
    [--host-loopback {allow,deny}]
    [--network-proxy IPV4:TCP-PORT]
    [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]...
    [--outcome-report PATH]
    [--cmdline TEXT]
    [--restore-snapshot NAME --store-dir PATH --state-dir PATH]
    [--layer ROLE,PATH,IMAGE_UUID]...
    [--layer-format {erofs,gpt}]
    [--scratch PATH]
    [--upper {scratch,ram}]
    [--ram-upper-percent PERCENT]
    [--entrypoint PATH]
    [--arg VALUE]...
    [--hostname NAME]
    [--workload-user UID:GID]
    [--timeout SECONDS]
    [--dry-run]
Option Default Description
--guest {alpine,ubuntu,azurelinux} alpine Select Alpine, Ubuntu, or Azure Linux userland with the same NVX kernel. A restore refuses it.
--hypervisor {auto,whp,kvm,mshv} auto Select the OpenVMM hypervisor. auto chooses WHP on Windows and KVM elsewhere.
--machine {microvm} microvm Select the fixed-topology microVM with shared-status edge interrupts.
--memory-mib MIB guest-specific Set guest memory in MiB, an ASCII decimal integer from 1 to 17592186044415, without signs, whitespace, or underscores. Defaults to 128 for Alpine, 512 for Ubuntu, and 512 for Azure Linux, and for a restore to 256, as for snapshot restore, which must be a multiple of 128 between the snapshot's boot memory and its capacity.
--memory-capacity-mib MIB none Reserve an immutable, 128 MiB-aligned RAM capacity for a fresh microVM snapshot, in the range of --memory-mib. A restore refuses it.
--processors {1,2,4,8} 1 Select the microVM processor count, written as one of the choices. A restore refuses it: its VM keeps the processors that the snapshot had online.
--cpu-profile ID auto Select the guest's CPU profile: auto for the built-in profile of the host's CPU, which falls back with a warning to a development profile derived from this host on an Intel or AMD CPU that no built-in profile serves, a built-in profile ID, or host for that development profile. A restore refuses it: it uses the snapshot's profile.
--mount GUEST_TARGET,HOST_PATH[,ro|rw] none Expose a host directory, or a single regular host file without its directory, at the guest target: an absolute, canonical path other than /, without whitespace, control characters, double quotes, backslashes, =, or any UTF-8 byte that Linux treats as whitespace, such as the 0xa0 in à. Cold boots also protect the system paths listed below. The mode follows the last comma, so HOST_PATH may contain commas when the mode is given. Repeat to expose more directories or files, each with its own target and mode, through an aggregate on the microVM's one virtio-fs device, which the guest binds at their targets. Targets must not overlap, but host paths may: a mapped path inside a mapped directory shares that directory's child, unless it is read-only inside a writable part of it, which is refused; see virtio-fs host mapping. The shares' kernel command-line tokens must fit its 1024-byte budget. An rw mapping accepts guest-created symbolic links, which the host never follows. A restore refuses it and the other share options.
--mount-deny HOST_PATH none Hide one existing file or directory inside a mounted host root; repeat to deny multiple paths. It may name the root itself when --mount-allow paths inside it remain. With several mappings, or a shared file, the path must be absolute; a shared file takes none.
--mount-allow HOST_PATH none Expose one existing file or directory inside a --mount-deny path again; the hidden directories on the way list only the entries that lead to it. Repeat to allow multiple paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none. See Access policy.
--mount-write HOST_PATH none Make one existing file or directory one of the only writable parts of an rw mapping; the guest's writes anywhere else in it fail with EROFS. Repeat to declare multiple writable paths. With several mappings, or a shared file, the path must be absolute; a shared file takes none.
--mount-owner {vmm,caller} vmm Select the host identity of the guest's operations on every --mount directory or file. caller performs each one as the guest caller's UID and GID and squashes guest root to the owner of the shared directory or file; it requires a Linux host. See Run.
--net IPV4/PREFIX none Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in fd00::/96, such as fd00::a00:2/120 for 10.0.0.2/24. A restore takes its NIC, with these addresses, from the snapshot, which snapshot capture --net gives one, and rejects this option.
--network-profile {portable} none Select the required cross-platform network behavior contract; must be specified with --net on a fresh boot, and without it to restore a snapshot with a NIC.
--network-egress {allow,deny} allow Set the default guest egress policy.
--network-ingress {allow,deny} deny Set the default host ingress policy. allow publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with --host-loopback allow; see Private-network ingress.
--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]] none Allow matching guest egress; repeat to add rules. CIDR is an IPv4 or IPv6 address or network, and a rule matches only its own family. PROTOCOL is tcp, udp, or icmp and matches every port when PORT is omitted; only tcp and udp take a PORT or an inclusive PORT-PORT range, such as 8000-8010. OpenVMM relays only IPv4 echo for icmp; see ICMP.
--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]] none Deny matching guest egress; repeat to add rules. Deny rules take precedence. PROTOCOL and PORT select traffic as for --network-egress-allow.
--network-egress-policy-file PATH none Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit --network-egress; cannot be mixed with explicit allow/deny rule flags.
--host-loopback {allow,deny} existing mapping Control access between the guest and host loopback services. Explicit allow is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with --network-ingress allow admits the host as well; it needs a --net prefix of /29 or shorter, which OpenVMM checks; see Host-loopback access.
--network-proxy IPV4:TCP-PORT none Allow one explicit host TCP proxy endpoint.
--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT none Publish one TCP or UDP localhost port to the guest; repeat to add forwards.
--outcome-report PATH none Write a bounded local JSON outcome report to a new file in an existing plain directory. A cold boot rejects an existing destination before launch and never replaces a file or follows a destination symlink. A missing report or publication failure is diagnosed without replacing OpenVMM's exit status. A restore refuses it.
--cmdline TEXT empty Append kernel parameters. The tokens that NVX renders for the guest's root, view, and shares, such as nvx_hostname, nvx_machine_id, and nvx_maps, and virtfs_* tokens, are refused, and OpenVMM refuses those that it owns; see Effective Command Line. Cold boots reject double quotes and a standalone -- token, which the VM control console cannot represent. A restore refuses it.
--restore-snapshot NAME none Restore the snapshot NAME of --store-dir, which snapshot capture published, in place of a cold boot, as described below. A raw OpenVMM snapshot directory is refused.
--store-dir PATH none The named snapshot store of --restore-snapshot, which a restore requires.
--state-dir PATH none The directory of a restore's sandbox, which a restore requires: one that does not exist, outside the store, on the store's filesystem. The run creates it, and removes it once it ends. A cold boot refuses it.
--layer, --layer-format, --scratch, --upper, --ram-upper-percent, --entrypoint, --arg, --hostname, --workload-user, --timeout as for sandbox The image, the upper, and the workload of a restore, and the seconds of its waits, as snapshot restore takes and checks them: the workload runs over the control protocol, so its entrypoint and arguments may contain whitespace and double quotes. A cold boot refuses them.
--dry-run off Print the generated OpenVMM command without running it. A restore refuses it.

Cold --mount targets may not cover, equal, or lie below /bin, /boot, /dev, /etc, /lib, /proc, /root, /run, /sbin, /sys, /usr, /var, or /init: these paths belong to the guest's blockless system. For example, use /config/single.txt rather than /etc/single.txt for a shared configuration file. Cold mount-policy options require a --mount, and network-policy options require networking. A proxy is the guest's sole egress path, so it cannot be combined with --network-egress-allow rules for direct connections.

nvx run checks its options, prints the OpenVMM command of a cold boot after >> , and unless --dry-run is given, runs OpenVMM in a private temporary VM directory, which it removes once OpenVMM has exited, on the terminal. A cold boot attaches the guest's console, the boot console hvc1, to the standard streams, so that the terminal's input reaches the guest through it rather than through the byte channel, hvc0, whose output, such as the kernel's early messages, goes to standard error with OpenVMM's log. It exits with status 1 for an error, 2 for a usage error, and 130 after Ctrl-C, and otherwise with the status that OpenVMM returns. Its errors and usage errors carry their error codes, as error[CODE]: MESSAGE and nvx run: error[CODE]: MESSAGE. A run requires the OpenVMM release binary, build/vmlinux, and the selected build/initramfs*.cpio.gz, and a restore the Alpine one. When one of them is missing from a Git checkout that holds none of the files that a release installs, run first installs the release of the checkout, as download does, reporting on standard error; a checkout that holds some of them gets a warning instead, and the missing artifact fails the run. An option takes its value from the next argument, or after = in its own, which a value that starts with - needs unless it starts like a negative number in ASCII digits, such as -1, and its name cannot be abbreviated. Ubuntu selection never falls back to Alpine. See Run for host setup, guest shutdown, networking, and virtio-fs examples.

With --restore-snapshot NAME, run restores the named snapshot of --store-dir in place of a cold boot, as snapshot restore restores one, but once: it provisions a managed sandbox in --state-dir, a new directory outside the store, on the filesystem of the snapshot's files, which the restore links into it, with the image, upper, and workload of its restore options, and starts it from the snapshot under the snapshot's lock. It prints no OpenVMM command, and forwards the workload's output, as it comes; the workload reads no standard input. Once the workload has exited with status 0 within --timeout, 60 seconds by default, it stops the VM; it then removes the sandbox and --state-dir, also after a failure or Ctrl-C, except as the next paragraph says, and exits with the workload's status, or with status 1 for a restore that fails, and 130 after Ctrl-C. --restore-snapshot names only a snapshot of the store, so run refuses a raw OpenVMM snapshot directory without reading it. It also refuses a --state-dir that exists or lies in the store, a workload that snapshot restore refuses, and the options that the snapshot or its sandbox fixes: --guest, --memory-capacity-mib, --processors, --cpu-profile, --net, the share options, --outcome-report, --cmdline, and --dry-run. A cold boot refuses --store-dir, --state-dir, and the restore options. A restore's NIC comes from the snapshot: --net belongs to the cold boot and to snapshot capture, which gives a snapshot its NIC. The restore of a snapshot without a NIC takes no network options, and that of a snapshot with a NIC installs them, with --network-profile, as the restore's own network policy, with its DNS server, egress, forwards, and published listeners, as snapshot restore describes.

A run that dies before it removes its sandbox, as kill -9 ends it, leaves --state-dir with the sandbox's files and its view of the snapshot, and its VM, whose OpenVMM process keeps running, also once its workload has exited, until a command of the sandbox ends it; nothing else does. So does Ctrl-C once the workload has exited with status 0, while the run stops the VM. The snapshot's lock goes with the run, so other restores of the snapshot proceed, and a later run refuses the existing --state-dir. While the workload ran, the start was unfinished, and nvx sandbox deprovision --state-dir PATH ends the VM, through the recovery of the start's launch journal, and removes the directory. Once the workload had exited with status 0, the VM runs as a started sandbox does, and nvx sandbox stop --state-dir PATH must end it before deprovision removes the directory.

If no built-in CPU profile serves an Intel or AMD host's CPU, OpenVMM falls back to a host profile and warns with NVX-CPU-PROFILE-FALLBACK: at every cold boot, as CPU profiles describes. After a failed cold boot on such a CPU, run names the CPU, lists the CPUs that the built-in profiles cover, says that auto falls back on it, and asks for the CPU's fingerprint through a CPU profile request, with the OpenVMM command that writes the fingerprint. OpenVMM's own error, above the guidance, names the cause of the failure, and says so where the host profile cannot boot. On a CPU of another vendor that no built-in profile serves, OpenVMM exits with E_PROFILE_HOST_UNKNOWN before it creates the VM, and run explains that host profiles cannot boot there either. A host whose hypervisor does not support its built-in profile fails with E_PROFILE_UNSUPPORTED instead, and on an Intel or AMD CPU OpenVMM's error itself names --cpu-profile host. nvx run identifies the CPU with CPUID, as OpenVMM does, derives the guidance from its copy of the CPUs that the built-in profiles serve, and prints it only after OpenVMM fails a cold boot, never after a dry run, a restore, an option that nvx refuses, or Ctrl-C.

sandbox

nvx/target/release/nvx sandbox
    [{run,provision,start,exec,stop,deprovision}]
    [--layer ROLE,PATH,IMAGE_UUID]
    [--layer-format {erofs,gpt}]
    [--scratch PATH]
    [--upper {scratch,ram}]
    [--ram-upper-percent PERCENT]
    [--state-dir PATH]
    [--entrypoint PATH]
    [--arg VALUE]...
    [--hostname NAME]
    [--workload-user UID:GID]
    [--memory-mib MIB]
    [--timeout SECONDS]
    [--wait SECONDS]
    [--exec-timeout-ms MILLISECONDS]
    [--cwd GUEST_PATH]
    [--environment KEY=VALUE]...
    [--environment-file PATH]
    [--inherit-default-environment]
    [--hypervisor {auto,whp,kvm,mshv}]
    [--mount GUEST_TARGET,HOST_PATH[,ro|rw]]...
    [--mount-deny HOST_PATH]...
    [--mount-allow HOST_PATH]...
    [--mount-write HOST_PATH]...
    [--mount-owner {vmm,caller}]
    [--net IPV4/PREFIX]
    [--network-profile {portable}]
    [--network-egress {allow,deny}]
    [--network-ingress {allow,deny}]
    [--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]]...
    [--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]]...
    [--network-egress-policy-file PATH]
    [--host-loopback {allow,deny}]
    [--network-proxy IPV4:TCP-PORT]
    [--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT]...
    [--outcome-report PATH]
    [--cmdline TEXT]
    [--dry-run]

Network policy and live-share options configure only the run and provision launches, which check the network options as one typed policy before anything launches or is recorded; see Sandbox networking.

nvx sandbox runs the operation, run by default, with the options given. It exits with status 1 for an error, 2 for a usage error, and 130 after Ctrl-C, and otherwise with the status that the workload or OpenVMM returns. Its errors and usage errors carry their error codes, as error[CODE]: MESSAGE and nvx sandbox: error[CODE]: MESSAGE. An exec, and the workload of a run, stream the command's standard input to the workload until it ends, and the workload's standard output and error back as they arrive, without a limit on their size; a command whose input is the null device gives the workload none. The workload must end within --timeout, or the command fails with timed-out. An error of the command's own follows the workload's output on a new line. Ctrl-C asks the guest to end the workload, and the command exits with status 130 without waiting for it: an exec leaves its sandbox running, while a run ends its VM. Options take their values as those of run do, and their names cannot be abbreviated either.

Option Default Description
{run,provision,start,exec,stop,deprovision} run Select a run, which keeps no state, or a managed lifecycle operation.
--layer ROLE,PATH,IMAGE_UUID required for run and provision Attach exactly one distro image by path, of the format of --layer-format. The filesystem UUID or GPT disk GUID must be lowercase hexadecimal in canonical hyphenated form; braces, prefixes, uppercase, and unhyphenated forms are refused.
--layer-format {erofs,gpt} erofs Select the image format of run or provision explicitly, without probing the file or inferring its format from its name. Other sandbox operations reject this option.
--scratch PATH required for run and provision with a scratch upper Attach a preformatted ext4 scratch image as the overlay's writable upper. --upper ram refuses it.
--upper {scratch,ram} scratch Select the overlay's writable upper for run or provision: scratch, the image of --scratch, or ram, a tmpfs in guest memory with no scratch image, which loses the workload's writes when the VM stops. The default RAM cap is half of guest memory, which --ram-upper-percent changes. See Overlay upper.
--ram-upper-percent PERCENT 50 Set the RAM upper's cap of run or provision to an ASCII decimal integer from 1 through 90 percent of VM memory. Requires --upper ram; other sandbox operations reject this option. The typed configuration of provision records the cap, image format and path, and upper kind, which each start reads unchanged. A configuration of another format is refused, not migrated; see Upgrades.
--state-dir PATH required for managed operations Select persistent sandbox state. run, which keeps no state, rejects this option.
--entrypoint PATH /bin/sh Select the absolute entrypoint of the workload of run or exec, which runs it over the control protocol. provision, which runs no workload, refuses one with whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace.
--arg VALUE none Append one nonempty argument of the workload of run or exec. Repeat to pass multiple arguments. provision refuses one with whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace.
--hostname NAME nvx-sandbox Set the workload UTS hostname.
--workload-user UID:GID 65534:65534 Select the fixed non-root workload identity for run or provision. Both IDs must be ASCII decimal integers from 1 to 4294967294, without signs, whitespace, or underscores.
--memory-mib MIB 256 Set guest memory in MiB, an ASCII decimal integer from 1 to 17592186044415, without signs, whitespace, or underscores.
--timeout SECONDS 60 Set the timeout for each managed control request and its response in run, start, exec, and stop, within which the workload of run and exec, whose output streams until it ends, must end; the longest time that run and stop wait for the VM to exit once the guest shuts it down; and the longest time that provision, start, exec, stop, and deprovision wait for another lifecycle transition of the sandbox. It is a decimal number, with an optional fraction and exponent, that must be finite and greater than 0.
--wait SECONDS 60 Set the longest time that a managed exec waits for the exec that another nvx process runs in the sandbox to end and release the sandbox's exec lock, since a sandbox runs one exec at a time; it then fails with sandbox-busy. stop never waits for an exec. Like --timeout, which still bounds each wait for another lifecycle transition, it is a decimal number that must be finite and greater than 0.
--exec-timeout-ms MILLISECONDS 0 Set the guest's timeout of the workload of run or exec to an ASCII decimal integer from 0 to 4294967295, without signs, whitespace, or underscores; zero disables the workload deadline. This is separate from the finite host --timeout control deadline.
--cwd GUEST_PATH / Set an absolute working directory inside the workload root for run or exec. It is resolved after the workload identity and root are applied; missing, inaccessible, or non-directory paths fail the workload launch. Default and layered environments point PWD at this directory; an exact replacement environment controls PWD itself.
--environment KEY=VALUE omitted Set the exact environment of the workload of run or exec. Repeat for multiple entries. Empty values, spaces, additional equals signs, and UTF-8 are preserved. Inline values are visible in the invoking host process arguments; use --environment-file for sensitive values.
--environment-file PATH omitted Read the exact environment of the workload of run or exec from a strict UTF-8 JSON array of KEY=VALUE strings, limited to 1 MiB of input. Invalid JSON, non-finite numbers, and unpaired surrogate escapes are refused. An empty array requests an empty environment. This option is mutually exclusive with --environment; omitting both preserves guest defaults.
--inherit-default-environment off Layer the environment of the workload of run or exec from --environment or --environment-file over the guest default environment instead of replacing it; an entry replaces the default variable of the same name. Without either option the workload already receives the defaults, so the flag has no effect.
--hypervisor {auto,whp,kvm,mshv} auto Select the host hypervisor.
--mount GUEST_TARGET,HOST_PATH[,ro|rw] none Live-share a host directory, or a single regular host file without its directory, at the absolute target inside the container rootfs for run or provision; defaults to ro. The target cannot contain whitespace, control characters, double quotes, backslashes, =, or any UTF-8 byte that Linux treats as whitespace, such as the 0xa0 in à. The mode follows the last comma, so HOST_PATH may contain commas when the mode is given. Repeat to attach more shares, each with its own target and mode, for example a read-write workspace and a read-only tool cache, as the children of an aggregate that the guest binds at their targets. Targets and host paths must not overlap, and run, provision, and start refuse a host path that equals, contains, or lies inside the sandbox's image, scratch image, OpenVMM, whether the artifact root's or the one that NVX_OPENVMM names, kernel, or initramfs, the state directory of provision and start, or the temporary VM directory of run; see Live host shares. The kernel command line announces only the number of the shares, whose mapping table the sandbox sends once its guest agent is ready, for at most 256 shares. An rw share accepts guest-created symbolic links, which the host never follows. /, /etc, /run, and /run/nvx, and the /etc/machine-id, /etc/resolv.conf, /proc, /sys, /dev, /.nvx-agent, and /run/nvx/time trees, are reserved.
--mount-deny HOST_PATH none Hide one existing file or directory inside a --mount host directory; relative paths are resolved inside it. It may name the directory itself when --mount-allow paths inside it remain. With several shares, it applies to the --mount before it, which must share a directory. Repeat to deny multiple paths.
--mount-allow HOST_PATH none Expose one existing file or directory inside a --mount-deny path of the same share again, for example a readable mcp-payloads directory inside hidden logs. The hidden directories on the way list only the entries that lead to it, and the workload cannot modify them. Relative paths are resolved inside the share; with several shares, it applies to the --mount before it. Repeat to allow multiple paths. See Access policy.
--mount-write HOST_PATH none Make one existing file or directory one of the only writable parts of an rw share; the workload's writes anywhere else in the share fail with EROFS. Relative paths are resolved inside the share; with several shares, it applies to the --mount before it, which must be rw. Repeat to declare multiple writable paths. provision persists the denied, allowed, and writable paths.
--mount-owner {vmm,caller} vmm Select the host identity of the shares' file operations for run or provision. vmm performs them as OpenVMM; caller performs them as the workload's UID and GID, squashes guest root to the owner of each shared host directory or file, and fails them with EPERM when OpenVMM cannot assume that identity. caller requires a Linux host and is persisted by provision. See File ownership.
--net IPV4/PREFIX none Enable virtio-net with a static guest address. The guest also gets the IPv6 address that embeds it in fd00::/96. The other network options need it. A policy with --network-egress deny and --host-loopback deny, and no allow rules, --network-ingress allow, or --network-proxy, attaches no network device, whatever its deny rules; a policy file's allow rules are those that it compiles to. Any other policy attaches one, even one that lets nothing through; see Sandbox networking.
--network-profile {portable} none Select the required cross-platform network behavior contract; must be specified with --net.
--network-egress {allow,deny} allow Set the default guest egress policy for run or provision.
--network-ingress {allow,deny} deny Set the host ingress policy for run or provision. allow publishes every TCP and UDP port on which the workload listens to clients on the host's private networks, and to the host itself only with --host-loopback allow; see Private-network ingress.
--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]] none Allow matching guest egress; repeat to add rules. Requires explicit --network-egress. CIDR is an IPv4 or IPv6 address or network, and a rule matches only its own family. PROTOCOL is tcp, udp, or icmp and matches every port when PORT is omitted; only tcp and udp take a PORT or an inclusive PORT-PORT range, such as 8000-8010. OpenVMM relays only IPv4 echo for icmp; see ICMP.
--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]] none Deny matching guest egress; repeat to add rules. Requires explicit --network-egress; deny rules take precedence. PROTOCOL and PORT select traffic as for --network-egress-allow.
--network-egress-policy-file PATH none Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions for run or provision. Managed provision persists the compiled rules in its network policy, not this path. Requires explicit --network-egress; cannot be mixed with explicit allow/deny rule flags.
--host-loopback {allow,deny} guest to host Control access between the workload and host loopback services for run or provision. Without the option, the workload reaches host loopback through its gateway, subject to the egress policy, and no port is published; deny closes both directions. Explicit allow is bidirectional: the workload reaches host loopback through its gateway, and every TCP and UDP port on which the workload listens is published on host loopback for the host alone, or with --network-ingress allow admits the host as well; see Host-loopback access.
--network-proxy IPV4:TCP-PORT none Make the network proxy-only: the workload reaches the HTTP or HTTPS proxy that listens on host 127.0.0.1:TCP-PORT at IPV4:TCP-PORT, where IPV4 must be the guest gateway, and nothing else. TCP-PORT can't be 53, where OpenVMM answers DNS at the gateway. Requires --network-egress deny and --host-loopback deny, and refuses egress rules, --network-ingress allow, and forwards. NVX gives the workloads of a cold boot no proxy settings; see Proxy-only networking.
--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT none Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Requires --host-loopback allow. A UDP forward also requires egress allowed by default and no deny rule that covers the guest network, which carries its replies; OpenVMM lets the replies on a forwarded TCP connection through whatever the egress policy.
--outcome-report PATH none Write a bounded local JSON outcome report: for run, OpenVMM's report of its VM, which the guest stopped after the workload; for exec, the workload's outcome.
--cmdline TEXT empty Append non-sandbox kernel parameters, which nvx reads as Linux does: split at whitespace, which includes the 0xa0 in à, outside double quotes, with names that match whether they spell a dash or an underscore. Parameters whose names start with nvx_ or virtfs_ are reserved and refused, even inside a quoted value, and so are a -- or unclosed quote that hides the tokens that OpenVMM appends, and double quotes, which the VM control console cannot represent; see Experimental single-workload sandbox. The guest powers off if its kernel applies a clock parameter that the time ABI forbids, such as tsc=unstable.
--dry-run off For run only, print the generated OpenVMM microVM command without launching it. Other operations reject this option.

run and start install the release of a Git checkout that holds no artifacts before they launch, as nvx run does. A run keeps no state: it launches the VM that a provision and a start of its options would launch, in a temporary VM directory that it removes, executes its workload once, as exec does, with its streams, has the guest shut the VM down, which it terminates if it does not exit within --timeout seconds, and exits with the workload's status; see Experimental single-workload sandbox.

See Run for artifact preparation, the security boundary, the managed lifecycle, and named snapshots.

snapshot capture

nvx/target/release/nvx snapshot capture baseline \
    --store-dir /private/snapshots --hypervisor kvm

Use mshv on Linux/MSHV or whp on Windows. The store's parent must exist; the store and its capture VM are private to the current user. Capture uses the identified OpenVMM, kernel and Alpine initramfs already installed in the artifact root, without downloading substitutes or accepting NVX_OPENVMM overrides.

The VM boots with nvx_sandbox=1 and no workload or layered root. Its sparse placeholder image is at most --image-capacity-mib (1024 by default), and its dormant scratch is --scratch-mib (64 by default). --processors (1) and --memory-mib (256) select boot resources; --processor-capacity and --memory-capacity-mib may reserve larger capacities. RAM capacity must be a multiple of 128 MiB. --cpu-profile selects the OpenVMM profile.

The VM has no NIC unless --net IPV4/PREFIX gives it one, at that static address, as for run: the NIC has the complete deny-all policy, without egress, ingress, host-loopback access, forwards, a proxy, or a DNS server, so the guest learns no DNS server and announces no listeners, and nothing that the snapshot holds depends on the policy. The snapshot keeps the NIC and its address. --net is the only way to give a snapshot's restores a NIC: each restore takes its NIC from the snapshot and refuses --net, so it cannot give the NIC another address, and installs a network policy of its own on it, as snapshot restore and run describe.

Only a successful capture whose own VMM exits within --timeout (60 whole seconds by default) can publish, and prints Snapshot captured. The immutable named directory contains exactly key.bin, snapshot.json and openvmm/; an existing name is never replaced. Capture's own VM directory is then removed. A timed-out VM is stopped but its capture is never finished. Other control or launch-record failures also stop the command's verified VM without finishing it. The next capture command using this store ends and removes an interrupted attempt through ordinary launch recovery, without resuming or publishing it. Indeterminate recovery retains its evidence and reports an error. Unknown files or nonempty unjournaled capture-slot contents are kept for inspection, not deleted; an empty launch lock alone is harmless. Known host and OpenVMM atomic staging files are recovered only with a valid launch journal. Captures in one store serialize, and one that waits for another longer than --timeout fails with sandbox-busy; snapshot locks remain separate from their immutable directories. snapshot restore starts managed sandboxes from them.

snapshot restore

nvx/target/release/nvx snapshot restore baseline \
    --store-dir /private/snapshots --state-dir /private/sandboxes/restored \
    --layer distro,/images/distro.erofs,UUID --upper ram --hypervisor kvm

snapshot restore provisions a managed sandbox in --state-dir with the options of sandbox provision, but for those that only an execution or a cold boot takes, --cwd, --environment, --environment-file, --inherit-default-environment, --exec-timeout-ms, --wait, --outcome-report, --cmdline, --dry-run, and --net, then starts it from the named snapshot of --store-dir. The sandbox's configuration records the snapshot, its store, and the sandbox's workload, --entrypoint with its --arg values, and each later sandbox start of the sandbox restores the snapshot again. Each restore runs the workload over the control protocol, as run and exec do, not from the kernel command line, so its entrypoint and arguments may contain whitespace and double quotes, which sandbox provision refuses; the command checks them as exec checks its workload before it provisions anything. The state directory must be outside the store. The guest of a restore accepts only a fresh scratch image, whose upper no earlier root wrote, such as a new empty ext4 image, so a later start of a restored sandbox with a scratch image refuses once an earlier start has created the sandbox's view of the snapshot, as each start does before its guest can use the image: deprovision the sandbox and restore the snapshot again with a fresh one. A missing snapshot provisions nothing; a sandbox whose start fails stays provisioned, as after sandbox provision.

A restore attaches the sandbox's image and upper in place of the snapshot's placeholder image and dormant scratch: the image must fit the placeholder, with its block sizes, and a scratch upper must be as long as the dormant scratch. The VM keeps the processors that the snapshot had online, and --memory-mib must be a multiple of 128 between the snapshot's boot memory and its capacity. The snapshot's key records the runtime identity that captured it: the OpenVMM, kernel, and Alpine initramfs of the artifacts, and the versions of the CLI's contracts with its guests. A restore under another one, as after an upgrade that installs new artifacts, is refused, and the snapshot must be captured again; see Upgrades.

The sandbox's NIC comes from the snapshot, with its presence and its address: --net belongs to snapshot capture, which gives a snapshot its NIC, and a restore takes no --net. A restore of a snapshot without a NIC refuses the network options before it provisions anything. That of a snapshot with a NIC needs --network-profile, and installs the policy of its network options, at the NIC's address, as the restore's own network policy, in place of the capture's deny-all. OpenVMM installs its egress, ingress, host-loopback access, and forwards, whose host ports it binds, before any vCPU runs. The guest applies the rest as it activates, before it reports Ready, so before the host sends it the mapping table of the sandbox's shares and starts the workload: it names in /etc/resolv.conf the DNS server that a cold boot under the policy would name, if any; it starts announcing its listeners when the policy publishes them, through private-network ingress or explicit host-loopback access; and it gives the proxy of a proxy-only policy, as --network-proxy names it, to the workloads of the sandbox as http_proxy, https_proxy, HTTP_PROXY, and HTTPS_PROXY in their default environment. A guest that cannot apply them ends its VM, which fails the restore. Each start of the sandbox installs the policy again.

Each start holds the snapshot's exclusive lock, which it waits for up to --timeout before it fails with sandbox-busy, from before it links the snapshot's files into the sandbox's view of it, openvmm/ in the state directory, until the start ends or the VM has exited, and sandbox deprovision holds it while it removes the view, so it needs the snapshot's store: concurrent restores of one snapshot start one at a time. Once the guest is ready, it maps the sandbox's shares and runs the workload, whose standard input ends at once and whose output the command forwards as it comes, and waits for it to exit with status 0 within --timeout, as the default workload, /bin/sh, does at once; any other end fails the start, which ends the VM. The sandbox then runs as a started one does, for sandbox exec and sandbox stop. A start journals its launch as a cold start does, so a restore whose command dies before the guest is ready never completes, and the next command of the sandbox ends its VM. run --restore-snapshot restores a snapshot once, in a sandbox that it removes.

A restore that fails before its guest is ready, in snapshot restore, a later sandbox start, or run --restore-snapshot, fails with the restore's evidence and what it attributes the failure to: the request, the host, the snapshot, the guest's repair, or nothing that it establishes. A guest that cannot activate the restore, such as one that cannot mount its scratch image as its upper, ends its VM with the status of the failure's class, which nvx::protocol::status defines, and the command reports the guest's exit with that evidence, such as OpenVMM exited during startup: guest-exit status 54; restore failed during Connect: UnusableRoot, attributed to the request.

download

nvx/target/release/nvx download
    [--repository OWNER/REPOSITORY]
    [--hypervisor {auto,whp,kvm,mshv}]
Option Default Description
--repository OWNER/REPOSITORY microsoft/nvx GitHub repository from which to download the release.
--hypervisor {auto,whp,kvm,mshv} auto Select the release platform. auto chooses WHP on Windows and KVM on Linux.

download installs a release into the repository checkout of the artifact root, which it refuses to be an extracted release. It first looks for the selected platform's package in the release that CI built from the checked-out commit. When that package exists, the packaged artifacts match the checkout. CI publishes a development release tagged v<VERSION>-dev.<first 12 characters of the commit> for each dev commit whose checks pass. When HEAD has no release, as with CI's [skip ci] baseline commits, commits whose checks failed, and local commits, or its release has no package for the selected platform, download uses the nearest first-parent ancestor that has one. Those fallback artifacts match that ancestor, not HEAD. It checks at most 20 commits, at one GitHub API request each, and reads the history with git. Update the checkout to install a newer release.

download checks the size of the package, the names of its members, and every file against the package's SHA256SUMS, and then installs it: it replaces the packaged guest artifacts under build/, among them the sandbox scratch template, build/ubuntu-smoke-scratch.ext4, that the sandbox scenarios of test-microvm copy, removes any known guest artifact that the release does not declare, such as the Azure Linux guest that source-inclusive packages omit, or the template that releases built before it shipped lack, and installs the packaged OpenVMM at openvmm/target/release/openvmm[.exe], executable, with the build provenance of the artifacts in build/. It installs everything or, if a step fails, nothing, and fails on a directory where the release has a file rather than delete what the directory holds. It leaves the checkout's nvx alone, which Cargo builds; see nvx. It warns when the release declares other OpenVMM contracts than the nvx that installs it speaks, such as another microVM ABI, time ABI, restore packet, or control contract, as a fallback release may, and names each field of its manifest that differs.

run, sandbox run, and sandbox start install the release of the checkout themselves, before they launch, when an artifact that they run is missing from a Git checkout that holds none of the files that a release installs: OpenVMM, the guest artifacts, and their build provenance. A command whose options they refuse downloads nothing. They report the download on standard error, so their output stays their own. They install no release over a checkout that holds some of those files, which may be local builds: they warn instead, and the missing artifact fails them. A file of a release that appears while they download, as from a local build, cancels their installation with a warning, so that none of the release is installed, and any file that it already installed and the build has since rewritten or replaced stays the build's. Set NVX_AUTO_DOWNLOAD=0 to turn this off, as CI does where it tests the artifacts that it built.

One download at a time installs into a checkout: it holds the lock of build/.nvx-download.lock while it searches for, downloads, and installs a release, and another waits for it. run, sandbox run, and sandbox start share that lock, creating its file if needed, from the moment they check the artifacts of a checkout, or sandbox run reads its distro layer, which may be the release's, until they have launched OpenVMM on them, and sandbox start until its VM is ready. They thus never see part of a release, and no download replaces the files that they read. sandbox provision shares the lock while it reads its distro layer and records the sandbox, and needs no artifact root otherwise. sandbox exec, stop, and deprovision share the lock too while they read the runtime identity of the artifacts of a sandbox that started, and download nothing. A run that waited for a download downloads nothing if the artifacts that it runs are then in place. With NVX_AUTO_DOWNLOAD=0, commands take no lock.

Windows release downloads support WHP. Linux release downloads support KVM and MSHV. download runs the host's curl and tar, those of %SystemRoot%\System32 on Windows 10 and later, and curl honors the usual proxy environment variables. download first uses GH_TOKEN or GITHUB_TOKEN when configured, which curl receives on its standard input and sends to the GitHub API alone, never to the host that serves the package. If GitHub rejects that token with HTTP 401 or 403, NVX reports the failure and retries without credentials so public releases remain downloadable. Private repositories require a token with read access to the repository contents that is authorized for the organization when it enforces single sign-on.

Commands

Command Description
init Initialize the OpenVMM submodule and its nested submodules.
build-guest Build the Linux kernel and selected guest artifacts.
check-guest-binaries Lint and test the Rust guest executables for musl.
build-guest-binaries Build and digest the pinned Rust executables for Linux guests.
test-guest-binaries Run the musl integration tests against prebuilt Rust guest executables.
build-kernel Build the pinned and patched Linux kernel natively.
kernel-config Generate kernel/config-microvm and its derived record from the kernel feature inventory.
check-kernel-config Check kernel/config-microvm and its derived record against the kernel feature inventory.
build-initramfs Build the selected Alpine, Ubuntu, or Azure Linux initramfs.
build-distro-layer Build a deterministic Ubuntu EROFS distro layer.
build-scratch-template Format the reproducible ext4 sandbox scratch template.
verify-guest-determinism Rebuild Ubuntu guest artifacts twice and compare SHA-256 values.
build-openvmm Build the OpenVMM release binary.
record-openvmm-provenance Bind an existing OpenVMM binary to the pinned source revision.
record-nvx-provenance Bind an existing nvx executable to its sources, Rust release, target, and features.
materialize-kernel-provenance-inputs Write kernel provenance inputs from raw run-head blobs.
setup-cross-os-cache Install GNU tar and zstd for GitHub Actions cross-OS caches.
classify-ci-changes Select the CI suites that a list of changed files runs.
kernel-patch-identity Print the digest of the kernel patch paths and contents that keys the kernel cache.
guest-binary-source-identity Print the digest of the Rust guest binary source paths and contents that keys the guest image caches.
record-guest-artifacts Record the cache keys and the SHA-256 of each file of guest artifact components, which a merge group or a dev push checks before it reuses them.
reuse-guest-artifacts Install guest artifact components that missed the cache from the latest same-repository pull request run for the merged head.
check-required-ci Validate required GitHub Actions job results.
report-nightly-ci Open, update, or close the report of failed nightly CI runs.
cancel-superseded-merge-groups Cancel the CI runs of the merge groups that the merge queue replaced.
persist-recorded-performance Persist the performance results that the CI runs of dev pushes recorded, in dev's first-parent order.
check-openvmm-ci Check that OpenVMM's own CI passed the pinned OpenVMM revision.
check-static-elf Check that executables are static ELF files without an interpreter or an executable stack.
test-openvmm-unit Run the OpenVMM workspace unit and documentation tests.
test-openvmm Run OpenVMM Petri VMM tests.
test-microvm Run NVX Linux and device correctness tests through OpenVMM.
plan-microvm-shards Split the default microVM suite into shards of balanced scenario durations for CI.
doctor Qualify this host for the NVX time ABI.
test-adversarial Run a brokered Copilot-driven adversarial campaign.
build Build the guest artifacts and OpenVMM.
benchmark Run the OpenVMM-native benchmark coordinator.
performance Collect, gate, and persist CI performance results, and check benchmark hosts.
collect-sources Materialize verified Linux, Alpine, and Ubuntu release sources.
collect-alpine-sources Collect exact Alpine recipes and upstream sources.
collect-ubuntu-sources Collect exact Ubuntu source packages.
create-linux-source-archive Create a Linux corresponding-source archive.
package Stage a binary distribution.
archive-release Create a deterministic archive from a staged distribution.
verify Verify source and submodule inputs.

Ctrl-C, that is SIGINT, interrupts a command, and so does SIGTERM outside Windows, which is how CI ends a cancelled step (see Cancellation). The command then runs its cleanup, as test-microvm and benchmark stop their OpenVMM processes and remove their scratch directories, prints Interrupted, and exits with status 130. On Linux, the OpenVMM processes that they start, and the process trees that they contain, end with nvx.py even when something kills it outright.

Initialization and verification

init

python3 scripts/nvx.py init

Initializes and recursively updates the OpenVMM Git submodule.

verify

python3 scripts/nvx.py verify

Verifies the repository's pinned source inputs and submodule state, and checks the kernel configuration against its feature inventory as check-kernel-config does.

setup-cross-os-cache

python3 scripts/nvx.py setup-cross-os-cache

Installs the GNU tar and zstd tools used by GitHub Actions cross-OS caches.

kernel-patch-identity

python3 scripts/nvx.py kernel-patch-identity [--root PATH]

Prints a digest of each file under kernel/patches/ by relative path, in path order, together with its content. The guest build appends it to both kernel cache keys, because hashFiles hashes contents only: a renamed patch changes the key even when its bytes do not.

guest-binary-source-identity

python3 scripts/nvx.py guest-binary-source-identity [--root PATH]

Prints the source digest that guest package manifests record for the Rust guest binaries: a digest of the relative path and content of every file that their build reads. Those are the crate's manifest and lockfile, its Cargo configuration under nvx/.cargo/, every file under nvx/src/ except the host side and the test modules, rust-toolchain.toml, and the build_constants.py and guest_binaries.py build modules. The host side is the host and testing modules and the nvx and nvx-fake-openvmm binaries; the test modules are the tests.rs files and the tests directories. GuestRustBuildConstants in scripts/nvx_tools/build_constants.py defines the set, and build-guest-binaries fails when Cargo's dep-info lists a file outside it. The Alpine, Ubuntu, and Azure Linux cache keys end with the digest and hash no other file of nvx, so a change to the crate's host side or tests rebuilds no guest image. Because the digest covers paths, and hashFiles hashes contents only, moving a module into its own mod.rs changes the keys even when no bytes change.

record-guest-artifacts

python3 scripts/nvx.py record-guest-artifacts --components COMPONENTS
    --keys JSON [--directory PATH]

Writes the guest artifact record, guest-artifacts.json, into the directory of the guest artifacts, build/ by default. --components lists, separated by spaces, some of the kernel, alpine, ubuntu, and azurelinux components of the CI guest build, whose ubuntu component also holds the sandbox scratch template; the CI debug kernel is in no record. The record holds the exact cache keys of those components' cache entries, from the JSON object of keys by name that --keys gives, and the SHA-256 of each of their files. It fails if a key or a file is missing. Each CI job that builds guest images records them, and uploads the record beside the components' own artifacts, so that the merge group and the dev push that merge the pull request can check them before they reuse them; see Guest artifact reuse.

reuse-guest-artifacts

python3 scripts/nvx.py reuse-guest-artifacts
    --repository OWNER/REPOSITORY
    --workflow-ref REF
    --record-artifact NAME
    --components COMPONENTS
    --keys JSON
    (--commit SHA | --head-sha SHA)
    [--dry-run]
    [--destination PATH]

When the destination, build/ by default, lacks a file of a component that --components lists, as after a cache miss, replaces every file of those components in it with the one from the artifacts of the latest pull_request run of the workflow that --workflow-ref, a GITHUB_WORKFLOW_REF value, names, for the pull request head: the second parent of the merge commit --commit, or --head-sha. The run's head must be in --repository itself, never a fork. The command first downloads the run's --record-artifact artifact, which holds the record that record-guest-artifacts wrote, and goes on only if the record gives exactly the cache keys of the components' cache entries in --keys, a JSON object of keys by name. It then downloads the artifacts in which CI hands over the components' files, guest-kernel, guest-alpine, guest-ubuntu, guest-ubuntu-distro, or guest-azurelinux, and installs the files only when no artifact has expired, each matches the SHA-256 digest that GitHub recorded for it, every file matches its SHA-256 in the record, and the kernel and initramfs provenance and the Ubuntu and Azure Linux manifests match the files and this checkout's sources. A failed installation puts back the files that it replaced. With --dry-run, it checks the record and the artifacts in the same way, even when the destination lacks no file, but installs nothing.

It reads the GitHub token from GH_TOKEN or GITHUB_TOKEN and gives up on the artifacts after two minutes, which a timer enforces on Linux until the installation starts. It never fails for a reason not to reuse, so the guest build proceeds: it prints the reason, as a GitHub notice when reuse is not possible, such as for other keys or an expired artifact, and as a warning for a fault, such as a failed request or a mismatched file, sets the reused step output to true or false, and adds the outcome to the job summary; see Guest artifact reuse.

classify-ci-changes

python3 scripts/nvx.py classify-ci-changes
    --event-name {pull_request,merge_group,push}
    --changed-files PATH
    [--null]
    [--summary PATH]

Reads one changed repository path per line by default, or NUL-delimited paths with --null, and prints the change detector's run-guest-artifacts, run-openvmm-binaries, run-openvmm-ci-check, run-microvm-tests, and run-benchmarks outputs as name=true or name=false lines. Documentation selects no suite. Pull requests, merge groups, and pushes run each test suite only when one of its own inputs changed, and check OpenVMM's own CI only when the OpenVMM pin or the code that checks it changed. A pull request or merge group runs the platform benchmarks and package jobs only when one of their inputs changed, but a push runs them for every change but documentation, because it publishes the development release and records the performance history. The guest artifacts and the OpenVMM binaries are built exactly when a job that uses them runs. The summary file receives a Markdown explanation of each decision. CI runs the full matrix instead when the command fails or omits an output; see Change classification.

check-required-ci

python3 scripts/nvx.py check-required-ci
    --event-name {pull_request,merge_group,push,schedule,workflow_dispatch}
    --same-repository {false,true}
    --run-guest-artifacts VALUE
    --run-openvmm-binaries VALUE
    --run-openvmm-ci-check VALUE
    --run-microvm-tests VALUE
    --run-benchmarks VALUE

Validates the GitHub Actions result values supplied through the QUALITY_RESULT, CHANGES_RESULT, and job-specific *_RESULT environment variables. CI supplies the true or false suite flags that classify-ci-changes reports. The command expects successful results for jobs enabled by the event, repository, and suite flags, and skipped for jobs that are not enabled; mismatches are reported as errors and cause a nonzero exit. As in CI, a job that uses the guest artifacts or the OpenVMM binaries is enabled only when the flags enable the jobs that build them too. A merge group expects the jobs of a same-repository pull request, including the nvx builds and the platform and package jobs when their suite runs, whatever --same-repository says, but not the rehearsal of the guest artifact reuse, which its pull request's run made. A pull request runs the debug kernel on MSHV, a merge group on KVM, and a push or full run on every backend; see Merge queue.

report-nightly-ci

python3 scripts/nvx.py report-nightly-ci
    --repository OWNER/REPOSITORY
    --run-id ID
    [--run-attempt N]
    [--assignee LOGIN]...
    [--label NAME]...
    [--summary PATH]
    [--dry-run]

Reports a completed scheduled run of the CI workflow, by default its latest attempt, in the repository's nightly failure report: a failing run opens or updates the report, and a passing run closes it, as Nightly failure reports describes. The command prints what it did. It reads a token from GH_TOKEN or GITHUB_TOKEN, which must be able to read the run's jobs, logs, and check annotations, and to read and write issues. A new report is assigned to every --assignee and names them as its triage owners, and carries every --label that exists in the repository; the command never creates a label. --summary appends the report to a Markdown file, such as $GITHUB_STEP_SUMMARY. --dry-run prints the changes that the command would make, with their Markdown, and makes none; it accepts a run of any event, so that a past failure can preview its report, and, for a public repository, needs a token only for the job logs.

cancel-superseded-merge-groups

python3 scripts/nvx.py cancel-superseded-merge-groups
    --repository OWNER/REPOSITORY
    --workflow-ref REF
    --run-id ID
    --ref REF
    [--summary PATH]

Cancels the unfinished runs, whether they run or wait, of the workflow that --workflow-ref, a GITHUB_WORKFLOW_REF value, names, for the earlier merge groups of the pull request of the merge group that --ref names, a refs/heads/gh-readonly-queue/BASE/pr-NUMBER-SHA ref, as Superseded merge groups describes. It cancels only the runs of the groups of the same base branch and pull request whose branches GitHub has deleted or given another head, as when the pull request returns to the queue on the same base, so that it never cancels the run of the pull request's live group. It holds the run --run-id, its caller, to the same test, but cancels it last: GitHub can start a replaced group's run late, after the run of the group that replaced it looked. It reads a token from GH_TOKEN or GITHUB_TOKEN, which must be able to read and cancel the repository's workflow runs and read its branches, and prints each run that it cancelled, that had completed meanwhile, or that it could not cancel, which doesn't stop the others. --summary appends the same lines to a Markdown file, such as $GITHUB_STEP_SUMMARY. A failed request warns, as a GitHub warning, but never fails the command, which only frees runners; a missing token or a --ref that names no merge group does.

persist-recorded-performance

python3 scripts/nvx.py persist-recorded-performance
    --repository OWNER/REPOSITORY
    --workflow-ref REF
    --download-dir DIR
    [--branch BRANCH]
    [--history-dir DIR]
    [--require-host]
    [--summary PATH]
    [--message PATH]

Appends to the performance history in --history-dir, data by default, the results that the completed runs of the workflow that --workflow-ref, a GITHUB_WORKFLOW_REF value, names recorded for pushes to --branch, dev by default, in their performance-results-RUN_ID artifacts, as Development releases and performance history describes. The history must lie in a git checkout of the branch. The command takes the runs of the commits of HEAD's first-parent history that are newer than the newest commit that the history holds, oldest first, and persists each run's results as performance persist does, whole or not at all, once they match their artifact's digest, come from the run's own commit, and, with --require-host, come from the host that the history expected at that commit. It leaves out the results of a platform whose expected host changed in a required field after the run's commit, as a reset of the platform's series does, and skips a run whose results expired. It reads the runs a day of their creation at a time, back to the oldest that GitHub could have rerun with results that remain. It keeps the downloaded results in --download-dir, so that another attempt, after a push of the history failed, downloads nothing again. It refuses the results of a run that fail a check, and persists the later runs, but stops at a run whose results it can't fetch, as when GitHub fails or a download breaks off, so that a later attempt persists that run and the ones after it, in order. It persists nothing when it can't list the runs, as when a day holds more runs than GitHub lists for a search. It reads a token from GH_TOKEN or GITHUB_TOKEN, which must be able to read the repository's workflow runs and download their artifacts. It prints what it did with each run, --summary writes the outcome to a Markdown file, and --message writes the message of the commit of the persisted results to a file, when the command persisted any. It exits with status 1 when it refused a run's results or stopped, after persisting the others.

check-openvmm-ci

python3 scripts/nvx.py check-openvmm-ci
    [--revision SHA]
    [--wait-minutes MINUTES]
    [--summary PATH]

Checks that OpenVMM's own CI passed the OpenVMM revision that the checkout's index pins, or the full commit SHA that --revision names, as OpenVMM CI describes. It reads the runs of the OpenVMM PR and OpenVMM CI workflows of microsoft/openvmm whose head is the revision, with the latest attempt of each of their jobs, and requires each job that covers NVX's use of OpenVMM, the x86-64 Linux and Windows unit tests and the VMM tests on KVM, MSHV, and Windows, to have passed in at least one of those runs. It ignores the runs of a pull request from a fork, which run the fork's copy of the workflows, and the workflows' other jobs. A job is pending while a run that has not finished may still pass it, failed when every run that ran it finished it without passing, and missing when no run will run it. With --wait-minutes, the command looks again every five minutes while a job is pending and none failed or is missing, for at most that long; by default it looks once. While it may still wait, it also looks again, with a GitHub warning, after a look that failed, as when GitHub's API is briefly unavailable, but a third failure in a row ends it. It prints each job's verdict with the job that decided it, appends the verdicts as a table to the Markdown file that --summary names, such as $GITHUB_STEP_SUMMARY, and fails, with a GitHub error for each, unless every job passed. It reads a token from GH_TOKEN or GITHUB_TOKEN when one is set; the repository is public, so it needs none, but GitHub allows fewer requests without one.

check-static-elf

python3 scripts/nvx.py check-static-elf PATH [PATH ...]

Prints each executable's ELF type, size, and linkage, and fails unless every one is a 64-bit little-endian ELF executable without a program interpreter (PT_INTERP) or needed libraries (DT_NEEDED), so that it runs in a guest without a dynamic loader. A static position-independent executable passes. It also fails for an executable stack: a PT_GNU_STACK program header with PF_X. The nvx CI job checks the guest binaries with it.

check-guest-binaries

python3 scripts/nvx.py check-guest-binaries [--toolchain TOOLCHAIN]

The Linux-only check command adds the x86_64-unknown-linux-musl target to the Rust toolchain that --toolchain names, by default the exact rust-toolchain.toml pin. It then runs Clippy with warnings denied on every target of the guest feature, and the feature's musl tests, whose static test executables run on the host. It ignores an inherited NVX_GUEST_BINARIES, so that the integration tests run the binaries that Cargo builds from the checkout. The nvx CI job runs it with the toolchain that its action selects, before it builds the binaries.

build-guest-binaries

python3 scripts/nvx.py build-guest-binaries --output DIRECTORY

The Linux-only build command checks the exact rust-toolchain.toml pin and builds the ten Rust guest executables for x86_64-unknown-linux-musl with one cargo build of the guest profile, a fresh target directory, Cargo's locked mode, incremental compilation disabled, inherited Cargo profile overrides removed, remapped source paths, and no linker build IDs. It then fails unless the source identity, which guest-binary-source-identity prints and the guest image cache keys end with, covers every file that Cargo's dep-info lists for the executables: each source file that rustc read, including the targets of include_str! and include_bytes!, and each path that the build script watches. Files that the build generates in its own target directory are exempt. It runs the static-ELF check and writes manifest.json with the source and executable SHA-256 digests. It updates the output of an earlier build in place, adding nvx-managed-agent, nvx-ingress, nvx-time, nvx-time-probe, and nvx-init when that build's manifest does not record them and removing the retired nvx-port-io executable when that build's manifest records it, and refuses any other existing DIRECTORY.

test-guest-binaries

python3 scripts/nvx.py test-guest-binaries --binaries DIRECTORY

The Linux-only test command validates the ten binaries and build manifest, sets NVX_GUEST_BINARIES, and runs the musl guest_tools integration test against that exact directory.

See Setup for host prerequisites.

materialize-kernel-provenance-inputs

python3 scripts/nvx.py materialize-kernel-provenance-inputs

Recreates the tracked kernel configuration and patch files from the current Git run head. It removes stale worktree patches that are absent from that revision and is used before packaging kernel provenance inputs.

Build commands

build-guest

python3 scripts/nvx.py build-guest
    [--guest {alpine,ubuntu,azurelinux,all}]
    [--native]
    [--debug-kernel]

By default, builds the guest kernel and initramfs with Docker. --native builds the selected artifacts directly on Linux instead. Alpine is the default. Azure Linux does not support --native and always builds through Docker. --guest all also builds the Ubuntu EROFS distro layer and the sandbox scratch template; see build-scratch-template. --debug-kernel also builds the CI debug kernel, build/vmlinux-debug; see Build.

build-kernel

python3 scripts/nvx.py build-kernel [--debug]

Fetches, verifies, patches, and builds the pinned kernel directly on Linux. --debug builds the CI debug variant instead of the production kernel. The production build also requires its configuration to be the one that the kernel feature inventory generates; see Kernel configuration.

kernel-config

python3 scripts/nvx.py kernel-config [--native] [--check]

Generates kernel/config-microvm and kernel/config-microvm.derived from the kernel feature inventory, scripts/nvx_tools/kernel_features.py: make allnoconfig with the inventory's settings, and the Kconfig rule that sets every other symbol. By default it generates them in the Docker build's image, whose toolchain builds CI's kernels; --native generates them on Linux with the host's toolchain. --check fails if the checked-in files differ instead of writing them. See Kernel configuration.

check-kernel-config

python3 scripts/nvx.py check-kernel-config

Checks, without a kernel source, that every setting of the kernel feature inventory holds in kernel/config-microvm and that every other symbol that it sets is recorded, with the same value, in kernel/config-microvm.derived. A newly enabled symbol that neither accounts for fails the check, which CI's quality job runs on every change and which verify runs too.

build-initramfs

python3 scripts/nvx.py build-initramfs [--guest {alpine,ubuntu,azurelinux}]

Builds the selected initramfs. Alpine and Ubuntu build directly on Linux; Azure Linux uses Docker. Alpine is the default.

build-distro-layer

python3 scripts/nvx.py build-distro-layer
    --guest ubuntu
    [--output PATH]
    [--replace]

Builds the immutable Ubuntu EROFS distro layer directly on Linux. The default output is build/ubuntu-distro.erofs. Existing output or manifest files are rejected unless --replace is present.

build-scratch-template

python3 scripts/nvx.py build-scratch-template
    [--output PATH]
    [--replace]

Formats the sandbox scratch template directly on Linux, with mkfs.ext4 from e2fsprogs, which it finds on PATH or else in /usr/sbin or /sbin, where Debian keeps it out of an unprivileged user's PATH: an empty 64 MiB ext4 file system, of which the sandbox scenarios boot a copy as their scratch. The default output is build/ubuntu-smoke-scratch.ext4. An existing output is rejected unless --replace is present. UbuntuBuildConstants in build_constants.py defines the template once: its name, size, UUID, block size, inode size and count, and features, which the command passes to mkfs.ext4 rather than leave them to mke2fs.conf and the defaults that distributions patch. The command also fixes the clock of mkfs.ext4 with E2FSPROGS_FAKE_TIME and its directory hash seed, and ignores the caller's other E2FSPROGS_* and MKE2FS_* variables, so e2fsprogs 1.46.5 and 1.47.0 of Debian 12, Ubuntu 22.04 and 24.04, and Azure Linux 3 format the same bytes, whose SHA-256 UbuntuBuildConstants.SCRATCH_TEMPLATE_SHA256 pins. It warns when mkfs.ext4 formats other bytes. The guest Docker build formats the template with the Ubuntu distro layer, in the pinned Debian image, for build-guest and for CI, which caches, records, and hands it over with the layer. On other systems than Linux, the command fails and names that build of every guest and download, which installs the template.

verify-guest-determinism

python3 scripts/nvx.py verify-guest-determinism
    --guest ubuntu
    [--work-dir PATH]

Currently supports only --guest ubuntu. It builds the Ubuntu initramfs, EROFS layer, and sandbox scratch template twice from separate roots and compares every artifact SHA-256, and fails unless the template's is the one that UbuntuBuildConstants.SCRATCH_TEMPLATE_SHA256 pins. A mismatch reports the first differing normalized rootfs entry when one exists.

build-openvmm

python3 scripts/nvx.py build-openvmm [--skip-restore] [--backend {kvm,mshv,whp}]

Builds the openvmm release binary. Before building, the command runs cargo xflowey restore-packages; use --skip-restore when those packages are already restored. Without --backend, Windows builds the native MSVC target and Linux builds the native GNU target. On Linux, --backend kvm selects GNU and --backend mshv selects musl; Windows accepts --backend whp. Build-target selection does not probe /dev/kvm or /dev/mshv, so compilation also works on build-only hosts and hosts exposing both devices. Unsupported OS/backend combinations are rejected.

record-openvmm-provenance

python3 scripts/nvx.py record-openvmm-provenance

Records the checked-out OpenVMM revision, clean-state flag, and executable SHA-256 for the existing openvmm/target/release/openvmm[.exe] binary in build/openvmm.provenance.json. The command takes no options. It requires the OpenVMM submodule to be checked out/initialized and the release binary to already exist at openvmm/target/release/openvmm on POSIX or openvmm/target/release/openvmm.exe on Windows. The command fails if either prerequisite is missing.

record-nvx-provenance

python3 scripts/nvx.py record-nvx-provenance [--target TARGET]
Option Default Description
--target TARGET the host target that rustc -vV reports Rust target that built nvx.

Records build/nvx.provenance.json for the nvx that the checkout runs without building it, the one that NVX names or else nvx/target/release/nvx[.exe]; see nvx. The record binds the executable's SHA-256 to the digest of this checkout's nvx sources, the Rust release that rust-toolchain.toml pins, the target, and the cli feature, so record it right after building nvx from this checkout. For x86_64-unknown-linux-musl, the target of the Linux release packages, the executable must be static. CI's nvx build jobs record the builds that they hand to the packages, and nvx.py records the builds that it makes itself.

build

python3 scripts/nvx.py build
    [--guest {alpine,ubuntu,azurelinux,all}]
    [--native]
    [--debug-kernel]
    [--skip-restore]
    [--backend {kvm,mshv,whp}]

Runs build-guest followed by build-openvmm. The options have the same meaning as on those individual commands.

See Build for dependencies, outputs, and native build details.

Test commands

test-openvmm-unit

python3 scripts/nvx.py test-openvmm-unit

Runs the OpenVMM workspace's unit-test binaries with cargo-nextest's agent profile and the ci feature. Packages that require specialized test harnesses are excluded, along with all fuzz crates reported by OpenVMM's xtask. Afterward, runs the workspace doctests with Cargo. On Windows, it appends /Brepro to _LINK_, so that MSVC links proc macros reproducibly and sccache can reuse the crates that use them; see Compilation cache.

test-openvmm

python3 scripts/nvx.py test-openvmm
    --backend {kvm,mshv,whp}

Builds and runs the OpenVMM checkout's VMM tests that the backend's filter selects, with OpenVMM's vmm-tests-run. The microVM TTRPC tests boot NVX's build/vmlinux and build/initramfs.cpio.gz, the Linux-direct lifecycle and snapshot test at one, two, four, and eight processors, and the device snapshot test at one, so build the guest first; the remaining test artifacts are produced by OpenVMM itself. On Windows, it appends /Brepro to _LINK_, as test-openvmm-unit does. No WHP test needs Hyper-V, so vmm-tests-run, which derives what a run needs from the tests that it selects, needs no Hyper-V administration on a WHP host.

CI runs neither this command nor test-openvmm-unit: OpenVMM's own CI runs OpenVMM's tests, and CI checks that it passed the pinned revision with check-openvmm-ci; see OpenVMM CI.

test-microvm

python3 scripts/nvx.py test-microvm
    --backend {kvm,mshv,whp}
    [--guest {alpine,ubuntu,azurelinux}]
    [--scenario SCENARIO]...
    [--debug-kernel]
    [--shard INDEX/COUNT]
    [--processors {1,2,4,8} ...]
    [--memory-mib MIB]
    [--timeout SECONDS]
    [--output-dir PATH]

Runs NVX-owned Linux, SMP, virtio, sandbox, and snapshot correctness scenarios against the public OpenVMM CLI. Repeat --scenario to select a subset; without it, every scenario supported by the selected guest runs, except smp-lapic, which runs only when named: it repeats smp and also asserts that every CPU uses the one-shot counting LAPIC. managed-exec-config, sandbox-lifecycle, sandbox-proxy-only, sandbox-ram-upper, and sandbox-gpt-layer, the public sandbox scenarios, drive the public nvx sandbox lifecycle commands (see Run), snapshot-core, snapshot-tiers, and untiered-network-policy capture and restore named snapshots, the last with a NIC, with nvx snapshot and nvx run (see Run), and vm-lifecycle drives a VM through the public API of nvx; they also read the Ubuntu distro layer, its manifest, and the sandbox scratch template, which download installs and the guest-artifact build makes; a run that selects one of them fails before its first scenario when one of those files is missing. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only interrupt-map, sandbox-blocks, scratch-snapshot, and virtio-features scenarios and the sandbox-control-dependent snapshot-core, snapshot-tiers, platform-restore-processors, platform-zero-fill, platform-filesystem, platform-aggregate-export, platform-network-policy, managed-exec-config, sandbox-lifecycle, sandbox-proxy-only, sandbox-ram-upper, sandbox-gpt-layer, vm-lifecycle, and untiered-network-policy scenarios. Ubuntu also rejects the Alpine-prompt-specific console-snapshot scenario. --debug-kernel boots the CI debug kernel, build/vmlinux-debug, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without --scenario, it runs only the same-host restore scenarios smp, smp-snapshot, restore-processors, and restore-downtime. --shard INDEX/COUNT, such as 1/2, runs shard INDEX of the COUNT shards of the default Alpine suite on the default kernel, the scenarios of a run without --scenario but the public sandbox scenarios, which CI runs in its guest and sandbox jobs, as microvm_shards.json splits it for the backend; CI runs each shard in a job of its own (see MicroVM test shards). It takes no --scenario, --debug-kernel, or other --guest, and fails before it runs a scenario unless that file puts each default scenario in exactly one shard of every backend and splits the backend's suite into COUNT shards. The cpu-profile-fallback scenario cold boots with OpenVMM's host-cpu-unknown test hook, which makes OpenVMM's default CPU profile selection, auto, fall back to a host profile on a host that a built-in profile serves, checks the fallback's warning, captures the guest, and restores it on the snapshot's host profile. The console-snapshot scenario first checks that OpenVMM refuses a boot console on TCP, as a listener or a client, with a network and without one, since a guest reaches host loopback through its gateway (see Host-loopback access); it then captures a guest whose boot console listens on a Unix socket beside the snapshot, or on Windows on a named pipe, and restores it twice. The time-host-correction scenario reads the host's own clock correction from a time sample of an unhooked cold boot, then cold boots with OpenVMM's host-rate-offset-ppm test hook. The harness composes the host and hook rates as OpenVMM does, checks Linux's correction after the TSC rate's kHz rounding, clears an earlier poll's PLL phase offset through the nvx-time binary that every guest ships, waits out Linux's cached tick length, and retries a sample pair when host UTC, the accepted-sample count, or the live discipline state changes, or when a sample fails to pair or is less certain than the discipline's 100 µs bound. It checks that the guest holds the published correction and runs its clock at the hooked rate of host UTC, then restores a snapshot with another rate, and with one beyond Linux's 500 ppm limit, which the guest must hold at the limit and report. The managed-lifecycle scenario boots a guest whose init hands the boot to the managed agent, and runs the ignored nvx unit test of the host's control session, in src/host/session/tests/managed_lifecycle.rs, with Cargo, which starts OpenVMM with the scenario's command line and drives the guest's control session; the scenario then checks OpenVMM's outcome report and the guest console. The vm-lifecycle scenario runs the ignored nvx test tests/vm_lifecycle.rs with Cargo, which creates and spawns a VM of nvx-init over the Ubuntu layer and a scratch copy through the public API of nvx, reattaches to it from the values that it saved, and stops it through its guest; the scenario then checks OpenVMM's outcome report. The virtio-features scenario boots a guest with every fixed virtio device: the NIC, the dormant host filesystem slot, both consoles, and both sandbox blocks. The guest reports, for each device, the base of its MMIO window, its virtio device ID, and the features that its driver negotiated, and the scenario requires exactly the devices and features of Table 2.8, naming each bit that a device has besides its documented ones or lacks. The interrupt-map scenario boots the same devices, with a share in the host filesystem slot, and reports the interrupt that each device's driver requested before and after it drives traffic through every device; each must be the IOAPIC input of its device's window that Table 2.7 lists, edge-triggered, unshared, and firing, and /proc/interrupts must count no APIC error, misrouted IOAPIC interrupt, or spurious interrupt. A snapshot of a guest with the NIC, the share, and the boot console must then restore with the same interrupts, which fire again. The host services that the network scenarios offer the guest listen on host loopback, which the guest reaches through its gateway (see Host-loopback access), so no other host reaches them; a scenario binds every host address only to check that a port is free or that OpenVMM publishes it, and never listens or reads there. The scenarios that run microVMs or sandboxes as the public commands do, or that compile egress policies, run nvx. The command requires build/vmlinux (with --debug-kernel, build/vmlinux-debug and its build/vmlinux-debug.config), the selected initramfs, and openvmm/target/release/openvmm[.exe].

Scenario duration report

test-microvm prints each scenario's progress line as the scenario starts, so that a log's timestamps follow the scenarios. It also records how long each scenario takes in scenario-durations.jsonl in --output-dir, replacing the previous run's report, and CI uploads the reports of its microVM jobs. The report is JSON Lines: one JSON object per line, whose record field names its kind. The first line, run, describes the run:

Field Value
schema, schema_version nvx-microvm-durations and 1. The version changes only when a reader of the previous version would misread a report, not when a field is added.
started_at When the run and its setup started, in UTC, such as 2026-10-21T13:50:00.123Z
guest, backend The --guest and the --backend
kernel debug with --debug-kernel, otherwise default
processors The --processors counts, without repeats
scenarios The selected scenarios
github In GitHub Actions, the job's repository, workflow, run_id, run_attempt, job, event_name, ref, sha, runner_name, and runner_os, as strings from the GITHUB_ and RUNNER_ variables of those names; otherwise null

Each later line records a span of the run as soon as the span ends, so that a run that fails or is interrupted keeps every span that it completed. The spans follow one another without gaps, so they add up to the run's time:

  • setup, from the start of the Python process, or from the report's creation where the OS doesn't report that start, until the first scenario starts;
  • a scenario for each scenario execution, until the next one starts. console-exit, smp, and smp-lapic execute once per processor count;
  • teardown, which sums up the time ABI evidence and records the run's outcome.

Every span has its started_at, its seconds, rounded to the millisecond, its outcome, passed, failed, or interrupted, for example by SIGINT or SIGTERM, and an error: for a failure, the exception's type and message, cut to 1,000 characters, and otherwise null. A scenario record also has the scenario, the guest, backend, and kernel, and its processors: the --processors counts that the execution covers, such as [4] for smp on 4 vCPUs, the largest count for time-abi-conformance, all of them for smp-snapshot and restore-processors, those up to 4 for platform-restore-processors, or null for a scenario that doesn't take them. A failed run ends with the failed scenario's record and the teardown; a run that is killed keeps every record that it completed and has no teardown.

plan-microvm-shards

python3 scripts/nvx.py plan-microvm-shards
    [--report PATH]...
    [--seconds SCENARIO=SECONDS]...
    [--shards BACKEND=COUNT]...
    [--partition PATH]

Plans the shards of the default Alpine suite on the default kernel, which CI runs in parallel (see MicroVM test shards), and writes them to microvm_shards.json, or to --partition. The file holds each backend's median duration of every default scenario, the CI runs whose reports gave them, and each shard's scenarios with their total duration.

--report reads scenario duration reports, or every report in a directory, such as the one that gh run download fills with a run's nvx-microvm-durations-* artifacts. Each backend's duration of a scenario is the median, to a tenth of a second, of the reports of the default suite, sharded or not: those of the Alpine guest on the default kernel with the default processor counts, of runs that ended, passed or not. A report counts for a scenario only when every execution of the scenario passed, such as the four of console-exit. A scenario that no report measures keeps its duration in the file. --seconds sets a scenario's duration on every backend instead, such as an estimate of a new scenario that no report measures yet; the command fails when a default scenario has no duration.

Each backend keeps its number of shards unless --shards changes it. Each scenario starts in its shard of the file, and a scenario that has none goes, longest first, to the shard that takes the least time so far. A shard that is still empty, as one that --shards adds can be, takes the longest scenario of the longest shard that has more than one. The command then moves a scenario to another shard, or swaps two scenarios of different shards, while that shortens the longest shard by at least a second, comparing the next longest where the longest stay alike, and takes the best such change each time. It prints each backend's shards with their durations, and the scenarios that changed shard.

doctor

python3 scripts/nvx.py doctor
    --backend {kvm,mshv,whp}
    [--checks ID ...]
    [--openvmm PATH]
    [--kernel PATH]
    [--initrd PATH]
    [--cpu-fingerprint PATH | --no-openvmm]
    [--ci-schedule]
    [--probe-dir PATH]
    [--summary PATH]
    [--timeout SECONDS]

Qualifies this host for the time ABI. It runs the selected checks in order, prints one NVX-DOCTOR: check=<id> status=<pass|fail> detail="..." line per check, and exits with status 1 if any check fails. A check that passes with a warning starts its detail with warning:. A check fails, and never passes, when it can't read a fact it gates on. CI describes each check and the subset that CI runs.

Option Default Description
--backend {kvm,mshv,whp} required Select the backend to qualify.
--checks ID ... H1 to H7 Run only these checks, from H1 to H7, in order.
--openvmm PATH openvmm/target/release/openvmm[.exe] Select the OpenVMM binary for H2, H3, and H6.
--kernel PATH build/vmlinux Select the guest kernel for H3 and H6.
--initrd PATH build/initramfs.cpio.gz Select the guest initramfs for H3 and H6.
--cpu-fingerprint PATH nvx-cpu-fingerprint-<backend>.json in the probe directory Select where H2 writes OpenVMM's CPU fingerprint.
--no-openvmm off Qualify without an OpenVMM binary. H2 then checks the CPU identity and generation but not the CPU profile, and --checks must exclude H3 and H6, which boot OpenVMM.
--ci-schedule off Run H4 and H6 on CI's short schedules: 3 rate samples 1 s apart instead of 13 samples 10 s apart, and two warp-probe runs instead of five.
--probe-dir PATH $RUNNER_TOOL_CACHE/nvx-host-time-probe, or build/host-time-probe Select the cache directory for the host probe, which the doctor builds with rustc.
--summary PATH none Append a Markdown summary, for example to $GITHUB_STEP_SUMMARY.
--timeout SECONDS 120 Set the seconds allowed for each probe or guest. H4 gets its sampling window on top.

test-adversarial

python3 scripts/nvx.py test-adversarial
    --backend {kvm,mshv,whp}
    --campaign {workload-isolation,guest-isolation,snapshot-isolation}
    [--budget-seconds SECONDS]
    [--budget-actions COUNT]
    [--budget-ai-credits CREDITS]
    [--seed SEED]
    [--output-dir PATH]
    [--replay ACTIONS.jsonl]
    [--model MODEL]
    [--host-type {baremetal,virtual-machine}]
    [--memory-mib MIB]
    [--phase-timeout SECONDS]
    [--action-timeout SECONDS]
    [--executor-command PATH]
    [--no-minimize]
    [--minimize-attempts COUNT]

Runs a bounded adaptive campaign in which an already installed and authenticated Copilot CLI selects one deterministic primitive at a time. Copilot has no tools or direct NVX access. The typed broker validates and records every action, while a credential-free executor and independent watchdog own VM operation, canaries, teardown checks, and the clean post-campaign boot.

Option Default Description
--backend required Select KVM or MSHV on Linux, or WHP on Windows.
--campaign required Select workload, privileged-guest, or snapshot boundary probes.
--budget-seconds 900 Bound preflight, actions, canary boot, and minimization wall time.
--budget-actions 8 Bound accepted and executed broker actions.
--budget-ai-credits 300 Bound charged Copilot usage; the minimum is 60 credits so authentication preflight and at least one action each retain a 30-credit CLI cap.
--seed 0 Seed deterministic candidate ordering and synthetic canary data.
--output-dir backend-specific path under build/test-results Parent for a unique campaign run directory.
--replay none Replay an actions.jsonl bound to its sibling replay-manifest.json, without invoking Copilot.
--model Copilot auto routing Select the strategist model.
--host-type NVX_HOST_TYPE or unspecified Record whether the executor is bare metal or a virtual machine.
--memory-mib 256 Set memory for deterministic microVM scenarios.
--phase-timeout 60 Set each underlying deterministic scenario phase timeout.
--action-timeout 600 Set the outer limit for one action or canary boot.
--executor-command local child executor Select one trusted no-argument wrapper for a separate disposable target.
--no-minimize off Disable fresh-target shorter-prefix replay after an anomaly.
--minimize-attempts 3 Bound shorter-prefix attempts within the campaign time budget.

Missing or unauthenticated Copilot CLI is a preflight failure in adaptive mode. Replay mode has no Copilot prerequisite. Production and CI campaigns must use a separate disposable executor through --executor-command; local mode cannot reliably classify a target-host crash. Local target logs are kept under the short build/adv state root to avoid Windows path-length failures; the run summary records their absolute location. See Copilot-driven adversarial testing for the trust boundary, wrapper contract, artifacts, and CI policy.

CPU profiles

Every microVM presents a CPU profile to its guest: the complete CPUID surface of one CPU generation, which OpenVMM pins and every backend shares (see the CPU profile design). By default, OpenVMM selects the latest revision of the built-in profile of the host's CPU generation:

Profile Generation CPUs (display family/model)
intel.skylake-sp.v1 skylake-sp Intel Xeon Scalable, first generation (6/85, steppings 0 to 4)
intel.icelake-sp.v1 icelake-sp Intel Xeon Scalable, third generation (6/106)
intel.emeraldrapids.v1 emeraldrapids Intel Xeon Scalable, fifth generation (6/207)
intel.graniterapids.v1 graniterapids Intel Xeon 6 with P-cores (6/173)
intel.alderlake.v1 alderlake Intel Core, twelfth generation (6/151 and 6/154)
amd.milan.v1 milan AMD EPYC, third generation (25/1, Milan-X included)
amd.genoa.v2 genoa AMD EPYC, fourth generation (25/17, Genoa-X included)
amd.turin.v2 turin AMD EPYC, fifth generation (26/2)

OpenVMM keeps every earlier revision pinned, so that its snapshots still restore, and --cpu-profile still selects it by its ID: amd.genoa.v1 on Genoa and amd.turin.v1 on Turin.

Any other CPU, including Cascade Lake, Sapphire Rapids, Tiger Lake, Raptor Lake, and AMD's Ryzen CPUs, has no built-in profile: on an Intel or AMD CPU, --cpu-profile auto falls back to a host profile, described below, and on another vendor's CPU, it fails with E_PROFILE_HOST_UNKNOWN. A host of a listed generation can still fail with E_PROFILE_UNSUPPORTED if its SKU or hypervisor lacks a feature of the profile, and on an Intel or AMD CPU OpenVMM's error then names --cpu-profile host. intel.alderlake.v1 derives from one Core i9-12900H on WHP, intel.graniterapids.v1 from one Xeon 6973P-C Azure VM on WHP and the KVM of two GitHub-hosted Actions runners with that CPU, amd.milan.v1 from one EPYC 7763 Azure VM on WHP, amd.genoa.v1 and amd.turin.v1 from KVM on one GitHub-hosted Actions runner each, an Azure VM with an EPYC 9V74 or 9V45, amd.genoa.v2 from that EPYC 9V74 runner's KVM and the nested MSHV of an Azure VM with the same CPU, and amd.turin.v2 from the KVM, under Linux 6.8 and 7.0, and the WHP, under Windows Server 2022 and 2025, of GitHub's runners with an EPYC 9V45. GitHub's runners with an EPYC 7763, 9V74, or 9V45, or a Xeon 6973P-C, pass their profiles' full test-microvm suite on KVM, and those with an EPYC 7763, 9V74, or 9V45 pass it on the WHP of Windows Server 2025 too. Alder Lake hosts on KVM and MSHV, Granite Rapids hosts on MSHV and WHP, Milan, Genoa, and Turin hosts on MSHV, bare-metal AMD hosts, and SKUs without the profiles' features are unverified. No profile presents AMX, whose tile states OpenVMM does not enable, or AVX10, whose leaf 0x24 no profile keeps; on Granite Rapids, the AVX-512 features enumerate every instruction of AVX10.1. The maximum basic leaf of intel.graniterapids.v1 is 0x24, which KVM enumerates from Linux 6.12, so a Granite Rapids host whose KVM is older does not support it. The AMD profiles leave out what KVM cannot present, BTC_NO and PSFD without a SPEC_CTRL control, and the enumerations of Intel's speculation controls that KVM adds on AMD hosts. They present none of AMD's speculation controls, IBRS, STIBP, and SSBD, because the Azure VMs' hypervisors withhold them; amd.milan.v1's Linux guests therefore use retpolines and report SSB, SRSO, and TSA as vulnerable on every host. amd.genoa.v1 pins what Azure presents to KVM on Genoa, and amd.turin.v1 what it presented to KVM on Turin when it was derived: the basic leaves up to 0x1c, and LFENCE serialization, NoSmmCtlMsr, and the TSA immunities in 0x80000021. The Microsoft hypervisor of the Azure Genoa VMs fingerprinted so far, under MSHV and WHP, and of the Turin ones, under WHP, presents none of them. The KVM of GitHub's Turin runners no longer reports the TSA immunities, and a bare-metal Genoa host's KVM does not report them either. amd.genoa.v2 and amd.turin.v2 leave them out, so their Linux guests report TSA as vulnerable, as amd.milan.v1's do. amd.turin.v2 also leaves out the StibpAlwaysOn and IBPB_RET enumerations of 0x80000008, which qualify controls that no AMD profile presents: the WHP of Windows Server 2022 presents neither, and KVM presents IBPB_RET only from Linux 6.13.

On an Intel or AMD development host that no built-in profile serves, --cpu-profile auto falls back to a host profile, intel.host.v1 or amd.host.v1, which nvx run --cpu-profile host also selects, for example where the hypervisor does not support the built-in profile. OpenVMM fingerprints the hypervisor on this host and applies the built-in profiles' derivation policy to it, so the guest sees the same kind of filtered CPU surface, and verifies it as it verifies a built-in profile: a cold boot still fails with E_PROFILE_UNSUPPORTED if the hypervisor lacks a CPU feature that the time ABI requires. Where auto fell back, the error also says so; with an explicit --cpu-profile host, it reports the failure alone. A host profile is for development only:

  • It is not pinned: a microcode, firmware, hypervisor, or OS update can change it.
  • Each cold boot fingerprints the hypervisor first, which #394 measured at 2.6 to 5.5 ms on WHP, 11 to 23 ms on KVM, and 52 ms on MSHV. The cost varies with the host, and the fallback's warning reports what each cold boot's fingerprint took.
  • A snapshot records its host profile, and restores only on a host with the same CPU model and stepping whose hypervisor supports the profile. It also restores only if OpenVMM's derivation policy leaves the recorded profile unchanged: OpenVMM refuses a recorded profile that the policy would change (E_PROFILE_DIGEST), such as an edited one that gives the guest VMX or SVM. It applies its own policy, whatever policy the profile's provenance names, so this also refuses a snapshot of a profile that an OpenVMM with another policy derived, if its policy would change the profile; recapture such a snapshot.
  • doctor never qualifies it, so CI hosts need a built-in profile, and benchmark refuses a CPU that no built-in profile serves.

Every cold boot that falls back logs one OpenVMM warning, at the default log level, that starts with NVX-CPU-PROFILE-FALLBACK: and names the CPU, the host profile with its digest, and these limits, with what that cold boot's fingerprint took. It also asks you to help add a built-in profile for the CPU: run openvmm --hypervisor <backend> --cpu-fingerprint fingerprint.json, which reports E_PROFILE_HOST_UNKNOWN on such a CPU but still writes the fingerprint, and attach fingerprint.json to a CPU profile request, whose link in the warning prefills the CPU. The fingerprint holds the CPU's identity and microcode, the OS kind and build, the hypervisor's interface, and the CPUID, XSAVE, MSR, and time surface that the backend supports; it holds no hostname, user name, or serial number. Fingerprints of the generation from other backends help too, because a profile is the intersection of its backends' fingerprints and released profiles are immutable. An explicit --cpu-profile host warns too, with the same request where no built-in profile serves the CPU.

#408 tracks built-in profiles for more Intel CPUs, such as Tiger Lake and Meteor Lake, and #409 for more AMD CPUs, and maintainers link each CPU profile request to one of them; host profiles serve no CPU of another vendor than Intel and AMD. A profile derives from fingerprints of its generation's hosts on every backend that it serves; see vmm_core/cpu_profile in the OpenVMM submodule.

Benchmarking

benchmark

python3 scripts/nvx.py benchmark [OPTIONS]
Option Default Description
--suite {boot,snapshot,restore,e2e,phase2,snapshot-profile,all,cold-start,device-io,device-restore-profile,network-snapshot,performance,platform-snapshot-restore-vcpu,shell-snapshot,shell-snapshot-restore,snapshot-restore-memory,snapshot-restore-vcpu,virtfs} boot Select an acceptance, diagnostic, or workload suite.
--backend {whp,kvm,mshv,both} both on Windows; kvm elsewhere Select the hypervisor backend.
--platform NAME inferred OS/backend Record the host-typed performance series.
--openvmm-dir PATH openvmm/ Select the OpenVMM repository.
--nvx-dir PATH repository root Select the NVX repository containing guest artifacts.
--warmups N 5 for device-io; 1 for device-restore-profile; 3 otherwise Set the number of excluded warmup attempts; zero is allowed.
--runs N 30 for device-io; 5 for device-restore-profile; 11 otherwise Set the number of retained attempts.
--memory-mib MIB 128 Set guest memory for the general suites.
--processors {1,2,4,8} 1 Run every cold, capture, restore, and workload launch with this microVM count.
--virtfs-runs N 3 Set the number of virtio-fs workload samples.
--virtfs-memory-mib MIB 512 Set guest memory for the virtio-fs workload.
--payload-mib MIB 64 Set the virtio-fs sequential I/O payload size.
--shell-memories MIB [MIB ...] 128 256 512 (128 256 512 1024 for snapshot-profile) Set the guest memory sizes for shell snapshot measurements.
--network-memory-mib MIB 256 Set guest memory for the network snapshot workload.
--restore-devices {console,net,virtiofs} [...] all three devices Select devices for the device-restore-profile suite.
--restore-modes {active,deferred} [...] both modes Select activation modes for the device-restore-profile suite.
--device-io-duration-seconds SECONDS 10 Set each storage-operation or UDP round-trip measurement window.
--device-io-size-mib MIB 512 Set the virtio-blk and virtio-fs backing-object size.
--device-io-port PORT 5201 Set the same-host UDP echo port.
--net IPV4/PREFIX none Enable virtio-net with a static guest address.
--network-profile {portable} none Required with --net; selects the portable KVM/MSHV/WHP contract.
--cpus CPUSET one logical CPU per physical core Set process affinity in taskset syntax.
--host-cpu-reserve N 2 Require this many affinity CPUs beyond the guest vCPU count for VMM/device work.
--timeout SECONDS 10 Set the time allowed for each boot marker.
--teardown-mode {guest-exit,host-terminate,host-sigterm} guest-exit Select how to stop a measured VM; host-sigterm is a deprecated alias.
--skip-build off Reuse existing release binaries.
--snapshot-profile off Retain OpenVMM lifecycle phase samples and host counters, and on Windows sample storage, CPU, writeback, and Defender telemetry during each capture; implied by the snapshot-profile suite.
--cache-state {warm,cold,both} both Select artifact cache states for the snapshot-profile suite.
--output PATH none Write the benchmark result as JSON.
--output-dir PATH none Write canonical workload logs to a directory.
--scratch-dir PATH system temporary directory Select an existing directory for temporary snapshots, guest RAM backing, and workload files.
--keep-kvm-stage off Keep temporary staged KVM benchmark binaries.

Measured counts must be at least 1; warmups may be zero, and timeouts must be greater than zero. Every suite but phase2, which boots no microVM, refuses a host before it builds or measures anything if no built-in CPU profile of the --openvmm-dir checkout serves its CPU, or if it cannot identify the CPU or read that checkout's profiles and so cannot confirm that one does. On such an Intel or AMD CPU, OpenVMM's default CPU profile selection, auto, falls back to a host profile, whose every cold boot fingerprints the hypervisor first and whose results no baseline shares, and the refusal names the CPU and links a CPU profile request. On another vendor's CPU, every cold boot fails with E_PROFILE_HOST_UNKNOWN, as it does on a CPU that profiles of more than one of the checkout's generations cover, which benchmark refuses too: auto rejects such a catalog without falling back. The e2e suite uses the general memory size and measures cold start, snapshot generation, snapshot restore, guest-exit teardown, and peak RSS against the shell-ready markers. CI uses the default 128 MiB baseline. JSON results and workload metadata record the benchmark host's provenance under host, without credentials or Azure resource identifiers. See Benchmark for suite semantics, platform support, metric definitions, and complete examples.

performance

performance processes benchmark outputs for CI. It requires one nested command.

performance collect

python3 scripts/nvx.py performance collect
    --platform PLATFORM
    --commit COMMIT
    --input-dir PATH
    --output-dir PATH
    [--require-network]
    [--require-shell-snapshot]
    [--require-shared-suite]
    [--require-shell-snapshot-restore-512]
    [--lifecycle-input PATH]
    [--lifecycle-attempt-log PATH]
    [--summary PATH]

Parses canonical benchmark logs into p50 CSV files. Each row records --commit as a full commit ID, resolving a revision such as HEAD in the NVX checkout. The --require-* flags reject incomplete inputs for their respective workload sets. --require-shell-snapshot-restore-512 accepts only the canonical 512 MiB restore metric from a 2-, 4-, or 8-vCPU run. --lifecycle-input validates and merges a 128 MiB, guest-exit e2e JSON result, producing the 29-metric microVM CI result. A directory whose metadata selects device-io is collected as five additional ABI-3, one-vCPU ops/s metrics; CI merges them into a 34-metric one-vCPU result. --lifecycle-attempt-log reads the attempt log that validate-openvmm records. The --lifecycle-input file must be a byte-identical copy of exactly one valid logged attempt, or collection fails before it writes a CSV; the command names that attempt. On MSHV and WHP, collection also fails before it writes a CSV when a shell-snapshot restore series fails the check of validate-workload. --summary writes the p50 table plus lifecycle min/max/sample-count and RSS diagnostics, the benchmark host provenance, and any snapshot-generation storage telemetry. With --lifecycle-attempt-log, it also lists every attempt, marks the published one, and reports the telemetry of rejected attempts. When the inputs record the benchmark host, the command also writes it to --output-dir as <platform>-host.json, the format of an expected benchmark host. It fails if only some inputs record a host, if the inputs, or a host file already in --output-dir, record another host, and if --output-dir would mix results that record a host with results that record none.

performance validate-openvmm

python3 scripts/nvx.py performance validate-openvmm
    --platform PLATFORM
    --input PATH
    [--attempt-log PATH]

Validates a complete 128 MiB, guest-exit OpenVMM e2e result without writing a CSV. Snapshot-generation instability exits with status 75 so callers can remeasure the temporary host condition selectively; other malformed or incomplete inputs exit with status 2. --attempt-log appends the input's path, SHA-256 digest, outcome (valid, unstable, or invalid), exit status, and error to a JSON attempt log, creating it on the first attempt.

performance validate-workload

python3 scripts/nvx.py performance validate-workload
    --platform PLATFORM
    --input-dir PATH

Checks the shell-snapshot restore series in the shell-snapshot.log and shell-snapshot-restore.log of a workload output directory without writing a CSV, so that callers can remeasure a series that a transient host stall split before they collect it. On MSHV and WHP, a series of at least ten samples must list them in its log, and it fails the check when its p50 is more than 25% above p25 or a gap of more than 25% separates its sorted samples with at least two on each side; the command reports every such series. KVM restore series are not checked, as Shell snapshot explains. A failed check exits with status 75; a missing log, a malformed or inconsistent sample list, or an unsupported platform exits with status 2.

performance collect-openvmm

python3 scripts/nvx.py performance collect-openvmm
    --platform PLATFORM
    --commit COMMIT
    --input PATH
    --output-dir PATH
    [--summary PATH]

Converts a complete 128 MiB, guest-exit OpenVMM e2e result into an eight-metric lifecycle p50 CSV. Like performance collect, it records --commit as a full commit ID and writes the recorded benchmark host to --output-dir as <platform>-host.json.

performance gate

python3 scripts/nvx.py performance gate
    --baseline-dir PATH
    --target-dir PATH
    [--window N]
    [--minimum-history N]
    [--threshold PERCENT]
    [--absolute-tolerance-ms MILLISECONDS]
    [--history-reset-dir PATH]
    [--summary PATH]

Checks target p50 values for regressions against rolling baseline histories. --window defaults to 10, --minimum-history to 10, --threshold to 40, and --absolute-tolerance-ms to 5. The gate uses the median of the available window and treats metrics with insufficient history as warmups. --summary writes a Markdown summary. --history-reset-dir names tracked candidate histories; metrics removed from an existing history file restart baseline warmup. When a candidate expected host, <platform>-host.json in --history-reset-dir, differs from the one in --baseline-dir in a required property, the gate fails unless the candidate history files keep no row of that series, of any metric.

performance persist

python3 scripts/nvx.py performance persist
    --source-dir PATH
    --history-dir PATH
    [--exclude-metric NAME]...
    [--require-host]

Appends current p50 values to branch history. Repeat --exclude-metric to omit more than one metric. When both directories contain a platform's host file, <platform>-host.json, the command refuses to append that platform's results if their recorded host differs from the history's in a required property, and warns about advisory differences. It also refuses results without a host file for a history that has one. --require-host requires both files for every platform. When --history-dir lies in a git checkout, even before the directory exists, the command refuses results whose commit is not a full commit ID or not an ancestor of HEAD, and skips results measured before a later commit removed rows of their metric from the history file, as a reset does; see CI collection.

performance check-host

python3 scripts/nvx.py performance check-host
    --platform PLATFORM
    --history-dir PATH
    [--scratch-dir PATH]
    [--output PATH]
    [--summary PATH]

Compares this host with the expected host of the PLATFORM series, <platform>-host.json in --history-dir, as CI does before it measures the series; see Expected benchmark hosts. The command exits with status 1 when a required property, such as the CPU, the VM size, the OS build, or a benchmark disk or volume, differs, and with status 2 when the expected host is missing or malformed. An advisory difference, such as the microcode revision or a Windows cumulative update, only warns. The command prints every compared property, and reports the required and the advisory differences as one GitHub Actions error and one warning annotation. --scratch-dir names the benchmark scratch directory whose volume the benchmarks will use; it defaults to the system temporary directory. --output writes this host in the host-file format, ready to replace the expected host, and --summary appends a Markdown comparison that includes it.

performance storage-events

python scripts\nvx.py performance storage-events
    --since TIME
    --output-dir PATH
    [--scratch-dir PATH]

Saves the Windows storage events from --since, an ISO 8601 time with a UTC offset or Z, such as PowerShell's round-trip format, until now, as a failed Windows platform job does; see CI collection. It writes the Microsoft-Windows-Storage-Storport/Operational events and the System log's storage events as rendered XML, and storage-events.txt, which summarizes them and lists every event, to --output-dir, and prints the summary. I/Os that took longer than Storport's threshold raise a GitHub Actions warning annotation, and so does a log that can't be read, which doesn't fail the command. --scratch-dir names the benchmark scratch directory, whose volume, with the workspace, temporary, and system volumes, the summary names for each Storport device; it defaults to the system temporary directory. A stalled volume can stall what reads it, so the command abandons a query after 120 seconds, and probes the volumes only once it has saved the events, for at most 30 seconds, after which the summary says why it names none. It also abandons creating --output-dir or writing a file there after 30 seconds, as the directory can share that volume, and the summary and a warning then name each file that it couldn't save. Reading the Storport log takes a service account, such as the runners' Network Service, an elevated prompt, or membership in Event Log Readers. On other hosts, the command exits with status 2.

Source and package commands

collect-sources

python3 scripts/nvx.py collect-sources

Materializes the verified Linux, Alpine, and Ubuntu source artifacts needed for a source-inclusive release. Azure Linux corresponding source is not collected, so source-inclusive packages omit the Azure Linux guest.

collect-alpine-sources

python3 scripts/nvx.py collect-alpine-sources MANIFEST [MANIFEST ...]
    [--output PATH]
    [--cache PATH]
    [--skip-upstream]
Option Default Description
MANIFEST required One or more Alpine package manifests to collect.
--output PATH build/sources/alpine Select the output directory.
--cache PATH .cache/aports Select the aports cache directory.
--skip-upstream off Collect exact aports recipes without running abuild fetch.

Each run replaces the generated recipes, upstream, manifest.json, and SHA256SUMS entries. The collector rejects an output path that is a symlink or a Windows junction, and an output directory that contains anything else. Release checksum manifests reject symlinks, so a recipe symlink is stored as a regular copy of its target. Links that leave the recipe, hard links, and special files are rejected.

collect-ubuntu-sources

python3 scripts/nvx.py collect-ubuntu-sources MANIFEST [MANIFEST ...]
    [--output PATH]
    [--cache PATH]
Option Default Description
MANIFEST required One or more Ubuntu package or EROFS manifests to collect.
--output PATH build/sources/ubuntu Select the output directory.
--cache PATH .cache/ubuntu-source-indexes Select the downloaded source-index cache.

The collector requires gpgv, deduplicates source package name/version pairs, authenticates live or historical Ubuntu source indexes through signed InRelease metadata and the pinned Ubuntu archive keyring, validates each .dsc, and downloads every referenced source member.

create-linux-source-archive

python3 scripts/nvx.py create-linux-source-archive
    --config PATH
    --output PATH

Creates the deterministic Linux corresponding-source archive from the pinned kernel inputs and the generated kernel configuration. --config must name an existing generated kernel configuration, and --output names the archive to create.

package

python3 scripts/nvx.py package
    [--version VERSION]
    [--destination PATH]
    (--include-source | --binary-only)
    [--force]
Option Description
--version VERSION Override the packaged version.
--destination PATH Override the staging destination.
--include-source Include the corresponding source artifacts in the package and omit the Azure Linux guest artifacts.
--binary-only Stage binaries only; publish corresponding source separately.
--force Replace an existing staging destination.

Exactly one of --include-source and --binary-only is required. Every package ships nvx with its provenance, which must describe nvx/target/release/nvx[.exe] as built from the checkout's sources for the release target of the host's operating system: x86_64-unknown-linux-musl, a static executable, on Linux, and x86_64-pc-windows-msvc on Windows; see record-nvx-provenance. See Package and source delivery for release procedures and source-publication requirements.

archive-release

python3 scripts/nvx.py archive-release
    --source PATH
    --destination PATH

Validates the staged distribution against its SHA256SUMS, snapshots the accepted inventory, and creates a deterministic archive at the destination. The destination must be outside the source directory and end in .tar.gz or .zip.