Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,19 @@ jobs:
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
signingKey: ${{ secrets.CACHIX_SIGNING_KEY }}
- run: nix flake check -L
# The checks `lib.caisson.mkVolatileCheck` returns, built into the
# store of the runner (tests/volatile-checks/check.sh): a pass writes
# its marker and a failure leaves none. A check of `nix flake check`
# is itself a build, and its builder cannot run `nix build` against
# the store of the runner, so the script runs as a step of the job.
volatile-checks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: DeterminateSystems/nix-installer-action@v22
with:
determinate: false
- run: bash tests/volatile-checks/check.sh
# A system-manager configuration activated in a container of each
# distribution (tests/system-manager/container.nix). system-manager's
# container test driver runs the container in the build sandbox, which
Expand Down
19 changes: 19 additions & 0 deletions docs/development/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,25 @@ plumbing works.
<path of the NixOps4 source> --impure
```

## Volatile Checks (Builds in the Local Store)

- **Scope:** The derivation `lib.caisson.mkVolatileCheck` returns, built:
a check whose program passes writes its pass marker at the output
path its derivation declares, building it again with `--rebuild`
keeps the marker, and a check whose program fails leaves no marker in
the store. The pinned-world suite holds what evaluation shows of the
derivation: its name, its fixed output and its `drvAttrs`.
- **Location:** `tests/volatile-checks/check.sh` and the checks it
builds in `tests/volatile-checks/default.nix`.
- **Mechanism:** the script runs `nix build` on each check and reads the
store afterwards with `nix path-info`. A check of `nix flake check` is
a build and cannot do either, so the `volatile-checks` job of the
check workflow runs the script on the runner.
- **Execution, from a checkout:**
```bash
tests/volatile-checks/check.sh
```

## Integration Testing (Test Flakes)

- **Scope:** Module composition, end-to-end evaluation, and build success.
Expand Down
35 changes: 34 additions & 1 deletion lib-overlays/integrations/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -3031,6 +3031,35 @@
else
[ ];

# Whether a gathered check (`gatheredChecksOf`) is of a volatile
# category. A check of a volatile category runs in a fixed-output
# derivation (`lib.caisson.mkVolatileCheck`), and one whose
# derivation has `drvAttrs` that declare no `outputHash` is
# refused. The tools that act on volatility, such as `topside
# check`, read it here, in the check record. The `checks` output
# reads no property of a category, so it does not refuse such a
# check: reading `volatile` evaluates the declarations of the
# category, which `nix flake check` has no use for.
volatileOf =
entry:
let
derivation = entry.check.derivation;
in
if !entry.volatile then
false
else if builtins.isAttrs (derivation.drvAttrs or null) && derivation.drvAttrs ? outputHash then
true
else
throw ''
caisson: the check `${entry.checkName}` of the configuration at ${showNamePath entry.namePath},
declared at ${showCode entry.check.code}, is of the volatile category
`${entry.check.category}`, and its derivation is not a fixed-output derivation:
its `drvAttrs` declare no `outputHash`. A check of a volatile category runs in a
fixed-output derivation, which Nix lets reach the network, and its output is a
pass marker. Build the derivation with `lib.caisson.mkVolatileCheck`, or name a
category that is not volatile.
'';

# The check record of a caisson top, keyed as `checks` of its
# outputs is, by system and then published name: for each check,
# `name` and `system`, `from.namePath`, the path from the caisson
Expand All @@ -3045,6 +3074,10 @@
# top as `check`, and the entry of the check record of the test
# caisson top as `record`, null when the test caisson top publishes
# no check record. For every other check `top` is null.
#
# The `volatile` field of a check of a volatile category is
# refused when the derivation of the check is not a fixed-output
# derivation (`volatileOf`).
checkRecordOf =
manifest:
builtins.mapAttrs (
Expand All @@ -3059,7 +3092,7 @@
code
top
;
inherit (entry) volatile;
volatile = volatileOf entry;
requiredFeatures = requiredFeaturesOf entry.check.derivation;
}
)
Expand Down
2 changes: 2 additions & 0 deletions lib-overlays/tooling/default.nix
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

mkMemoizedDerivationRead = import ./mk-memoized-derivation-read.nix { lib = final; };

mkVolatileCheck = import ./mk-volatile-check.nix;

};

};
Expand Down
104 changes: 104 additions & 0 deletions lib-overlays/tooling/mk-volatile-check.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# SPDX-License-Identifier: MIT
#
# This file is the function `mkVolatileCheck`. It returns the
# derivation of a check whose category is volatile: a check whose result
# depends on something outside the build, such as a service reached
# over the network.
#
# A configuration declares such a check with the derivation it returns:
#
# caisson.checks.mirror-answers = {
# category = "online";
# derivation = lib.caisson.mkVolatileCheck {
# inherit pkgs;
# name = "mirror-answers";
# program = pkgs.writeShellApplication {
# name = "mirror-answers";
# runtimeInputs = [ pkgs.curl ];
# text = ''
# curl --fail --silent --cacert ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt \
# https://mirror.example.org/health
# '';
# };
# };
# };
#
# The derivation is a fixed-output derivation. Nix lets the builder of
# a fixed-output derivation reach the network, and in exchange the
# derivation declares the hash of its output in advance. The builder
# runs the program. When the program exits with status 0, the builder
# writes the pass marker, a file holding the line `pass`, as its
# output. When the program fails, the build fails and leaves nothing in
# the Nix store.
#
# Nix computes the store path of the output of a fixed-output
# derivation from its name and the declared hash alone. Every pass
# marker has the same hash, so the name tells markers apart: it is
# `<name>-<hash>`, where `<hash>` is the hash part of the store path of
# the program, the 32 characters after `/nix/store/`. A change to the
# program, or to anything its store path depends on, gives a new store
# path for the program and so a new marker. While the program stays the
# same, the marker path stays the same, and a marker already in the
# store, or in a cache, counts as a pass without the program running.
# A tool that runs the check again builds the marker with substitution
# off and `--rebuild`, as `topside check --rerun-volatile` does.
#
# Arguments:
#
# - `pkgs`: the package set for the system of the check. The builder
# is the bash of this set, and the derivation is built for its host
# platform.
# - `name`: the name of the check, the first part of the name of the
# marker.
# - `program`: the executable the check runs, with no arguments. It is
# a derivation whose output is an executable file, as
# `pkgs.writeShellScript` returns, or a string naming an executable
# inside a store path, as `lib.getExe` returns for
# `pkgs.writeShellApplication`, or a path, which Nix copies into the
# store. The builder sets no `PATH`, so the program reaches only the
# commands it names by store path or puts on its `PATH` itself, as
# `pkgs.writeShellApplication` does with its `runtimeInputs`. A
# program that reaches a server over TLS names a CA bundle, such as
# that of `pkgs.cacert`, since the build sandbox has none.
# - `impureEnvVars`: the names of the environment variables of the Nix
# process that runs the build which Nix passes on to the builder.
# With a Nix daemon, that process is the daemon. Of the machine that
# builds the check, the program reaches the network and these
# variables.
#
# The derivation is a plain `derivation`, so its `drvAttrs` hold the
# attributes above, and the store paths it depends on are the context
# of those attributes: the bash of `pkgs` and the program.
{
pkgs,
name,
program,
impureEnvVars ? [ ],
}:
let
programPath = "${program}";
storeHash = builtins.match "${builtins.storeDir}/([0-9a-z]{32})-.*" programPath;
in
if storeHash == null then
throw ''
caisson: mkVolatileCheck: the program of the check `${name}` is
${builtins.toJSON (builtins.unsafeDiscardStringContext programPath)}, which is not
inside the Nix store. The name of the pass marker carries the hash of the store
path of the program. Pass a derivation, a path, or a string inside a store path,
such as `lib.getExe` of a package.
''
else
derivation {
name = "${name}-${builtins.head storeHash}";
inherit (pkgs.stdenv.hostPlatform) system;
builder = "${pkgs.bash}/bin/bash";
args = [
"-c"
''"$program" && printf 'pass\n' > "$out"''
];
program = programPath;
inherit impureEnvVars;
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = builtins.hashString "sha256" "pass\n";
}
14 changes: 12 additions & 2 deletions modules/generic/caisson/types.nix
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,10 @@ in
The derivation that is the check, for the system of the
evaluation result that declares it: the check passes when the
derivation builds. Null for an entry that annotates a check the
configuration already has, and for an entry with a `top`.
configuration already has, and for an entry with a `top`. A
check of a volatile category (`caisson.checkCategories`) has a
fixed-output derivation, such as the derivation
`lib.caisson.mkVolatileCheck` returns.
'';
};
top = lib.mkOption {
Expand Down Expand Up @@ -166,7 +169,14 @@ in
description = ''
Whether the result of a check of the category depends on
something outside the build, such as a network, a service or a
piece of hardware.
piece of hardware. The derivation of a check of a volatile
category is a fixed-output derivation, the kind of derivation
Nix lets reach the network, and its output is a pass marker.
`lib.caisson.mkVolatileCheck` builds it from the program the
check runs. The check record of a caisson top refuses a
check of a volatile category, when its `volatile` field is
read, if the `drvAttrs` of its derivation declare no
`outputHash`.
'';
};
};
Expand Down
Loading
Loading