What to look at when something is wrong, what each command needs before it will act, and where a workspace stops being comfortable.
snailmail status also reports size: each repository's lock in bytes, the count
and total size of the distinct artifacts it binds, and the workspace's lock total
with the largest repository named. The lock is parsed whole on every plan and
every apply, so its size is the number that predicts where a workspace stops
being comfortable — and which repository to split when it does.
snailmail status reports committed workspace evidence without contacting
hosts or providers. Human output summarizes visible and retained versions,
visible publication-binding completeness, and whether a managed deployment
receipt is recorded; --json emits the same deterministic schema for
automation. A receipt is evidence of a prior successful apply, not proof that a
host currently serves those bytes.
snailmail rollout answers when each version reached a client, derived from the
publication ledger rather than stored anywhere. The ledger already records one
append-only entry per publication, so the date is read back rather than kept a
second time. A version is listed with the date it was first published, the
number of published trees that have carried it, and whether the repository still
serves it; --withdrawn includes versions that were published and later yanked
or pruned, because publication is immutable even when the offer is not.
snailmail ci github and snailmail ci gitlab emit a pipeline that publishes
this workspace, to stdout rather than to a file: it carries decisions snailmail
cannot make — which registry to pull verification images through, which secret
names a project uses — and a file the tool owned would be rewritten over an
operator's edits. What it does derive is what the workspace already says: which
signing keys need materialising, whether a foreign architecture needs emulation,
and which repositories publish into a directory something else must serve. Both
providers derive the same facts; only the rendering differs, and it differs more
than wording. A GitLab runner has no Docker daemon, so client verification needs
one as a service; its jobs share no filesystem, so what apply builds is declared
as an artifact; and Pages there serves a job artifact rather than a moved ref, so
there is no orphan commit.
snailmail check is a read-only integrity audit of every retained package
version, including yanked and pruned versions. It verifies local CAS bytes or
fetches the configured S3 authority into temporary storage, reparses native
package facts, and revalidates historical publication bindings. Upstream release
discovery remains unavailable until releases are modeled. check --origins
re-fetches explicitly adopted URLs and compares their pinned bytes; default
checks remain offline from external sources. Origin checks process at most four
sorted records per run; --origin-offset selects subsequent batches.
plan requires the manifest, configured locks, publication ledgers, and
deployment receipts to be committed in a complete, non-shallow Git repository. apply
executes the reviewed plan without replanning, verifies staged repository
bytes, commits its exact publication ledger records with a compare-and-swap,
and publishes through the selected host's conditional release switch. After
canonical verification succeeds, apply commits a deployment receipt separately
from the pre-effect publication ledger. S3 keeps
the verified tree under an immutable digest prefix and conditionally updates a
small root index that points clients at that tree.
Every command exits 0 on success. A failure exits with what to do about it,
so a CI job can retry a flaky network without also retrying a malformed bucket
name:
| Code | Meaning | What to do |
|---|---|---|
1 |
Failed | Read the message. |
2 |
The configuration is wrong | Fix the workspace or the host configuration; retrying fails again. |
3 |
The host or network failed | Retry. |
4 |
The plan no longer matches the world | Run plan again, review the diff, then apply. |
5 |
A publication may or may not have taken effect | Do not retry blindly. Check the host, then snailmail status. |
Codes 3, 4 and 5 also print a line saying the same thing, for whoever is
reading the log rather than branching on the number.
Only failures reported by a host carry this detail; anything else exits 1.
Codes are stable — a job that branches on them should not break under an
upgrade.
Render the read-only public matrix and machine-readable status from committed locks, ledgers, deployment receipts, and an optional current plan:
go run ./cmd/snailmail dashboardThe workspace lock is a file inside the checkout, so it serialises operations on one machine and not two. What protects a repository when two CI runners publish at the same time is the conditional commit: each states the revision it observed, and the second to arrive is refused because that is no longer what is live.
So the failure mode is one run fails, not one run waits. It fails as a stale plan, which is a retry signal — replanning against what is actually live and applying again succeeds. A pipeline that publishes from more than one place at once wants a concurrency group so this is rare rather than routine; the generated workflows set one.
Two things worth knowing. Staging is not exclusive, deliberately: two runners can both stage, because a stage writes only under its own identifier and touches nothing a client reads. And a local host cannot be shared at all — its output path is relative to its workspace — so this only arises for object storage and Pages, where two runners can name the same bucket and prefix or the same repository and branch.
Use one workspace for the whole organisation unless you have a reason not to.
It sounds like fifty teams would contend on one review queue and one lock, and
they do not. Each repository has its own lock file, so two teams publishing to two
repositories touch two different files and git merges them; snailmail.toml is the
only shared file, and it changes when a repository is configured rather than when
one is published to. Point CODEOWNERS at repos/<name>.lock.toml and each team
reviews its own publications. The workspace lock covers a single checkout, so CI
runners with their own checkouts never wait on each other — what orders two
concurrent publications is the host refusing the second, per repository.
Split when one of these is true:
- A team needs its own blob store. A plan is bound to one, so a workspace has exactly one.
- A team cannot share read access to the state repository, because a workspace is a git repository and access to it is all-or-nothing.
Separate workspaces can still publish into the same bucket under different
prefixes. What you lose is the shared view: snailmail site describes one
workspace, so splitting means several index pages and no single answer to where a
package is published.
A repository lock is parsed whole on every plan, apply, status and check, so its size is what decides how long those take and how much memory they need. Measured at roughly 385 bytes per package-version, with parsing costing about one and a half times the file in heap:
| package-versions | lock file | parse | heap |
|---|---|---|---|
| 2,000 | 0.8 MB | 5 ms | 1 MB |
| 20,000 | 7.7 MB | 44 ms | 12 MB |
| 100,000 | 38.5 MB | 226 ms | 59 MB |
snailmail refuses a lock over 128 MiB — about 330,000 package-versions — and says so, naming the repository, the size and what to do:
repository "repos/apt.lock.toml" has a 200 MiB lock, over the 128 MiB limit, and
parsing it needs roughly 300 MiB of memory on every plan and apply: prune retained
versions, split the repository, or set SNAILMAIL_MAX_LOCK_BYTES to proceed anyway
That is the honest state of things: one lock per repository is the current design,
and a workspace past this size wants the lock sharded per package rather than a
larger limit. The limit exists so that a workspace which has outgrown the design
finds out in a sentence instead of getting slower until something is killed.
snailmail status reports every lock's size, so the number can be watched before
it becomes a refusal.
Artifacts themselves are not held in memory — they stream from content-addressed storage — so the ceiling is index and lock size, not repository size. Generated indexes are held whole, which for Debian and yum is the binding constraint at around a million package-versions.
Past 2,000 package-versions a repository's lock is written as one file per package, under a root that indexes them and a Merkle digest over the set:
repos/apt.lock.toml the root: schema, placement, shard index, Merkle root
repos/apt.lock.d/3f/… one file per package
Nothing about a lock's meaning changes — it still has a single identity that a plan pins, and a shard edited on its own is refused by name rather than accepted quietly. What changes is what a publication costs. Adding one version rewrites one small file instead of the whole lock, so a reviewer sees the package that changed rather than a multi-megabyte diff. Measured at 10,000 packages: adding a version takes 108 ms and reading the lock 102 ms, against 3.4 s to write when every shard is compared and 950 ms to read them one at a time.
The transition happens once, automatically, when a repository grows past the threshold, and a lock never goes back to one file — returning would itself be a diff rewriting everything. The first sharded write of a large repository creates every file and takes seconds; subsequent ones do not.
Back to snailmail.