Staging telemetry is on unless --no-staging says otherwise (#563)

[anth-volk]

Fixes #563.

Staging telemetry recorded nothing from 2026-07-02 onward. Every national release since carried "staging": null, the staging repo's runs.json stayed frozen at that date, and no gate treated any of it as a problem — the dashboard served three-week-old data with two runs stuck at status: "running".

--no-staging remains a supported way to build and is untouched. The defect is that other things produced the same outcome without anyone choosing it, and the manifest could not tell the cases apart.

The bug

os.environ.get(name, default) falls back to the default only when the variable is absent. Exported empty, it returns "", which is falsy, so an empty POPULACE_STAGING_REPO_ID defeated the on-by-default staging repo and _staging_telemetry took its "no destination" branch. Staging then did nothing for an entire build.

The second half is that this was unreadable afterwards. A deliberate --no-staging and a silently defeated one both wrote "staging": null, which is why several releases shipped this way before anyone noticed, and why it still cannot be said with certainty which of buildm/n/o were choices.

What changed

Commit
366958f	Name the staging repo id and run prefix defaults instead of inlining them
524ef8a	Treat an empty staging environment override as unset, not as off
a01ad02	--no-staging is the only way a build produces no staging telemetry
4200543	Record in the build manifest why a release has no staging run
7f67e43	Refuse to publish a release whose staging telemetry never arrived
249d90d	Normalize the staging path prefix where the class owns it
6f6e606	Mark a staging run failed when the build does not finish
639c3f4	Changelog

Three things beyond the literal fix are worth calling out, because each closes a way the same failure could recur:

The manifest now states what happened. A skip is a positive statement carrying its reason ({"enabled": false, "reason": "--no-staging"}), following the shape warm_start and selection_source already use in the same manifest. A staged run carries its run id, destination, and uploads_succeeded.

That last field matters more than it looks. Uploads are best-effort and self-disable after three consecutive failures, so on a machine without a Hugging Face write token repo_id is already cleared before _staging_telemetry returns. The manifest previously recorded the configured repo id, asserting a destination that had received nothing. Without this field a publish-time gate would pass on exactly the misconfiguration it exists to catch.

Publishing refuses the dishonest case only. A build that meant to stage and delivered nothing is refused (--allow-missing-staging overrides); a declared opt-out publishes silently. Scoping is by presence of the staging key rather than country or dataset role: a builder that stages writes the key on every build, one with no staging path writes none, so the ACS local-area product stays out of the gate with no exception list. A release-id prefix test would not have worked — local-area ids also begin with populace-us-.

A crashed build marks its run failed. StagingTelemetry.fail() had no call sites and the entry point had no handler, which is why two runs read "running" indefinitely. main() is now a thin wrapper; the build body is _main, unmoved and unindented.

Verification

Full suite green across all five libraries (pytest packages, exit 0), ruff check . clean, and all five wheels build with the new symbols present in the packaged artifacts.

The three cases that define the fix, checked against the real parser:

1. POPULACE_STAGING_REPO_ID=""  -> repo='policyengine/populace-us-staging' prefix='runs'
2. --no-staging                 -> telemetry=None  block={'enabled': False, 'reason': '--no-staging'}
3. --staging-repo-id ""         -> SystemExit(2), before any download

The regression test for the bug was also confirmed to fail when the fix is reverted (assert '' == 'policyengine/populace-us-staging'), so it cannot pass vacuously.

Two source-inspection tests read the build body by name and correctly failed on the main → _main rename; both now read _main.

Notes for review

Operationally significant. Every release built before 4200543 has staging: null, so re-publishing any of them now requires --allow-missing-staging. That is intended, but it is the change most likely to surprise someone mid-release. 7f67e43 is the natural split point if it should land separately.

The root cause is still unconfirmed for buildm/n/o. experiments/build_j_recert/buildj_sparse.sh:86 passes --no-staging explicitly and builds exactly the buildj release in #563's evidence table, so at least one null block there has a legitimate cause. No checked-in script covers buildm/n/o. This PR handles both causes, and after 4200543 the question stops being answerable retroactively but stops recurring; printenv POPULACE_STAGING_REPO_ID on the build machine settles the history.

Local-area datasets are out of scope. Everything here applies to the national fiscal refresh only. StagingTelemetry has exactly one consumer — tools/build_us_fiscal_refresh_release.py — so the ACS local-area build (tools/build_us_acs_local_release.py) emits no staging telemetry at all, writes no staging key, and is untouched by every change in this PR including the publish gate. Its publishes keep working, verified by test.

That leaves a real gap this PR does not close: a local-area candidate still cannot be reviewed on the staging dashboard before it is published. Giving it telemetry is not a matter of copying the wiring — the two builds are shaped differently. The local-area build has five separately resumable stages (materialize, calibrate, qa, finalize, package) bound by an on-disk run_identity.json, and no release id exists until the last of them, whereas StagingTelemetry assumes one process from start to finish and keys the run on the release id. Its default --stage all does run in one process, so the common path would work as-is; the split-stage recovery path would need the telemetry to resume across invocations, which is a change to the class rather than to the caller.

Not covered. SIGKILL is out of reach of any in-process handler, so an out-of-memory kill still leaves a stale run — the likelier explanation for the two live ones, given this build has been jetsam-killed before. Reaping those needs a staleness rule on the reader side and is left out.

Unrelated finding from the same investigation, filed separately: #575.