Repository navigation
Recertify: uncertified in-place change to deployed deploy_forge.sh (build #39297) + missing product-readiness report #22
Description
Summary
The deployed Certification Forge release 795ca1a927b8af649c91bfe74b84f55dc2e5c852 was modified in place on the production host on 2026-09-24 (~11:44 UTC) by an autonomous build (build prompt #39297, "echo-certforge-dispatcher restart storm / giveup"). The change never went through this repository or a CertForge certification run.
What changed
deploy/deploy_forge.sh (dispatcher unit template), and the same line in the live dispatcher systemd unit:
[Unit]
Description=echo-certification-forge - durable subscriber run dispatcher
After=network.target $SERVICE.service
-Requires=$SERVICE.service
+Wants=$SERVICE.serviceAssessment
Benign. With Requires=, every restart or crash of the API service was propagated to the dispatcher. That cascade pushed the dispatcher into start-limit-hit during the #39297 incident. With Wants=, the dispatcher keeps its ordering after the API service and still starts it, but a failure of the API service no longer takes the dispatcher down. No code, policy, signing or evidence paths were touched. The pre-change copy was kept next to the modified file, so we left the change in place instead of reverting it.
Why this needs follow-up
- The running release no longer matches its certified source tree. This is the kind of uncertified drift the platform is supposed to prevent.
/v1/statuscurrently reportsNOT_READYwithproduct_readiness_report_missing. The signed exact-source readiness report for795ca1ais no longer present on the host.scripts/master_acceptance.pycannot regenerate it from the committedartifacts/alone, because the committed P4 evidence (artifacts/p4_forge_acceptance.json, source8ccfa65) has nocompleted_phase_gate: "P4"field. P1-P3, P5 (adapter gate GO), P6 (p6_acceptance.summary.json, 12/12) and P7 validate. A fresh P4 hostile acceptance run is required. Hosted CI for795ca1ais green (run 33543830708).
Recommended actions
- Upstream the
Requires=→Wants=change as a normal PR againstmain(with a test covering the unit template). - Cut a new exact-SHA release from that PR's merge commit and re-run the full release pipeline: P4 hostile acceptance,
scripts/master_acceptance.py, and the deploy gate. This recertifies the product and restores a signed product-readiness report. - Treat any future in-place edit of a deployed release as an incident. Autonomous builders are now gated on a CertForge verdict before any merge, but direct edits to live files bypass that gate.
Filed by the ECHO ops pass on 2026-09-24.
Activity
echo-build-tracker commentedon Sep 24, 2026
ECHO Build Tracker linked this event to plan
github-echoomegaprime-echo-certification-forge-issue-22. Live evidence: https://tracker-cp.echo-op.com/?plan=github-echoomegaprime-echo-certification-forge-issue-22echo-build-tracker commentedon Sep 24, 2026
ECHO Build Tracker linked this event to plan
github-echoomegaprime-echo-certification-forge-issue-22. Live evidence: https://tracker-cp.echo-op.com/?plan=github-echoomegaprime-echo-certification-forge-issue-22echo-fleet-builder commentedon Sep 24, 2026
Fresh P4 hostile-acceptance attempt: blocked by new HIGH CVEs, not by keys (ops-20260924)
I ran
scripts/run_p4_gate.sh 795ca1a927b8af649c91bfe74b84f55dc2e5c852on FORGE. The run directory is~/.cache/echo-certforge/p4-runs/p4-795ca1a-ops-20260924.Toolchain. The toolchain under
~/.cache/echo-certforge/p4-9c07eb7/toolshad been deleted, so I re-provisioned it:ae1ecd21…prefix recorded in rerun5.clamav/clamav@sha256:7f5389cc…, pulled by digest.Result. A fresh attestation key was generated, and all 12 role images built from the exact commit. Sealing then failed closed on the vulnerability policy (
runner.a). These are HIGH findings, each with a fix available:The first two rows come from the pinned base
python:3.12-alpine@sha256:6d43704b….Keys are not the blocker. The P4 gate mints its own attestation key. The product-readiness signer is
var/run-signing-key.pem, the private half of trusteded25519:a07f417e…, and it is present.Exact work needed before P4, then
master_acceptance.py, can go green:python:3.12-alpinedigest (libcrypto3/libssl3 >= 3.5.8-r0, libuuid >= 2.42.3-r1) everywhere it appears:scripts/run_p4_gate.shBASE_DIGEST,scripts/verify_p4.py,sandbox.DEFAULT_IMAGEand the role Dockerfiles.cryptographyto >= 50.0.0 in the hash-locked role dependencies.run_p4_gate.sh <new sha>,verify_p4.py, thenscripts/master_acceptance.pywith--p4pointing at the verify_p4 closure report (completed_phase_gate: P4). Sign with--signing-key var/run-signing-key.pem. Write the output tovar/product-readiness/<sha>/product-readiness.json.deploy/deploy_forge.sh.Environment identity. Changing the sandbox image changes the worker environment identity (
8e6466d2…). The following must move together:CERTFORGE_GATE_ENV_DIGESTandRUNNER_IMAGEincertforge_merge_gate.py.ECHO_CERTFORGE_ENVIRONMENT_DIGESTin the GitHub Appruntime.env.<target>.<env>./v1/statusstill reportsNOT_READY / product_readiness_report_missing.🤖 Generated with Claude Code
fix(p4): keep the worker health probe's deployment ledger on the work…
echoomegaprime commentedon Sep 24, 2026
Security release progress: P4 is green, hosted CI is blocked by billing (ops-20260924)
Release PR: #23, head
d8ad9fd881365801354b78a758632d2a31ab6828.Done
python:3.12-alpine@sha256:4c47124a…is pinned in all 6 role images, the sandbox default and the P4 gate.cryptographyis 50.0.0, hash-locked. Trivy: 0 HIGH/CRITICAL on the new base.Requires=toWants=change is upstreamed (recommended action 1).INFRA_FAILED. The worker health container died withRead-only file system: '/opt/var'. Since P6 (bfc2159), the worker opens its deployment ledger at import time, and the default ledger path is on the read-only image root. The last green P4 (9c07eb7) predates P6. The probe now puts the ledger on the writable workspace, and a new test covers it.p4-d8ad9fd-ops-20260924-rel3). 12 images sealed (manifest52b7bc67…), 18/18 checks passed,completed_phase_gate: P4,run_outcome: COMPLETE, report sha2564001cb03….verify_p1,p6_acceptanceandp7_acceptanceexit 0.allowed=True, runcert_5b6eaf3ae172cfdc183f203ab1fd5949bec672bf(STAGING_CERTIFIED). ECHO Certification Forge check: neutral, runcert_16ba3e7d….master_acceptance.pydry run: P1–P7, SDK and MASTER all pass. The run stops only at the CI gate.Blocked: hosted CI (external). Runs 36053908255 (bc1b8ad, attempts 1–2) and 36059155489 (d8ad9fd) all report "The job was not started because your account is locked due to a billing issue." The signed product-readiness report requires hosted CI
successat the exact SHA, anddeploy_forge.shfails closed without that report. The release is therefore not deployed, and/v1/statusstill reportsNOT_READY / product_readiness_report_missing. I did not use the local CI replica as CI evidence.To finish once billing is fixed: rerun CI run 36059155489 and save its
gh run view --json status,conclusion,headSha,...output asvar/product-readiness/d8ad9fd…/hosted-ci.json. Then run/home/forge/ops_work/cert20260924/certforge_release_d8ad9fd.shon FORGE. The script runsmaster_acceptance, verifies the signed readiness, runsdeploy_forge.sh(the previous release directory is kept for rollback), and checks/v1/status.Environment identity: no change. The worker environment identity is derived from constant labels plus the adapter set. For both 795ca1a and d8ad9fd it computes to
8e6466d2d285f4e1c8ba3e83258fd097b384d4e9ae7dd723436d449728ccfcb6. Therunner_image_digestin/v1/status(7e41caae…) issha256("certforge-worker-env:runner-image"). The merge gate constants, the GitHub Appruntime.env, the R&D env and the production-E2E attestations therefore need no update. Consequence worth a separate decision: the environment identity does not bind the real sandbox image, so certifications cannot tell the patched sandbox from the unpatched one.Leaving this issue open until the signed readiness report exists and the release is deployed.
🤖 Generated with Claude Code