fix(ci): tell a TestPyPI outage apart from a refused artifact

[podonos-eunseo]

Problem

Publish (TestPyPI) failed three times in one day, every time with the same error:

Trusted publishing exchange failure: audience retrieval failed:
repository at test.pypi.org responded with unexpected 503: Service Unavailable

None of it was ours. Measured while diagnosing it:

• The failing request is GET https://test.pypi.org/_/oidc/audience and it carries no credentials — reproducing it from a laptop with zero configuration returns the same 503. Nothing about this repo, the testpypi environment or the Trusted Publisher binding is even transmitted, so none of them can influence the result.
• 200 responses come back server: gunicorn / application/json; 503 responses come back server: Varnish / text/html with retry-after: 0. The origin is healthy — Fastly is synthesizing the 503 when its edge→origin fetch fails.
• pypi.org is unaffected because it has an origin shield (x-served-by shows an IAD chain, x-cache: MISS, HIT, HIT). test.pypi.org has none: a single POP, MISS every time, so every edge node fetches origin directly with no cached layer to absorb a failure.
• The action makes exactly one such request and has no retry, so one failed fetch kills the job. Failure rate climbed 5% → 44% → 70% → 94% over an hour with nothing moving in this repo — and at the end, audience and /legacy/ were never healthy in the same 15-second window across 32 checks.
• The action image digest is identical across the last success and both failures (sha256:a68d0551…), and publish.yml had not changed since that success.

The real cost is not the lost .devN. It is that "Warehouse refused this artifact" — the one finding this lane exists to surface — was being reported with the same red X as somebody else's CDN wobble. A gate that cries outage is one nobody reads on the day it has something real to say.

And it does have something real to say. twine check is not Warehouse's upload validator; it checks that the README renders. Warehouse separately enforces accepted Metadata-Version values, PEP 625 filename normalization, size limits and wheel tag validity. This repo is actually exposed there: hatchling>=1.27 has no upper bound, so python -m build picks up whatever hatchling exists that day and emits a Metadata-Version (currently 2.5) that nobody here chose. If that moves ahead of what PyPI accepts, every local check passes and only the upload says no — on a day when no one committed anything.

That matters because the prod publish cannot be undone. Per RUNBOOK, twine delete is forbidden and twine yank leaves the version number permanently occupied: a metadata defect found on PyPI burns a release number, while the same defect found on TestPyPI costs nothing.

Solution

Judge the upload failure instead of inheriting it. The unauthenticated audience endpoint answers the one question that separates the two cases — was the index even there?

Situation	Before	After
Index reachable, artifact refused	🔴 cause unlabelled	🔴 named as the artifact
Index down, main push	🔴 + Slack	⚠️ warning, main stays green
Index down, release tag or manual replay	🔴 + Slack	🔴 + Slack (unchanged)
Healthy	🟢	🟢 (identical)

testpypi-smoke is now gated on needs.test-pypi.outputs.uploaded; without that it spends five minutes retrying an index that never received the file and reports the same outage a second time under a job name pointing at the wrong thing. notify-failure is untouched: the tolerated path ends the job green so Slack stays quiet, and both failing paths still reach it.

A release ref is strict whether it arrives as a tag push or as a workflow_dispatch replay — the input names the tag rather than github.ref, and it is the same rehearsal.

Deliberately not included: retrying the publish. At the rates measured here three attempts still fail ~34% of the time, so retry does not solve this and would only blur what the job is reporting.

Verification

• publish.yml parses; test-pypi exposes uploaded, the publish step carries continue-on-error + id=publish, and Judge the failure follows it.
• The judge script was run against all four branches: artifact refused → exit 1 naming the artifact; outage + release → exit 1; outage + main push → exit 0 with a warning; curl connection failure (probe=000) → exit 0 with a warning.
• Fixed a bug found while testing that: || echo 000 produced probe=000000, because curl already prints 000 itself on a connection failure. It is || true now, verified against a dead host.
• The live probe under set -euo pipefail returns the current 503, so the tolerated path is the one this change was written for.

🤖 Generated with Claude Code