Make transactional mail fail safely, and keep data until the researcher is told

[jodeleeuw]

Three commits, one failure story. Mail that could not be delivered, and research data deleted while the researcher still had not been told. Each section below is one commit, in order.

Verification codes and upload-failure notifications are not the same kind of mail, and the difference only shows up at the daily quota.

	Verification code	Upload-failure notification
Nature	Realtime. Someone is watching a form; it expires in 24h.	Deferrable. Still true an hour later.
Delivered by	The request itself, synchronously	onmailcreated, then the sweeper
Retried?	Never	Yes
On failure	503, vague message, cooldown cleared	Stays queued, swept later

Until now both were queued and neither was retried, which produced the worst of both.

Two live bugs this fixes

Verification lied to the researcher. The endpoint returned 200 before delivery was attempted:

await verificationRef(uid).set({ ..., sentAt: now });  // cooldown armed
await sendMail({ ... });                                // queues; sends nothing
res.status(200).json({ success: true });                // ← before any send

So during a quota outage a researcher was told to check an inbox nothing was coming to, and given a 60-second cooldown before they could ask again.

Upload-failure notifications were silently skipped. A retryable ERROR was retried by nobody — onDocumentCreated doesn't re-fire on updates. Meanwhile upload-failure-notify.ts arms uploadFailure.notifiedAt in the same transaction that enqueues the mail, and lastNotifiedAt is a 24-hour floor across episodes. So the researcher was never told their data stopped arriving, nothing re-notified, and the experiment document positively asserted they had been told.

The breaker — mail-availability.ts, systemStatus/mail

Resend returns x-resend-daily-quota (quota used today) on successful responses, not just 429s. That's what makes this proactive: we learn we're at 94/100 while sending still works, instead of by failing.

Verification stops at a ceiling of 90; upload-failure notifications keep going to the full 100. That asymmetry is the design decision — a researcher waiting on a code can come back later, but a notification that data has stopped arriving is the only signal they get, so it should have the last sends of the day.

Two things I checked rather than assumed:

• The rate-limit headers don't help. ratelimit-reset / retry-after describe the per-second window (10 req/s), not the daily quota.
• Resend documents no daily reset time, so nothing depends on knowing one. unavailableUntil is next UTC midnight as a ceiling; what actually reopens sending is the sweeper landing a success. The deferrable path probes, the realtime path only reads. If the real reset is later, the sweeper re-arms the breaker; if earlier, it finds out first.

The header is free-plan-only, so it vanishes on a paid plan. Absent reads as "no daily cap" — this logic turns itself off on upgrade rather than needing removal.

Verification is now synchronous

The breaker is checked before a code is minted, a record written, or a cooldown armed — so clicking repeatedly during an outage costs zero Resend requests. The send is then awaited, and on failure the verification record is deleted so the researcher can retry immediately rather than waiting out a cooldown for a code that doesn't exist.

onMailCreated stands down for inline mail. The claim machinery would make the race safe, but the loser gets skipped-in-flight and can't report an outcome — which is exactly the ambiguity the inline path exists to remove.

Inline failures are terminal, never retryable: nothing will retry them, and a retryable error never gets delivery.expireAt, so calling them retryable would park an address outside the TTL's reach indefinitely.

We can't send verification codes right now. Please try again in a little while.

Vague on purpose, and the same for quota, an unverified domain, and a revoked key — none of it is actionable by a researcher. The code-entry form isn't opened, because there's no code to enter.

The sweeper — scheduled-mail-retry.ts, every 10 minutes

Re-drives retryable mail, and refuses three things:

1. Inline mail, ever. A code delivered an hour late may already have expired.
2. Ambiguous errors past 24h. Send transactional mail through Resend, not SES #211 made timeouts retryable only on the strength of Resend's Idempotency-Key, and Resend honours it for a day. REFUSED_ERRORS (refused, or never reached Resend) carry no duplicate risk and sweep at any age; everything else only inside the window.
3. Anything while the breaker is shut. Sweeping into an exhausted quota fails every document and spends one of its three MAX_ATTEMPTS doing it — so a day-long outage would exhaust every queued mail's budget and turn them all terminal, the exact opposite of the point.

It also ages out mail nobody delivered in three days, marking it terminal so delivery.expireAt is finally written. That closes the hole runbook §4 flags: a retryable ERROR otherwise holds a researcher's address forever, outside the TTL, with nothing to catch it.

Alerting: documented, not built (runbook §6)

You cannot email yourself that you are out of email — same account, same quota. It has to be Cloud Monitoring delivering it. Two log-based metrics, no code, since mail-delivery.ts already logs everything at error level: daily_quota_exceeded (reactive) and the quota reading above ~80 (the leading indicator, and the useful one). Routed to a channel that isn't on the jspsych.org sending domain.

I left these as instructions rather than creating them, since alert policies are live changes to your GCP projects — happy to run them if you want.

Testing

Full suite: 81 suites, 1152 tests, plus tsc and lint.

33 new: 21 pure (both decision predicates as tables — availability, staleness, the UTC-midnight arithmetic, and the sweep predicate's safety argument) and 12 emulator-backed (breaker writes, inline terminality, and every sweep refusal).

The emulator suite uses its own status document. systemStatus/mail is a singleton, and half the assertions need the breaker shut — a suite that shut the real one would fail every other suite that sends mail, including contact-email-verify-emulator.test.js driving the real endpoint over HTTP, differently on each run under --maxWorkers=2. That's why _setMailStatusDocForTests exists in production code.

One thing worth knowing: the inline path had to get its own FUNCTIONS_EMULATOR gate. Without it, inline delivery bypassed the never-send guarantee onMailCreated makes, and four existing verification tests started failing on a send that was never going to happen — which is the gate earning its keep, in the direction it was designed for.

Also

Composite index for the sweep query (delivery.state, delivery.retryable), and a firestore.rules note that systemStatus stays unmatched and therefore default-denied — the account page renders from the 503, not from reading DataPipe's operational posture.

🤖 Generated with Claude Code

---

Second commit: stop deleting data the researcher was never told about

Follow-up in the same PR, because it's the other half of the same failure: the breaker and sweeper keep the notification alive, and this keeps the data it is about alive long enough to matter.

Two ways cleanup lost data

scheduled-upload-retry.ts deleted on createdAt <= now - 7d with no check on status and no check on whether anyone had been told. There's no GCS lifecycle rule on the bucket (I checked), so that sweep is the entire retention policy.

1. A storage-provider outage. The entry is still pending with retries left — it would have uploaded fine on day eight. Deleting it on day seven throws away data that was never actually lost.
2. A notification that never arrived. The seven days count from submission, so part of the window is spent before anything goes wrong. If the notification died on a quota outage, the window closes without the researcher ever learning there was one.

upload-retention.ts

Condition	Outcome
Older than 14 days from createdAt	delete — the ceiling wins over everything
pending with retries left	retain — the upload may yet succeed
retainUntil in the future	retain — the researcher has not been told
otherwise	delete — unchanged

Its own module and a pure function: it decides the least reversible thing in the codebase, and scheduled-upload-retry.ts pulls in the whole provider stack, so a test wanting just this predicate would have to load all of it.

retainUntil is written by the sweeper while a notification is undelivered, covering every unresolved entry for the experiment — datapipe.queueDocId records only the entry that tripped the episode, so extending that alone would leave the rest of the episode's files expiring on schedule.

The retention pass runs before the breaker check, so it works while sending is paused. Extending is a Firestore write costing no quota, and a quota outage is exactly when data ages toward deletion behind a notification that never arrived — putting it after the check would disable the protection in the only case that needs it.

No contact email → no extension, and it falls out rather than being special-cased. upload-failure-notify.ts records suppressedReason: "no-contact-email" and returns before enqueuing, so no mail document exists for the sweeper to find. Plain seven days, which is right when there's nobody to tell.

The 14-day ceiling is load-bearing. Without it, an experiment whose provider is dead and whose owner never reads their mail would hold research payloads forever, silently.

The warning banner — unverified, not missing

ContactEmailGate already walls off every admin route until a usable address exists, so "no contact email" is nearly extinct. What the gate doesn't check is whether the address works: hasContactEmail() tests format only, never contactEmailVerified.

That's the gap. A typo passes the gate. So does anything the 2026-08 backfill seeded from Firebase Auth. And upload-failure-notify.ts mails the address regardless of verified status — so it bounces and is marked terminally failed somewhere nobody looks. The symptom of a notification you can't receive is silence, which is indistinguishable from everything being fine.

UnverifiedEmailBanner renders on the dashboard when contactEmailVerified is false and an address exists. Not dismissible, no flag to maintain — contactEmailVerified is written server-side by verify-contact-email.ts and only there, so it removes itself the moment it's acted on.

Testing

81 suites, 1162 tests (10 new), plus tsc and lint.

Runbook §7 and §8 document the retention rules and the banner.

---

Third commit: close the ways mail could stop for a month, or stall forever

The first two commits assume the only way sending fails is a daily quota that resets at midnight. That assumption is wrong in three directions, and each one turns a recoverable outage into permanent loss.

Why sending stopped decides how long it stays stopped

The free plan has two caps — 100/day and 3,000/month — on different clocks. Reusing the daily reset for a monthly exhaustion is not a rounding error:

Hit the monthly cap on the 20th → the breaker reopens at midnight → the sweeper probes into a cap with eleven days left to run → fails → spends one of each queued document's MAX_ATTEMPTS doing it. Three nights of that and every queued notification is terminal — the exact retry-budget exhaustion the breaker exists to prevent.

PauseKind	Reset	Why
daily-quota	Next UTC midnight	Unchanged
monthly-quota	Start of next UTC month	Different clock, up to 30× longer
systemic	now + 15 min	Nothing is exhausted; no reset time exists

pauseKindFor() is the table that maps a Resend error name onto one, and both call sites go through it rather than deciding for themselves.

Pinning dailyQuotaUsed at the limit now happens only for the daily cap. A monthly exhaustion says nothing about today's counter, and pinning it there would block verification on the reserve rule for a reason that has nothing to do with the reserve — and go on doing so after the daily counter reset.

Systemic failures shut it too, briefly

A revoked key or an unverified domain is not exhaustion, and nothing shut on it. So verificationAvailability answered "available" forever while every click minted a code, wrote a document, spent a real Resend request and failed — with no server-side ceiling on how often. Tripping the breaker bounds that to one request per 15 minutes however many researchers are pressing the button.

validation_error is deliberately not in SYSTEMIC_ERRORS. Resend uses it both for "your sending domain is unverified" (systemic) and "this recipient address is malformed" (one researcher's typo). One typo must not switch verification off for everybody, so the ambiguous name stays out and the unambiguous ones carry the rule.

The header reading is a guess, and it is now bounded

Worth being explicit about, because it is load-bearing and it is not documented. Resend documents x-resend-daily-quota's existence, not its semantics — and "used today" and "remaining today" are the same shape.

If it is really the plan limit, every reading is 100, the reserve rule is true forever, and verification is off for good — and because each send rewrites dailyQuotaObservedAt, the staleness escape hatch never fires either. Three things bound that now:

1. usableQuotaReading() refuses anything outside [0, 100] at the write and ignores it at the read, logging loudly. That catches a header that turns out to be the monthly counter, or a remaining-quota value on a paid plan.
2. Refusing a verification on the reserve alone logs at ERROR with a stable token, so the condition is alertable rather than silent.
3. Runbook §5 says how to capture a real response header and check the number against the Resend dashboard. Do that before trusting it.

The verification record is the rate limit, so it stays

The first commit deleted it on a failed send, reasoning that a researcher whose code never arrived shouldn't wait out a cooldown for a code that doesn't exist. Right about the researcher, wrong about the endpoint:

sentAt is the only server-side throttle on that path, and deleting it removed the throttle from exactly the case that needs one. A failed send still costs a Resend request and still writes a mail document, so a signed-in researcher holding the button — or a loop on one valid ID token — drove both without any bound at all.

The record now survives with codeHash cleared: a cooldown, not a secret nobody received. verify-contact-email.ts answers "request a new one" rather than spending one of the five attempts on a code that was never sent. Two things make that safe rather than merely throttled — the systemic breaker above, and clearing the code itself.

Every document the sweep finds leaves by a door

The queries are unordered limit()s, so any document the pass can look at without changing, it will look at again next pass, forever — occupying the budget a deliverable notification needed. The decision table has no permanent "skip" left in it: every outcome either sends, or writes a terminal state that takes the document out of both queries. The only skips left are documents another invocation is actively holding, which resolve within LEASE_MS.

That matters beyond starvation. A retryable ERROR never gets delivery.expireAt, so it sits outside the TTL policy holding a researcher's address indefinitely. Ageing one out is how that address is finally deleted.

A second query finds stranded claims. deliverMailDocument's claim rewrites the document to PROCESSING with retryable: null before the send — so an instance that dies mid-send left a document that neither the retryable-ERROR query nor the TTL could see. A preempted instance or a mid-deploy roll stranded an address permanently. claimDecision already knew how to recover an expired lease; nothing ever asked it to.

The idempotency window now measures from startTime, which is when the key was first used and never moves. Measuring from the last attempt slid the window forward with every retry — so a document retried at +20h and again at +40h was sent on a key that expired at +24h, and Resend treated it as a new message. Exactly the double-send the window exists to prevent.

Retention moves next to the predicate that reads it

extendRetentionForExperiment lives in upload-retention.ts now. Splitting "may this data be destroyed, and what holds it back" across the mail sweeper, that module and the deletion sweep meant three files in two subsystems had to be read together to answer one question — and the mail subsystem had to know the uploadQueue's schema. The sweeper now says only what it actually knows ("this experiment's researcher has not been told"), and retention decides what that means.

Inverting it completely was considered and rejected. Asking "is a notification still undelivered?" at deletion time is the tidier shape, but it makes the least reversible operation in the codebase depend on a query that can fail — and a failing query there deletes data. retainUntil fails the other way: a write that doesn't happen costs an extension, not the data.

Two changes to the rule itself:

• It covers documents about to be aged out, not just sent. That's the more important half — giving up is the case where the researcher will never be told at all, and it must not also be the moment their data quietly goes back on the original clock.
• It skips a rewrite when the stored value is already more than half the grace window out. The sweeper runs every ten minutes for as long as a notification is undelivered, so without that floor a day-long outage with a couple of dozen queued notifications rewrites the same field tens of thousands of times to the same effect — against a 20,000/day free tier.

The deletion sweep pages past what it may not delete

cleanupOldEntries found entries by age, ascending, under one limit(50). But whether one may be destroyed is a separate question, and a retained entry is not removed from the result set by being looked at. Fifty retained entries at the head of the queue meant a pass that deleted nothing, forever: one experiment stuck behind a dead storage provider could stop every other experiment's payloads from being deleted, until the blockers crossed the 14-day ceiling up to a week later.

It now scans up to 500 with a cursor to find its 50. Same deletion budget, same retention rule — it just no longer mistakes a blocked head for an empty queue.

Also

• mailCollection(), so a suite can isolate the collection the sweep queries. Isolating the status document was never enough on its own — the collection is the other shared singleton, and a suite that swept would deliver another suite's fixtures through its own transport. The trigger still binds to MAIL_COLLECTION, which is deploy-time configuration a test may not move.
• One users/{uid} subscription on /admin, passed down, replacing two live listeners on the same document with their own loading flickers.
• leaseIsExpired, pauseKindFor and unresolvedQueueEntriesQuery each now own a rule that was written out twice.
• Age-out and retention writes run concurrently. A backlog was up to fifty serialized round-trips. Not batched — a batch is atomic, and one document purged mid-sweep would take the rest with it.
• Composite index for delivery.state + delivery.leaseExpiresAt.

Testing

81 suites, 1190 tests — 1158 passing, plus tsc and lint clean. 28 new tests over the second commit, covering the monthly and systemic pause kinds, the bounded header reading, stranded-claim recovery, the age-out terminal write, the retention rewrite floor, and the paging deletion sweep.

32 failures in 5 suites, all pre-existing and unrelated: collision-cache, providers-zenodo-oauth, resolve-token-gdrive, oauth-connect-refresh-emulator, connect-static-token-emulator. I confirmed these by checking the same suites out at the previous commit and running them there — identical failures, and none of the five touches any module this commit changes. Worth a look on their own, but not from this branch.

Runbook §5 and §6 gain the breaker's three reasons, the "check what that number means" procedure, and the note that the deletion sweep pages.