Send transactional mail through Resend, not SES - #211
Merged
Conversation
AWS denied the SES production-access request, which would have left every
send restricted to verified recipient addresses -- failing for every real
researcher, and terminally, since mail-delivery.ts classifies a rejected
recipient as permanent. Resend has no equivalent gate.
Only the transport moved. mail.ts's document contract, the claim/lease
machinery, the delivery.* outcome fields, the TTL policy and
purge-user-data.ts's owner query are all untouched, which is what kept this
to one file plus its tests -- the same property that made the earlier
extension-to-SES swap cheap.
- buildSendEmailInput emits Resend's JSON body. It uses reply_to, not
replyTo: the SDK takes camelCase, the raw REST API takes snake_case and
ignores unknown keys silently, so getting it wrong would mean every
notification going out with no Reply-To and nothing saying so. Pinned by
its own test.
- classifySesError -> classifyMailError, rebuilt on Resend's error codes
with HTTP status as the fallback.
- The transport is one fetch to one endpoint. @aws-sdk/client-sesv2 is
gone, and with it the dynamic import that existed to keep the SDK off
apidata's cold-start path.
- Undici reports every network failure as `TypeError: fetch failed` with
the real diagnosis on .cause, so errorName unwraps it. Without that, a
momentary DNS blip classifies as unrecognised, which is terminal, which
silently loses a notification.
One behavioural change, deliberate: ambiguous failures (timeouts,
ECONNRESET) are now retryable. They were terminal because SESv2 has no
idempotency token, so a request that went out and never answered could not
be retried without risking a second copy of a notification whose whole value
is arriving once. Resend takes an Idempotency-Key; we send the mail
document's id on every attempt, so the retry is a no-op at Resend rather
than a second send. The key expires after 24h, which bounds any future retry
mechanism -- noted in AMBIGUOUS_ERRORS and in both test suites.
Six *_SES_* repo secrets become one *_RESEND_API_KEY per environment. The
test site is now expected to send: it is the only place delivery is
exercised before production, since the emulator short-circuits before
sending and the unit suites mock the transport. Reputation and daily quota
are shared with prod -- documented in runbook 2(d).
Also corrects three things found while verifying the deploy steps, none of
them caused by this change:
- The production project is osf-relay, not datapipe-prod (.firebaserc's
default, and what `firebase use default` selects). GCP project ids are
immutable, so datapipe-prod never existed; the runbook and three
migration scripts had it wrong.
- Runbook 4's console instructions named delivery.endTime where the gcloud
command says delivery.expireAt. Following the console path would create
a policy that reaps every mail document the moment it is delivered.
- mail.ts's header still described the extension as the transport and
claimed switching providers was a config change rather than a redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL
jodeleeuw
added a commit
that referenced
this pull request
Aug 29, 2026
The TTL on mail/delivery.expireAt was created by hand with gcloud, as the runbook told us to. The next deploy deleted it: 14:51 gcloud ... --enable-ttl --project=datapipe-test -> ACTIVE 15:02 PR #211 merged to `test` 15:04 firestore: Deleting 1 field overrides... <- the deploy 15:10 TTL gone `firebase deploy --only firestore` reconciles field overrides against firestore.indexes.json, and that file said `"fieldOverrides": []` -- so a hand-made TTL was not merely unmanaged, it was something the deploy was actively instructed to delete. --force suppressed the prompt that would have said so, and the deploy reported success. The retention promise stops being kept, silently, while the runbook says it is. Declared in firestore.indexes.json instead, so it is version-controlled and recreated by the same deploy that would otherwise remove it. The block keeps the three default single-field indexes: a fieldOverride replaces the field's whole index configuration, so `"indexes": []` would additionally turn off single-field indexing for delivery.expireAt. What is committed here is what `firebase firestore:indexes` emits for a field with TTL on and default indexing, read off osf-relay while its policy was still active, so it round-trips. Runbook 4 rewritten around the declarative mechanism, keeping the timeline above -- following the old instructions produced a policy that survived exactly until the next deploy, and that is worth being unable to rediscover the hard way. Also corrects two comments in mail-delivery.ts that said the TTL keys on delivery.endTime. It keys on delivery.expireAt; endTime only gates whether expireAt gets written. That same confusion is what put the wrong field in runbook 4's console instructions. Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS denied the SES production-access request. That would have left every send restricted to verified recipient addresses — failing for every real researcher, and failing terminally, since
mail-delivery.tsclassifies a rejected recipient as permanent and does not retry it. Resend has no equivalent gate, so the transport moves there.Only the transport moved
mail.ts's document contract, the claim/lease machinery, thedelivery.*outcome fields, the Firestore TTL policy andpurge-user-data.ts's owner query are all untouched. That's what kept this to one file plus its tests — the same property that made the earlier extension→SES swap cheap.buildSendEmailInputemits Resend's JSON body, usingreply_to, notreplyTo. The Node SDK takes camelCase; the raw REST API takes snake_case and ignores unknown keys silently — so getting this wrong means every notification goes out with no Reply-To and nothing says so. Pinned by its own test.classifySesError→classifyMailError, rebuilt on Resend's documented error codes with HTTP status as the fallback.fetchto one endpoint.@aws-sdk/client-sesv2is gone, and with it the dynamic import that existed to keep the SDK offapidata's cold-start path.TypeError: fetch failedwith the real diagnosis on.cause, soerrorNameunwraps it. Without that, a momentary DNS blip classifies as unrecognised → terminal → a silently lost notification. Tested in the wrapped form, since that's the only form the code ever receives.One behavioural change, deliberate
Ambiguous failures (timeouts,
ECONNRESET) are now retryable. They were terminal because SESv2 has no idempotency token: a request that went out and never answered couldn't be retried without risking a second copy of a notification whose entire value is arriving once, so the old code chose to lose the mail.Resend takes an
Idempotency-Key, and we send the mail document's id on every attempt — so the retry is a no-op at Resend rather than a second send. We get the retry and exactly-once instead of choosing.The key expires after 24h. Every retry path here is minutes wide, but that bounds any future retry mechanism, and it's called out in
AMBIGUOUS_ERRORSand in both test suites so a later change can't quietly invalidate it.Config
Six
*_SES_*repo secrets become one*_RESEND_API_KEYper environment.The test site is now expected to send, reversing this runbook's earlier "leave it unset" default. Reason: the test site is the only place delivery is exercised before production — the emulator short-circuits before sending, and the unit suites mock the transport at a function seam. Reputation and daily quota are shared with prod, which is documented in runbook §2(d) along with what to actually smoke-test.
Also corrects three things found while verifying the deploy steps
None caused by this change:
osf-relay, notdatapipe-prod—.firebaserc's default, and whatfirebase use defaultselects. GCP project ids are immutable, sodatapipe-prodnever existed. The runbook and three migration scripts all had it wrong, including the backfill'sFIREBASE_PROJECT_ID || 'datapipe-prod'default.delivery.endTimewhere the gcloud command saysdelivery.expireAt. Following the console path creates a policy that reaps every mail document the moment it's delivered — no debugging window, and nothing saying it happened.mail.ts's header still described the extension as the transport and claimed switching providers was a config change rather than a redeploy. Now also explains why the collection exists at all (thetx.createinside the episode transaction) and why the TTL does.Testing
Full suite green on the rebased tree: 79 suites, 1119 tests, plus
tscand lint.The two mail suites were rewritten rather than patched — 48 pure tests (request mapping, error taxonomy, claim state machine, config) and 19 emulator-backed ones. No test in the repo can reach
api.resend.com: the transport is injected at a function seam, so the real one is never constructed.Deploy prerequisites — status
jspsych.orgverified in ResendPROD_RESEND_API_KEY/TEST_RESEND_API_KEYsetmailTTL policy ondatapipe-test—delivery.expireAt, stateACTIVEmailTTL policy onosf-relay— still absent, needed before prodmaildocument reachesdelivery.state: "SUCCESS"withdelivery.info.transport: "resend"Known gap, not addressed here
Hitting the daily quota leaves the mail document as a retryable
ERRORthat nothing retries on a timer — andupload-failure-notify.tssetsnotifiedAtin the same transaction that enqueues, so the episode is already armed as "told them." The researcher is never notified and the system believes it did. Closing that needs a scheduled sweeper, which is separable from this change and has its own design question (it must be kind-aware, and it collides with the 24h idempotency window).🤖 Generated with Claude Code