feat(ingest): mirror Tinybird writes into a second workspace

[Makisuo]

Why

The Tinybird workspace (api.tinybird.co — GCP Frankfurt) and the ingest fleet (AWS us-east-1) are on different clouds and continents, so every exported byte pays the public-internet egress rate rather than the same-region one. resolveAwsRegion in stage.ts:43 already spells out the stakes and ends with "Verify the instance's TINYBIRD_HOST before changing a mapping" — this is the first step of that change.

A single-step cutover would strand every read behind an empty warehouse: the new workspace's materialized views only build rollups from rows inserted into it. So the gateway needs to write both for a while, and a month of live dual-emit is what makes the new workspace readable at cutover.

This PR is the ingest half. Read cutover, JWT/signing-key migration, and history backfill are separate steps.

What changed

A third export lane, not a fan-out. ExportDestination::TinybirdMirror joins Tinybird and ClickHouse. The lane machinery already exists for exactly this reason — each destination gets its own WAL file, cursor, bounded channel and worker, so "a stalled destination can only back-pressure its own lane" (telemetry.rs:82). Two properties make this safe on a rolling deploy:

• Appending to ExportDestination::ALL leaves the existing lane ordinals (0, 1) untouched.
• WAL files are addressed by destination name (shard-000-tinybird_mirror.wal), not lane index, so bumping LANES_PER_SHARD 2→3 does not orphan in-flight files.

Wire tag 3 is documented as permanently reserved — a mirror-written WAL file must still decode under a rolled-back binary.

Best-effort is structural, not conventional. commit_frames splits into a fail-closed commit_frames_to_lane (unchanged behaviour, runs first) and a commit_mirror_frames that cannot return an error. A full lane, an exhausted org byte budget, or a WAL failure each meter to ingest_tinybird_mirror_dropped_total{reason} and continue. Because the primary commits first, it always wins the race for the last reservable bytes.

ClickHouse-routed orgs are never mirrored: a BYO-ClickHouse org's rows never reached the workspace being migrated and must not start reaching its replacement.

Mirror config carries its own endpoint/token, a smaller retry budget (5 vs the primary's 20) and a per-attempt timeout the primary deliberately lacks — a sick workspace should release its lane worker in seconds rather than hold a batch through the full budget. sample_percent ramps by org hash, so an org is consistently in or out; a percentage that split one org's stream would leave it half-mirrored, which no backfill boundary can describe.

Metrics gained a destination label on tinybird_export_{succeeded,dropped,retry} and on export_batch_completed / wal_exported_bytes. Comparing per-datasource row counts between the two workspaces is the acceptance test for the whole window, and without the label the mirror lane's drain was indistinguishable from the primary's in the same shard.

Feature is off by default. Present only when both TINYBIRD_MIRROR_HOST and TINYBIRD_MIRROR_TOKEN are set; half a config fails at startup rather than 401-ing a lane forever.

Reviewer notes

• Capacity is the thing to check before enabling. The per-lane WAL budget is queue_max_bytes / num_lanes, which just went /2 → /3 — so the primary Tinybird lane's disk share shrinks by a third at the same moment its traffic doubles. WAL_MAX_BYTES in alchemy.run.ts needs raising (within Fargate's ephemeral ceiling) for the duration of the window, and INGEST_ORG_QUEUE_MAX_BYTES roughly doubling. Comments record this at both constants.
• Mirror host/token resolve through optionalPlain, not process.env — alchemy reads --env-file/.env through its own ConfigProvider and never copies those into process.env, so a bare read works in CI and silently misses locally.
• During the window the fleet pays both egress rates. That is the price of the month.
• The CD matrix gains a production-us leg. Both production workspaces must receive every schema deploy for the whole window — a month of drift between them is the quiet way this fails. It no-ops until the environment's secrets exist.
• alert_checks is deliberately not mirrored (it is written by the API worker, not the gateway). It cuts over at the end and starts empty in the new workspace.

Rollout

Deploy with INGEST_TINYBIRD_MIRROR_SAMPLE_PERCENT=1, confirm rows land and ingest_tinybird_mirror_dropped_total stays flat, then ramp 1 → 10 → 100. The signal that decides whether this was safe is the primary lane's shed rate, which must not move at all.

Testing

cargo test --locked — 200 pass, 0 fail (122 lib + 78 bin), including 8 new tests:

• a_stalled_mirror_never_fails_the_accept_path — the load-bearing one. Mirror points at an unbound port, its lane backs up, and 20 accepts all still return Ok while the primary keeps receiving.
• mirror_lane_receives_the_same_rows_as_the_primary — two real HTTP servers, byte-identical gzip NDJSON, each authenticated with its own token.
• clickhouse_bound_rows_are_never_mirrored, destination_tag_round_trips_all_variants, mirror_lane_ordinals_stay_stable_when_a_destination_is_appended, mirror_sampling_is_all_or_nothing_per_org, mirror_config_rejects_half_configuration, tinybird_target_resolves_per_destination.

cargo clippy clean on both touched files; cargo fmt diffs limited to those already present at HEAD. bun typecheck and tsc -p tsconfig.alchemy.json pass.

Not run: the two-container tinybird-local end-to-end. Its unique value over mirror_lane_receives_the_same_rows_as_the_primary would be validating against real datasource schemas, which this change does not touch — same schema, different host. Worth running before the staging ramp.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[github-actions]

🍁 Maple PR preview

Warning

Preview cleanup could not be confirmed. The Alchemy teardown outcome was skipped.

Final commit b7d5072 · View workflow run