deps: pin pkg/etl v1.6.3 - #1009
Closed
rickyrombo wants to merge 1 commit into
Closed
Conversation
Moves off the 2026-07-27 pseudo-version onto the real tag. Two changes come
with it, both pkg/etl-only, so the root module pin is untouched:
* album saves and reposts now record save_type/repost_type 'playlist'
rather than deriving 'album' from the mutable playlists.is_album at index
time (OpenAudio/go-openaudio#428). The 670 saves and 528 reposts already
written as 'album' are backfilled separately by ddl 0236; that backfill
only sticks once this bump is deployed, or the old indexer rewrites them.
* one current row per user is now enforced, and the fifteen DISTINCT ON
dedupe subqueries in genesis-writer collapse to plain joins
(OpenAudio/go-openaudio#425).
Also corrects the comment above the ETL config. It claimed ETL migrations
are safe to auto-run because they are purely additive DDL. That is no longer
true: 0035 deletes duplicate is_current rows from users before adding a
unique index over them, and since SkipMigrations is left false, it runs
against api's database on the next indexer start. That is intended — it is
why no separate api migration is needed to purge the duplicates — but a
pkg/etl bump can now modify production data, which is worth saying out loud
at the call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rickyrombo
marked this pull request as draft
August 4, 2026 23:48
Contributor
Author
|
Superseded by #1011, which combines this with the other two api-side changes. They're only correct together in one deploy — merging the module bump without the users backfill would fail ETL migration 0035's index creation and stop the indexer starting. |
rickyrombo
added a commit
that referenced
this pull request
Aug 5, 2026
Supersedes #1008, #1009 and #1010, which were the same work split three ways. ## Why one PR These three changes are only correct **together, in one deploy**. Split, each one alone breaks something: | merged alone | result | |---|---| | the bump | ETL `0035` creates `users_current_uniq_idx`, **fails on the existing duplicates**, `RunMigrations` errors and the indexer won't start | | `0236` (album) | the old indexer keeps deriving `album` from `is_album` and undoes the backfill | | `0237` (users) | harmless, but pointless without the index that stops it recurring | As one PR the deploy is atomic, and the ordering inside it is guaranteed by existing machinery: `bridge migrate` runs as a pre-roll Job that every serving Deployment `DependsOn` (serving pods get `runMigrations=false`), so both ddl migrations complete before the indexer starts and runs the ETL's. ## Contents **`0237_users_one_current_row_backfill`** — deletes 5 duplicate `is_current` rows from `users`. Small count, large blast radius: joins from an entity to its owner's wallet fan out, measured at **+18 tracks and +787 follows** on a production clone. Must precede the ETL index. **`0236_saves_reposts_album_to_playlist`** — 670 saves and 528 reposts written as `album` collapse to `playlist`. `on_save`/`on_repost` are disabled for the update (their notification `group_id` embeds the type, so a plain UPDATE would mint duplicate favourite notifications); `trg_saves`/`trg_reposts` stay enabled so the search indexer sees the change. **`deps: pin pkg/etl v1.6.4`** — brings OpenAudio/go-openaudio#428 (album type), #425 (the `users` invariant + genesis-writer join simplification) and #433 (`0035` no longer deletes anything). ## The delete moved out of the ETL migration `v1.6.3`'s `0035` deleted the duplicates itself. Since ETL migrations run automatically at indexer start, that made a `go get` able to remove rows from this database. #433 split it: the index stays in the ETL migration, the repair moved to `0237` here. **`v1.6.4` ships zero `DELETE` statements** — verified against the resolved module, not just the tag. The comment above the ETL config now records that line, and its corollary: an ETL migration can depend on a ddl one having run, and `0035` fails loudly if `0237` hasn't. ## Verified - Resolved module `pkg/etl@v1.6.4` contains `0035` with `CREATE UNIQUE INDEX` and **0** `DELETE` statements. - Both migration orders against fixtures: backfill→index applies cleanly (`violations 0`, `indisvalid = t`); index→backfill fails with `could not create unique index … Key (user_id)=(98311147) is duplicated`, which is the intended signal that `0237` hasn't run. - Both migrations idempotent; re-running is a no-op. - `0236` fires **zero** `on_save`/`on_repost` triggers against a fixture with the real wiring, and exactly one `pg_notify` per updated row. - `go build ./...` and `go vet ./indexer/` clean. - No FK references `users`; its triggers are INSERT / INSERT OR UPDATE, so the delete fires neither. - Cutting `pkg/etl/v1.6.4` did not move `openaudio/go-openaudio:stable` — still the 2026-07-30 `v1.8.2` digest, so no node-operator rollout. ## Not established The cause of the duplicate `users` rows. Both indexer create paths reject an existing user, so a single writer can't produce them; a second writer can, since check-then-act isn't atomic across transactions. Three of five pairs put a bare-hex `txhash` next to a `0x`-prefixed one, which fits but doesn't prove it. The index will surface it if it recurs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Blocked: repin to
v1.6.4before merging.v1.6.3still contains the delete-and-index version of ETL migration0035, so merging this as-is would ship a data-deleting ETL migration to production — the exact thing OpenAudio/go-openaudio#433 exists to prevent. The pre-roll backfill (#1010) would make that delete a no-op, but ago getstill should not be able to remove rows at all.Sequence:
chore(pkg/etl): release 1.6.4PRv1.6.4Moves off the 2026-07-27 pseudo-version onto the real tag.
Both changes it brings are
pkg/etl-only, so the root module pin is untouched. (golang.org/x/netmoves between the direct and indirect blocks — same version, ago mod tidyreclassification.)What it brings
OpenAudio/go-openaudio#428 — album saves/reposts record
save_type/repost_typeas'playlist'instead of deriving'album'from the mutableplaylists.is_albumat index time. Pairs with #1008, which backfills the 670 saves and 528 reposts already written as'album'.OpenAudio/go-openaudio#425 — one current row per user is enforced, and genesis-writer's fifteen
DISTINCT ONdedupe subqueries collapse to plain joins.SkipMigrationsis leftfalse, so bumping the module runs the migrations it brought against api's database on the next indexer start (etl_db_migrations, separate fromschema_version; up-migrations only).Until now that was benign — every ETL migration was additive
IF NOT EXISTSDDL, and the comment at the call site said so.0035is the first that is not: it deletes duplicateis_currentrows fromusersbefore adding a unique index over them (5 rows on a production clone of 3.15M). Still idempotent, and this is intended — it is why no separate api migration is needed to purge the duplicates — but the comment is now corrected to say apkg/etlbump can modify production data.0035builds a unique index onusersnon-concurrently (golang-migrate wraps each migration in a transaction), so expect a brief lock onusersat the next indexer start.Ordering with #1008
Merge this before or with #1008. #1008 backfills
album→playlist; if the old indexer is still deployed it re-derivesalbumand undoes the backfill. The reverse order is fine — this bump alone just stops newalbumrows being written.The two touch different tables (
0035: users,0236: saves/reposts) and run from different processes (indexer start vs api server start), so there is no interaction between them.Verified
go build ./...andgo vet ./indexer/clean.resolveSaveTypeno longer readsis_album) and ships0035_users_one_current_row.{up,down}.sql.pkg/etl/v1.6.3contains both [PE-6997] Update track access token gate check to use token decimals #425 and Bettermake test-schema#428 (checked by ancestry, not by changelog).Unrelated oddity spotted
pkg/etl/go.moddeclaresrequire github.com/OpenAudio/go-openaudio v1.6.3 // x-release-please-version— release-please rewrites that line with the pkg/etl version, so it points at an unrelated old root release. MVS resolves upward to our root pin so nothing breaks, but the declared requirement is meaningless. Worth fixing upstream.🤖 Generated with Claude Code