Fix the failures shown on /status: DDNet imports, rollup deadlocks and stalls - #38
Open
needs wants to merge 2 commits into
Open
Fix the failures shown on /status: DDNet imports, rollup deadlocks and stalls#38needs wants to merge 2 commits into
needs wants to merge 2 commits into
Conversation
- Parse the DDNet stats dump with MySQL-style backslash escapes; every backfill tick died on a player name containing an escaped quote. - Tolerate out-of-order days in the DDNet online CSVs, which finalized the same day twice and hit the (day, kind, key) unique constraint. - Close the dump download and the online CSV reader when an import aborts. - Serialize rollup writers with an advisory lock: the hourly day rollup and the midnight backfill deadlocked on Player and PlayerPartner rows. - Give the map count status check an hourly window instead of 10 minutes. - Prefix the game type count dedup id so an empty game type name schedules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Since late August every hourly rollup-day job ran out its 10 minute budget with the database and the worker both idle: each 2000-snapshot batch waited seconds for a pooled connection behind the hundred concurrent poll transactions. Give the rollup reads their own two-connection pool and log progress so the next stall is measurable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
teerank.io/status showed the DDNet import with 1259 failed jobs and no backfill started, the rollup with 214 failed jobs and the latest day two days late, and the map count as down. The failed-job reasons in Redis point at four independent defects, all fixed here.
teamrace.csv(a player namedTobias\"). The dump is a MySQL export that escapes quotes and backslashes with a backslash, whilecsv-parseexpected"". The parser now usesescape: '\\'(the dump contains no unescaped backslashes, verified over the full 494 MBteamrace.csv), with a regression test on real dump lines.(day, kind, key)unique constraint. Thebycountry/bymodfiles contain a handful of samples logged after the next day already began (e.g.2021-10-21 23:58after2021-10-22 00:00), so the same day was finalized twice in one batch. Days are now kept open until flushed and a stray sample for an already-closed day is dropped. Regression test added.rollup-dayand therollup-backfillscan that runs at 00:06 write the samePlayerandPlayerPartnerrows for different days; theUPDATE ... FROM unnest()and upsert do not lock rows in a guaranteed order, so they deadlocked most nights. Rollup writers now take a transaction-scoped advisory lock (lockRollupWrite.sql), and the transaction timeout grows to 15 minutes to cover waiting.Also:
game-type-countenqueue fail on an empty dedup id; the id is now prefixed.Not changed, worth knowing
ddnet-backfill,ddnet-online,rollup-dayandrollup-backfillfailed sets after the deploy.GameServerSnapshot_pkey5.2 GB,..._gameServerId_createdAt_idx7.3 GB,GameServerClient_snapshotId_idx3.2 GB, unusedGameServerClient_pkey6.3 GB) left over from the archive drain.REINDEX INDEX CONCURRENTLYon them would reclaim the space and cut the I/O wait the Postgres health check reports, but it is a manual maintenance step, not part of this PR.Test plan
nx run-many -t lint build test(pre-commit) passes, 58 worker tests including the two new regression testscsv-parsewithescape: '\\'verified against the real dump lines that broke productionddnet-backfillenters the download phase and progresses,ddnet-onlinecompletes,rollup-dayfor yesterday completes within budget, no new deadlocks at 00:06🤖 Generated with Claude Code