Skip to content

Fix the failures shown on /status: DDNet imports, rollup deadlocks and stalls - #38

Open
needs wants to merge 2 commits into
masterfrom
diagnose-status-page
Open

Fix the failures shown on /status: DDNet imports, rollup deadlocks and stalls#38
needs wants to merge 2 commits into
masterfrom
diagnose-status-page

Conversation

@needs

@needs needs commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Summary

teerank.io/status showed the DDNet import with 1259 failed jobs and no backfill started, the rollup with 214 failed jobs and the latest day two days late, and the map count as down. The failed-job reasons in Redis point at four independent defects, all fixed here.

  • DDNet backfill never started. Every 15-minute tick died in the CSV parser on line 3664 of teamrace.csv (a player named Tobias\"). The dump is a MySQL export that escapes quotes and backslashes with a backslash, while csv-parse expected "". The parser now uses escape: '\\' (the dump contains no unescaped backslashes, verified over the full 494 MB teamrace.csv), with a regression test on real dump lines.
  • DDNet online import failed every hour on the (day, kind, key) unique constraint. The bycountry/bymod files contain a handful of samples logged after the next day already began (e.g. 2021-10-21 23:58 after 2021-10-22 00:00), so the same day was finalized twice in one batch. Days are now kept open until flushed and a stray sample for an already-closed day is dropped. Regression test added.
  • Rollup deadlocks at midnight. The hourly rollup-day and the rollup-backfill scan that runs at 00:06 write the same Player and PlayerPartner rows for different days; the UPDATE ... FROM unnest() and upsert do not lock rows in a guaranteed order, so they deadlocked most nights. Rollup writers now take a transaction-scoped advisory lock (lockRollupWrite.sql), and the transaction timeout grows to 15 minutes to cover waiting.
  • Rollup-day stalls since Aug 30. Every hourly attempt hit the 10-minute budget. The database side of a full day costs a few seconds and the worker sat idle (flat RSS, ~10% CPU) while only ~50 of ~140 batches got through, i.e. each batch waited seconds for a pooled connection behind the 100 concurrent poll transactions on the same default 9-connection pool. The rollup reads now go through a dedicated two-connection Prisma pool, and the job logs progress and reports how far it got when it runs out of budget.

Also:

  • Aborted DDNet imports now close the dump download and the CSV reader instead of leaving them open (the worker process had ~520 open fds after 1000 failed ticks).
  • The status page gave the hourly map count job a 10-minute staleness window, so it showed "Down" 50 minutes of every hour; the window is now 75 minutes.
  • A game type with an empty name made the scheduler's game-type-count enqueue fail on an empty dedup id; the id is now prefixed.

Not changed, worth knowing

  • The BullMQ failed-job sets are kept by count, so the status counters will not drop on their own once this deploys; clear the ddnet-backfill, ddnet-online, rollup-day and rollup-backfill failed sets after the deploy.
  • The snapshot tables are ~74 MB of heap under ~22 GB of bloated indexes (GameServerSnapshot_pkey 5.2 GB, ..._gameServerId_createdAt_idx 7.3 GB, GameServerClient_snapshotId_idx 3.2 GB, unused GameServerClient_pkey 6.3 GB) left over from the archive drain. REINDEX INDEX CONCURRENTLY on them would reclaim the space and cut the I/O wait the Postgres health check reports, but it is a manual maintenance step, not part of this PR.
  • The 225-day "backfill gap" in the rollup section is days with no archived snapshots at all; nothing can fill them.

Test plan

  • nx run-many -t lint build test (pre-commit) passes, 58 worker tests including the two new regression tests
  • csv-parse with escape: '\\' verified against the real dump lines that broke production
  • After deploy: ddnet-backfill enters the download phase and progresses, ddnet-online completes, rollup-day for yesterday completes within budget, no new deadlocks at 00:06

🤖 Generated with Claude Code

needs and others added 2 commits September 8, 2026 13:50
- Parse the DDNet stats dump with MySQL-style backslash escapes; every
  backfill tick died on a player name containing an escaped quote.
- Tolerate out-of-order days in the DDNet online CSVs, which finalized the
  same day twice and hit the (day, kind, key) unique constraint.
- Close the dump download and the online CSV reader when an import aborts.
- Serialize rollup writers with an advisory lock: the hourly day rollup and
  the midnight backfill deadlocked on Player and PlayerPartner rows.
- Give the map count status check an hourly window instead of 10 minutes.
- Prefix the game type count dedup id so an empty game type name schedules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Since late August every hourly rollup-day job ran out its 10 minute budget
with the database and the worker both idle: each 2000-snapshot batch waited
seconds for a pooled connection behind the hundred concurrent poll
transactions. Give the rollup reads their own two-connection pool and log
progress so the next stall is measurable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant