Reduce API read-replica recovery conflict storms

[raymondjacobson]

Summary

• cap each read-replica pool at 8 connections by default, configurable with readDbMaxConns
• add a concurrent partial covering index for track comment list and count reads

Why

The production latency monitor is flapping because the sole read replica cancels waves of queries with:

canceling statement due to conflict with recovery (SQLSTATE 40001)

An Axiom query for 2026-08-03 15:42 UTC found 156 recovery-conflict cancellations in one minute. The corresponding 500s affected unrelated routes, which points to shared replica pressure rather than an individual handler. Track comment reads were the largest visible amplifier in that window:

• /v1/tracks/:trackId/comment_count: 54 errors
• /v1/tracks/:trackId/comments: 38 errors
• all other routes combined: 64 errors

Production currently has three bridge pods and one four-vCPU read replica. Without an explicit limit, pgx sizes each pool from the 16 CPUs visible to the pod, allowing up to 48 concurrent request reads. This change halves that connection budget to 24 while retaining an environment override.

The comments table has no index beginning with entity_id, so both high-volume track comment routes scan the table before their moderation joins. The new partial index supports both the filter and the default timestamp order without blocking writes during creation.

Validation

• go test ./api/dbv1 -run 'Test(NewDBPools|ChooseReplica|DBPoolsProxyMethods)$' -count=1
• go test ./config -count=1
• go test ./indexer -run '^$' -count=1
• go test ./api -run '^$' -count=1
• applied the migration twice against the checked-in schema to verify idempotency
• verified PostgreSQL selects the new index for track comment list and count predicates

Rollout

The connection cap can be changed without another code change by setting readDbMaxConns. Pool wait time and replica CPU should be watched during rollout. The expected tradeoff is bounded application-side queuing instead of allowing enough concurrent work to saturate the replica and trigger a recovery-conflict wave.

Non-goals

This PR does not change standby feedback, statement timeouts, query retry behavior, replica capacity, or Kubernetes resources. Those have broader operational tradeoffs and should be reviewed separately.