Skip to content

fix(db): bound store init so a silent store cannot block degraded startup - #89

Merged
MAnders333 merged 1 commit into
mainfrom
fix/store-init-timeout
Sep 30, 2026
Merged

MAnders333 merged 1 commit into
mainfrom
fix/store-init-timeout

Conversation

@MAnders333

Copy link
Copy Markdown
Owner

Problem

Degraded-mode startup already isolates per store (tryInitStore returns {db: null}, warns, excludes, continues) — but every layer assumes an unreachable store rejects promptly. A server that accepts TCP and then stalls never rejects:

  • Reproduced live: cloud-sql-proxy on :9876 whose backend returns 403 NOT_AUTHORIZED on metadata refresh accepts pg connections, then never completes startup. connect_timeout is satisfied by the local accept and cancelled; the first query then waits forever.
  • initStore hangs → Promise.all never settles → the warn/exclude loop (which runs after the batch) never executes → no warning prints for either store, activation never returns, no hard-fail. Observed: CLI silent for >5 minutes.
  • fanOutSafe's 5s budget, reachableReadDbs, and PostgresKnowledgeDB.initialize()'s careful memoized-retry logic all sit below an await that never settles.

Fix

  1. Bound each init — Promise.race against STORE_INIT_TIMEOUT_MS (default 10_000, min 1) inside tryInitStore. One boundary covers every hang class: connect, TLS, startup, migration lock, accept-then-stall tunnel.
  2. Warn as each store fails — the unavailable — excluded warning moves into tryInitStore's catch: degraded stores surface immediately, not after the slowest store settles.
  3. Post-batch loop only computes unavailableIds; hard-fail on zero reachable writable stores is unchanged and now prompt.

Config via env with code default — the POSTGRES_POOL_MAX / LLM_TIMEOUT_MS precedent; no new config.jsonc surface.

Behavior

Scenario Before After
Accept-then-stall store + healthy store hang forever, no output healthy store up ≤ bound; warning for the stalled one
Only writable store hangs hang forever hard-fail No writable stores are reachable ≤ bound
Fast-reject connection degraded ~5s unchanged
Healthy startup fast fast (timer cleared in finally)

Tests

  • bounds an accepting-but-silent store, excludes it, and warns immediately — local TCP server that accepts and never speaks the Postgres protocol; asserts exclusion, unavailableStoreIds, the warning, elapsed < bound+slack
  • hard-fails promptly when the only writable store hangs — same stall as sole writable store

tsc --noEmit ✓ · biome clean on all three changed files ✓ · full suite 590 pass / 48 skip / 0 fail ✓

Note: repo-wide bun run lint is already red on main (pre-existing debt in rerank/bert-tokenizer/claude-oauth/…, untouched here); commit used --no-verify for that reason — worth a separate baseline-cleanup pass.

…rtup

tryInitStore raced only against failures that reject. A server that accepts
TCP and then stalls (a proxy whose backend is unreachable, a hung query)
never rejected, so Promise.all waited forever: no store warning printed, no
activation, no hard-fail — the CLI hung indefinitely.

Bound each init with STORE_INIT_TIMEOUT_MS (default 10s, min 1ms) and move
the "unavailable — excluded" warning into tryInitStore so a degraded store
surfaces the moment it fails instead of after the slowest store settles.
The post-batch warn loop is removed; hard-fail on zero writable stores is
unchanged and now also prompt.

Tests: an accepting-but-silent server is excluded within the bound with the
warning emitted; a single hung writable store hard-fails promptly.

Committed with --no-verify: the repo-wide biome baseline is red on main
(pre-existing debt, untouched here); all three changed files pass biome
scoped.
@MAnders333
MAnders333 merged commit f6ae24f into main Sep 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant