Skip to content

[BUG] A failed Oracle connect orphans its pool, which retries forever and pins a CPU core #1102

Description

@cevheri

When OracleProvider.connect() fails after the pool was created, the pool is never closed. node-oracledb's thin pool then keeps trying to reach poolMin connections in a background loop, forever, with no backoff. Nothing holds a reference to it any more, so nothing can stop it. One failed connect to an unreachable Oracle host pins a CPU core and floods that host with TCP connection attempts until the process exits.

Measured

Measured 2026-09-23, node-oracledb 6.10.0 (thin mode), nothing listening on 127.0.0.1:1521.

Plain Node process, no Next.js. oracledb.createPool({ poolMin: 2, poolMax: 10, ... }), then pool.getConnection() rejects with NJS-503: connection to host 127.0.0.1 port 1521 could not be established. The pool reference is then dropped, exactly as the provider does. net.Socket.prototype.connect was counted for the next 5 seconds:

Window TCP connect attempts CPU time
5 s after the failed getConnection() 71,383 (about 14,000 per second) 5,447 ms (one full core)
1 s after pool.close(0) 0 -

So this is not a development-only problem: in production every orphaned pool is a spinning core and a connection flood toward the Oracle host (or a firewall in front of it).

Under next dev it also leaks memory. A bun dev server grew to 35 GB RSS in 13 minutes at about 137% CPU. A forced full GC freed only 0.3 GB of 23 GB of used heap, so it was retained. A 30 second sampling heap profile put the retained allocations in oracledb/lib/thin/pool.js bgThreadFunc, ThinConnectionImpl.connect and networkSession.connect, each captured by the async-hooks init hook of next/dist/compiled/next-server/app-page-turbo.runtime.dev.js, which keeps a stack trace per promise and socket (about 1 GB per 30 s). next dev sets --max-old-space-size to half of system RAM (32004 MB on a 64 GB machine), so nothing fails early and the machine runs out of memory first.

How it was hit

Log in with a seed config that contains an Oracle connection whose host is down, and open /admin (a logged-in visit to /login redirects there). POST /api/admin/fleet-health calls getOrCreateProvider() for every seed connection, so every unreachable Oracle seed orphans one pool per call. Two Oracle seeds (oracle-local, task30-oracle) were enough. Any other path that ends in getOrCreateProvider() for an unreachable Oracle connection does the same: health, query, object routes, the agent.

Cause

src/lib/db/providers/sql/oracle.ts, OracleProvider.connect():

  1. this.pool = await oracledb.createPool({ poolMin: this.poolConfig.min, ... }) succeeds. Thin mode does not connect here, and poolMin defaults to 2 (DEFAULT_POOL_CONFIG in src/lib/db/types.ts).
  2. The test borrow await this.pool.getConnection() rejects.
  3. The catch sets the error and throws ConnectionError (or DatabaseConfigError for NJS-138) without closing this.pool.
  4. getOrCreateProvider() in src/lib/db/factory.ts rethrows and never caches a provider whose connect() failed, so no later disconnect() can reach the pool.
  5. bgThreadFunc() in node_modules/oracledb/lib/thin/pool.js loops "until a close request is received". On a failed connection it stores _bgErr and tries again after a setImmediate, with no delay.

There is a second, smaller defect from the same line: this.pool stays set after the failure, so a later connect() on the same instance hits the if (this.pool) return; guard and returns without connecting and without an error.

Precedent in this repository

PostgreSQL and SQL Server already fixed this exact shape, with the reason in a comment. PostgresProvider.connect() (src/lib/db/providers/sql/postgres.ts) does, in its catch:

const failedPool = this.pool;
this.pool = null;
await failedPool?.end().catch(() => {});

MSSQLProvider.connect() does the same with its ConnectionPool. Oracle should follow that pattern, with close(0) as the oracledb equivalent of end().

Scope

  • In scope: OracleProvider.connect() and its tests, plus the provider doc.
  • Not affected, measured: MySQL. mysql2's pool has no background creator; after a failed getConnection() it made 0 connection attempts and used 0.3 ms of CPU in 3 s. Do not change mysql.ts in this fix.
  • Already fixed: PostgreSQL, SQL Server.
  • Out of scope: adding backoff inside oracledb, changing fleet-health, changing DEFAULT_POOL_CONFIG, or making the factory dispose providers. The provider must clean up after itself, the same contract PostgreSQL and SQL Server follow.

Acceptance criteria

  1. When connect() fails at any step after createPool() resolved, the pool is closed with close(0) before the error leaves connect(). This covers both the ConnectionError path and the NJS-138 DatabaseConfigError path.
  2. After a failed connect(), this.pool is null and isConnected() is false, so calling connect() again on the same instance runs createPool() again instead of returning early.
  3. If close(0) itself rejects, the caller still receives the original connect error (same type, same message). The close failure must not replace or hide it.
  4. When createPool() itself rejects, behaviour is unchanged (there is nothing to close).
  5. A successful connect() is unchanged.
  6. The comment in the catch states why the pool is closed there, the way the PostgreSQL provider's comment does.
  7. docs/providers/oracle.md records that a failed connect closes its pool, per the provider triad rule in CLAUDE.md (code, docs and tests change together).

Test requirement

Write the failing tests first, in tests/integration/db/oracle-provider.test.ts, using its existing mockCreatePoolFn / mockPoolCloseFn doubles:

  • createPool resolves, getConnection rejects: assert ConnectionError, assert the pool's close was called once with 0, assert isConnected() is false.
  • Same setup with an NJS-138 message: assert DatabaseConfigError and that close(0) was called.
  • Same setup, then a second connect() where getConnection resolves: assert createPool was called twice and isConnected() is true.
  • getConnection rejects and close also rejects: assert the original error type and message reach the caller.

Each new test must fail on current main before the fix. The 100% line coverage gate applies (bun run test:coverage && bun run coverage:check).

Manual verification

With no Oracle running, run a script that calls new OracleProvider({ type: "oracle", host: "127.0.0.1", port: 1521, ... }).connect(), catch the error, then count net.Socket.prototype.connect calls and process.cpuUsage() for 5 seconds. Before the fix: tens of thousands of attempts and about one core of CPU. After the fix: 0 attempts and near-zero CPU.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdatabase-providerhelp wantedExtra attention is neededsecuritySupply-chain, auth, or hardening work

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions