Skip to content

fix: RedisCheck can now back a readiness probe - #85

Open
RomainLanz wants to merge 1 commit into
11.xfrom
fix/redis-check-timeout
Open

RomainLanz wants to merge 1 commit into
11.xfrom
fix/redis-check-timeout

Conversation

@RomainLanz

Copy link
Copy Markdown
Member

RedisCheck could not be used as a readiness probe. We found this when Tenkai added Valkey to its /readyz after a production outage: pods were marked ready while Valkey was unreachable. In v11.0.0:

  • ping() on a ready connection had no time bound. When the server accepts the connection but does not answer (dropped packets, half-open TCP), the check waited until the socket failed.
  • While connecting, the check waited up to 3 × 1 s. You could not configure this wait, and it is longer than a typical probe budget (Kubernetes timeoutSeconds defaults to 1 s).
  • #pingAttempts was never reset. HealthChecks keeps one check instance for the life of the process, so after 3 waits in total the check never waited for a connecting connection again.
  • On a closed connection, the check called ioConnection.connect(): the health check reconnected the connection as a side effect.

RedisMemoryUsageCheck had the same code.

Solution

Both checks now use one helper, probeConnection, and accept a timeout in the fluent style of warnWhenExceeds:

new RedisCheck(redis.connection()).timeout(2_000)
new RedisMemoryUsageCheck(redis.connection()).timeout('2s')

The timeout covers the wait for a connecting connection, the ping and the action (the memory info for RedisMemoryUsageCheck). By connection status:

  • ready or connect: ping, then the action. Fails with "Redis did not respond within Nms" when the timeout elapses.
  • connecting: wait for the connection to become ready, then ping. Fails with lastError when the connection closes instead.
  • Any other status (reconnecting, wait, close, end): fails at once with "Unable to connect to the redis server" and lastError.

#pingAttempts is removed, and the checks never connect or reconnect the connection. RedisMemoryUsageCheck still pings before it computes the memory usage, also with a custom compute.

Behavior changes (for the release notes)

  • The default timeout is 1 s. Before, the wait while connecting could take up to 3 s, and the ping had no limit.
  • The check no longer reconnects a closed connection. It fails and the connection stays closed.
  • A wait connection (lazyConnect, never connected) fails at once, without an error in the result. No test in this PR covers this case.

Accepted tradeoff

On a silent ready connection, each timed-out run leaves one PING in the ioredis queue until the socket settles. The timeout bounds the result, not the work already sent. The application's own commands pile up on that connection the same way.

Tests

New tests use a fake server that answers the handshake, then stops answering:

  • A ready connection on a silent server fails within the 200 ms timeout, on two runs of the same check instance. On v11.0.0 it hangs until the test timeout.
  • One check instance waits for a connecting connection on 4 runs. On v11.0.0 the 4th run fails.
  • A closed connection is not reconnected. On v11.0.0 the check reconnects it.
  • A reconnecting connection fails at once with ECONNREFUSED, on two runs.
  • RedisMemoryUsageCheck with .compute(async () => 1024) on a silent server fails with the timeout message.

The full suite passes (85 tests), and lint, typecheck and prettier are clean.

RedisCheck could not back a readiness probe. Its ping had no time
bound, so a server that accepted the connection but did not answer
held the check until the socket failed. While connecting, it waited up
to 3 x 1s, which nobody could configure. The #pingAttempts counter was
never reset, so a long-lived check stopped waiting after three waits in
total. On a closed connection, the check reconnected it as a side
effect. RedisMemoryUsageCheck shared the same code.

Both checks now go through probeConnection and accept a timeout
(default 1s), set with `.timeout(2_000)` or `.timeout('2s')`. The
timeout covers the wait for a connecting connection, the ping and the
action. A connection that is neither ready nor connecting fails at once
with its last error, and the checks never connect or reconnect it.
RedisMemoryUsageCheck still pings before it computes the memory usage,
custom compute included.

On a silent ready connection, each timed out run leaves one PING in the
ioredis queue until the socket settles: the timeout bounds the result,
not the work already sent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant