Repository navigation
fix: RedisCheck can now back a readiness probe - #85
Open
RomainLanz wants to merge 1 commit into
Open
RomainLanz wants to merge 1 commit into
RomainLanz wants to merge 1 commit into
Conversation
RedisCheck could not back a readiness probe. Its ping had no time
bound, so a server that accepted the connection but did not answer
held the check until the socket failed. While connecting, it waited up
to 3 x 1s, which nobody could configure. The #pingAttempts counter was
never reset, so a long-lived check stopped waiting after three waits in
total. On a closed connection, the check reconnected it as a side
effect. RedisMemoryUsageCheck shared the same code.
Both checks now go through probeConnection and accept a timeout
(default 1s), set with `.timeout(2_000)` or `.timeout('2s')`. The
timeout covers the wait for a connecting connection, the ping and the
action. A connection that is neither ready nor connecting fails at once
with its last error, and the checks never connect or reconnect it.
RedisMemoryUsageCheck still pings before it computes the memory usage,
custom compute included.
On a silent ready connection, each timed out run leaves one PING in the
ioredis queue until the socket settles: the timeout bounds the result,
not the work already sent.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RedisCheckcould not be used as a readiness probe. We found this when Tenkai added Valkey to its/readyzafter a production outage: pods were marked ready while Valkey was unreachable. In v11.0.0:ping()on a ready connection had no time bound. When the server accepts the connection but does not answer (dropped packets, half-open TCP), the check waited until the socket failed.timeoutSecondsdefaults to 1 s).#pingAttemptswas never reset.HealthCheckskeeps one check instance for the life of the process, so after 3 waits in total the check never waited for a connecting connection again.ioConnection.connect(): the health check reconnected the connection as a side effect.RedisMemoryUsageCheckhad the same code.Solution
Both checks now use one helper,
probeConnection, and accept a timeout in the fluent style ofwarnWhenExceeds:The timeout covers the wait for a connecting connection, the ping and the action (the memory info for
RedisMemoryUsageCheck). By connection status:readyorconnect: ping, then the action. Fails with "Redis did not respond within Nms" when the timeout elapses.connecting: wait for the connection to become ready, then ping. Fails withlastErrorwhen the connection closes instead.reconnecting,wait,close,end): fails at once with "Unable to connect to the redis server" andlastError.#pingAttemptsis removed, and the checks never connect or reconnect the connection.RedisMemoryUsageCheckstill pings before it computes the memory usage, also with a customcompute.Behavior changes (for the release notes)
waitconnection (lazyConnect, never connected) fails at once, without an error in the result. No test in this PR covers this case.Accepted tradeoff
On a silent ready connection, each timed-out run leaves one PING in the ioredis queue until the socket settles. The timeout bounds the result, not the work already sent. The application's own commands pile up on that connection the same way.
Tests
New tests use a fake server that answers the handshake, then stops answering:
readyconnection on a silent server fails within the 200 ms timeout, on two runs of the same check instance. On v11.0.0 it hangs until the test timeout.ECONNREFUSED, on two runs.RedisMemoryUsageCheckwith.compute(async () => 1024)on a silent server fails with the timeout message.The full suite passes (85 tests), and lint, typecheck and prettier are clean.