feat(uptime): Detect whole-store config loss with sentinel keys - #125153
Open
Conversation
Name one sentinel key per partition and store as {<prefix>uptime:configs:N}:sentinel. The hash tag makes Redis Cluster hash only the partition key's text, so the sentinel lands on the same slot, and shard, as the hash it stands for: a shard that loses its data loses its sentinel with it.
It is a separate key rather than a field in the config hash because the checker decodes every field of uptime:configs:N as a config at boot and would log invalid_config_message for anything else. The checker reads its hashes by exact name, so it never sees the sentinel.
The test computes slots offline with rediscluster's NodeManager and checks every partition for both an empty and a test key prefix.
Refs INFRENG-621
github-actions
Bot
added
the
Scope: Backend
Add check_config_sentinels, scheduled at one minute, which pipelines EXISTS on the 128 sentinel keys of each config store and reports the number absent as the gauge uptime.config_drift.sentinel_missing by cluster. A wiped store loses every sentinel, so the gauge jumps to 128 within a minute with no Postgres read, well before the checker's next restart drops the configs from memory. It is a gauge rather than a counter because the same 128 keys are re-checked every minute; the value is a level and must not sum across runs like the sweep's checked/missing counts do. The task shares the sweep's uptime.config-drift.enabled kill switch, so with it off nothing is emitted. Each store is checked on its own: a Redis error on one store is the event being looked for, and must not stop the others from reporting that minute. Nothing writes a sentinel yet: until repair lands the gauge reads as missing everywhere, which is the ticket's intended pre-enable state. Sentinels have no TTL by design; 128 fixed keys per cluster are recorded on the ticket as accepted durable data. Refs INFRENG-621
Starrao123
force-pushed
the
vinayakrao/infreng-621-5-detect-whole-store-config-loss-with-sentinel-keys
branch
3 times, most recently
from
September 21, 2026 23:38
2c9115a to
fe40268
Compare
When a sentinel is absent and uptime.config-drift.repair is on, check_config_sentinels logs the missing partitions and queues repair_config_store for that store only. The task compares the whole store at once: HKEYS on each of the 128 partition hashes against one replica read of the store's ACTIVE rows, the orphan direction's read in reverse. It hands the missing ids to repair_missing_configs under its per-run cap and writes the absent sentinels back once a run has handed off everything it found, so a wipe of N configs keeps the sentinel absent for ceil(N/cap) runs. Looping the sweep's per-prefix find_missing_configs here would cost one HEXISTS per config and 256 queries per run; on a 100k-config store that measured 100,513 Redis commands and 891 ms against 131 commands, one query and 90 ms for the same result. The per-prefix shape exists to touch 1/256 of the table per task, which is the opposite of what this task needs. The ticket asked for the write-back only when the comparison reports zero missing. That has no bound: one ACTIVE row the update task can never publish would keep the comparison non-empty forever, so every minute would re-run the store scan and the gauge could no longer tell that row from a wipe. Gating on the cap leaves such a row to the hourly sweep, which already reports it. The comparison runs in its own task so the minute check stays a pure detector and one slow store cannot delay the others. expires=60 drops a request that was not picked up before the next tick re-queues it, and that re-queue is also the retry, so the task declares none. The warning is logged only when repair is on: before the first repair pass every sentinel is absent, so the partition list would say nothing. Refs INFRENG-621
Starrao123
force-pushed
the
vinayakrao/infreng-621-5-detect-whole-store-config-loss-with-sentinel-keys
branch
from
September 21, 2026 23:40
fe40268 to
08d5e4b
Compare
Starrao123
marked this pull request as ready for review
September 21, 2026 23:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves INFRENG-621. Builds on #124809.
The hourly drift sweep from #124640 and #124809 reads every subscription row over a 24-hour cycle, so a wiped config Redis is only fully repaired a day later. This detects whole-store loss within a minute and repairs it in one pass, behind the same
uptime.config-drift.repairoption.The change is one sentinel key per config partition, and the three commits follow it from key to repair:
{uptime:configs:N}:sentinelis a string key hash-tagged onto the same Redis Cluster slot as its partition hash, so a shard that loses its data loses the sentinel too. It is a separate key, not a field, because the checker decodes every field of the config hash as a config at boot.check_config_sentinelsruns every minute under the sweep'suptime.config-drift.enabledswitch, pipelinesEXISTSon the 128 sentinels of each store, and reports the number absent as the gaugeuptime.config_drift.sentinel_missingby cluster.repair_config_storefor that store only. It compares the whole store at once (HKEYSon each partition against one replica read of the store's ACTIVE rows), hands the missing ids torepair_missing_configsunder its cap, and writes the sentinels back once a run has handed off everything it found.The sweep's metrics and tasks are unchanged. Sentinels have no TTL by design; 128 fixed keys per cluster are recorded on the ticket as accepted durable data.
Deploying changes nothing until
uptime.config-drift.enabledis on, and then only the gauge is emitted. Nothing writes a sentinel until repair is on, so the gauge reads 128 per cluster until the first repair pass. This PR doesn't change how the hourly sweep reads or slices, and doesn't guard against two repair runs overlapping on one store; a run that outlasts a minute queues the same idempotent republishes twice.Tests, one per acceptance criterion on the ticket:
uptime:configs:Nanduptime:updates:Nuntouched.Existing tests are unchanged.