[redis-ha] Self-heal a masterless set instead of waiting for sentinel

[ShmuelOps]

What this PR does / why we need it:

Fixes the masterless deadlock in #412.

When the master pod is lost and its replacement rejoins as a replica behind the same announce address, sentinel keeps advertising that address as master (it answers pings), every member's local view agrees with sentinel, and the set ends up with no master at all. Replicas loop on -NOMASTERLINK Can't SYNC while not connected with my master; automatic failovers abort with -failover-abort-no-good-slave / -failover-abort-not-elected, and an explicit SENTINEL FAILOVER returns -NOGOODSLAVE, because sentinel considers every remaining candidate stale. We hit this in production and it persisted for 7.5h until manual intervention.

The split-brain-fix container cannot help as written, for two reasons:

1. It only reinitializes a pod whose local view disagrees with sentinel — here everyone agrees, on a master that is really a replica.
2. It is blind in this state anyway: it reaches sentinel through the headless service, which drops all endpoints once every pod fails its readiness probe, so identify_master cannot resolve the name. MASTER comes back empty and the existing quorum-reset is equally a no-op.

The change. The announce-N services set publishNotReadyAddresses, so they keep resolving throughout the outage. Poll those for the one fact that matters — does any member hold role:master. After MASTERLESS_CONFIRMATIONS consecutive observations of none (default 5, MASTERLESS_CONFIRM_INTERVAL seconds apart, default 5), promote the reachable member with the highest replication offset via REPLICAOF NO ONE, repoint the others, and reset the sentinels so they rediscover the topology.

The reinit crashloop branch. The branch where sentinel names this pod as master while it runs as a replica used to call reinit, which re-derives slaveof and shuts the pod down — it restarts straight back into a replica while sentinel keeps naming it, an endless shutdown loop (#383). That branch now routes into the same masterless healer: the misnamed pod is the fastest observer of this state, so it heals on its first detection cycle instead of waiting for its peers.

Safety properties:

• Any member reporting role:master aborts the sequence, so concurrent containers never double-promote, and ordinary sentinel failovers are never interfered with.
• Promotion is offset-aware: the healer promotes the reachable member with the highest replication offset, never a specific pod merely because sentinel names it. A freshly replaced (empty) pod behind a stale sentinel record is therefore never promoted, which would otherwise resync every replica from an empty dataset.
• The scan retries up to PROMOTE_SCAN_ATTEMPTS times, so a member that happens to be mid-restart doesn't postpone recovery.
• All knobs are env-only, matching the existing MAX_QUORUM_FAILURES pattern — no new chart values.

Relation to #410

#410 addresses the same family of failure (sentinel naming an address whose real role is slave) by having the named pod promote itself with replicaof no one. This PR deliberately does not do that, for three reasons:

1. Unconditional self-promotion risks data loss. MASTER == ANNOUNCE_IP + role:slave is exactly what a freshly replaced pod looks like behind a stale sentinel record — sentinel never re-elected it, it just kept the address. Promoting that pod without comparing replication offsets can promote an empty node, and the other replicas (which already point at it) then full-resync from the empty dataset. Here the healer always promotes the highest-offset reachable member — which is the named pod whenever it genuinely is the best candidate.
2. fix(redis-ha): recover no-master cycle when sentinel names a replica as master #410's path is blind in the production outage. It still resolves sentinel through the headless service, which has no endpoints once every pod is unready (the [chart/redis-ha][BUG] Masterless deadlock: sentinel keeps naming a master whose actual role is replica; split-brain container never heals it #412 state), so its new branch never executes exactly when it is needed. This PR polls the announce-N services, which publish not-ready addresses.
3. No escalation or confirmation. fix(redis-ha): recover no-master cycle when sentinel names a replica as master #410 promotes after a single re-check; this PR tries SENTINEL FAILOVER first and promotes directly only after N consecutive masterless confirmations, aborting the moment any master appears.

#410's valid core idea — stop the reinit shutdown loop in that branch (#383) — is folded in here, with the promotion decision kept offset-aware.

Which issue this PR fixes

• fixes [chart/redis-ha][BUG] Masterless deadlock: sentinel keeps naming a master whose actual role is replica; split-brain container never heals it #412

Validation

Verified on an isolated 3-replica release. Fault: point every member at an unreachable address, then SENTINEL RESET — this reproduces the signature above exactly.

chart	outcome
4.39.0	no recovery after 5m41s
this patch	recovered in 64s

The named pod detects the state on its next detection cycle and heals directly (previously 1m57s via the peers' stale-master counters):

Redis role is slave but sentinel names this pod as master; checking whether the set is masterless
No member held the master role across 5 consecutive checks.
ERROR: no member holds the master role. Promoting redis-announce-0 (replication offset 39704).
Promotion complete: redis-announce-0 is now master.

Peers, confirming only one member acts:

A member holds the master role; standing down after 2/5 confirmations.

No false positives: deleting the master pod produces an ordinary sentinel failover — new master within seconds, the deleted pod rejoins as a replica, and no healer on any pod promotes or enters the confirmation loop. It acts only in the state sentinel cannot leave on its own.

helm lint clean; the rendered script passes sh -n with defaults and with auth + sentinel.auth enabled.

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [stable/mychartname])