Worker count and range changes in values never reach a running RayCluster #41

Problem

A helm upgrade that changes only worker.replicas, worker.minReplicas, or worker.maxReplicas updates the RayService and stops there. The live RayCluster keeps its old numbers indefinitely, with no event, no status condition, and no log line beyond RayService Conditions changed. The change only lands as a side effect of some other rayClusterConfig field changing in the same upgrade, and then it lands via a full cluster roll.

This is deliberate on KubeRay's side. The RayService controller strips replicas, minReplicas, maxReplicas and workersToDelete from the spec hash it compares against the live cluster so that the Ray autoscaler can own those fields (kuberay #2331). A fix that propagated min/max in place was declined (kuberay #2333), the maintainer's guidance being to edit the RayCluster CR directly. The behaviour is unchanged in KubeRay 1.7.0.

Reproduced on jamesolds-dev (EKS 1.34, KubeRay 1.3.0), autoscaling off: helm upgrade from 1/1/1 to 2/2/2 left the RayCluster at 1 1 1 for over four minutes with pods unchanged. The same numbers applied by kubectl patch raycluster produced a second worker pod within a second on the same cluster, same head pod. With autoscaling on, a patched maxReplicas held for two minutes with no revert and no new cluster.

It affects every consumer that manages worker counts through values. ATEP runs replicas=min=max=4, and that bump only took effect because the same change also touched tolerations and probes. It is also the next thing #31 hits: once #35 ships, widening the ATEP range from 4..4 in values will not change the running cluster.

Proposal

Make the chart do what upstream tells users to do by hand: a post-upgrade,post-rollback Helm hook Job that patches the live RayCluster's worker counts in place to match values.

• The Job reads the release's RayService and patches every RayCluster named in status.activeServiceStatus.rayClusterName and status.pendingServiceStatus.rayClusterName. Both matter: with serveApplications: [] the RayService never reaches an active cluster and lives in pendingServiceStatus permanently.
• The target numbers are baked into the Job's args at render time rather than read from the live RayService spec, so it also works under an Argo CD Application that ignores /spec/rayClusterConfig.
• With autoscaling off it patches replicas, minReplicas, maxReplicas. With autoscaling on it patches only minReplicas and maxReplicas, leaving replicas to the autoscaler, which avoids the clobber the KubeRay maintainer objected to.
• Matches the worker group by groupName, not index. Patches only fields that differ. Exits 0 when no cluster exists yet (first Argo CD sync), non-zero on API errors so a broken hook fails the upgrade visibly.
• The patch does not trigger a roll: the RayService compares its spec hash against the hash annotation stamped on the RayCluster at creation, with counts muted, and a live edit changes neither.
• Runs in the Ray image already configured in image.*, using the ServiceAccount token against the API server, so there is no new image to mirror for air-gapped consumers. Overridable.
• Adds a ServiceAccount, Role (rayservices get; rayclusters get, patch), RoleBinding, ConfigMap for the script, and the Job, behind a scalingHook.enabled value.

Under Argo CD the hook maps to PostSync and runs on every sync, so it also reconciles a hand-patched RayCluster back to values.

Alternative considered

Recording the counts in a pod-template annotation so they re-enter the spec hash and a change rolls the cluster. Tried in #27 and dropped: it turns a natively in-place operation into a blue/green roll (new head, new workers, every Ray client session dropped, every model reloaded, double GPU footprint during the switch), rolls every existing deployment once on upgrade, and on a GPU-constrained cluster the scale-up that motivated it is the case most likely to leave the pending cluster stuck.

Tests

Until this lands, scaling.md should say plainly that count and range changes do not reach a running cluster and give the kubectl patch raycluster command.

_{🤖 claude-fable-5-1 (high) · reviewed by @oldsj}