feat(chart): add opt-in enableInTreeAutoscaling support

[oren-openteams]

Summary

Adds an opt-in autoscaling.enabled value that wires Ray's in-tree autoscaler into the RayCluster spec. Default is off — chart output is byte-identical to today's behaviour for existing consumers.

The gap this closes

The chart already exposes worker.minReplicas and worker.maxReplicas, and the README values table labels both "for autoscaling". But the RayCluster template never emits enableInTreeAutoscaling: true, so KubeRay only ever uses those fields as clamps on worker.replicas. The result is that setting minReplicas / maxReplicas has no observable effect today beyond that clamp.

This PR wires the missing top-level flag through a new autoscaling: values block. When autoscaling.enabled: true:

• Chart renders enableInTreeAutoscaling: true on the RayCluster spec
• KubeRay attaches an autoscaler sidecar to the head pod
• The already-existing worker.minReplicas and worker.maxReplicas become the scaling range
• autoscalerOptions.idleTimeoutSeconds and .upscalingMode are configurable via the new block (defaults per Ray docs: 60 s and Default)

Changes

• chart/values.yaml — new top-level autoscaling: block with enabled, idleTimeoutSeconds, upscalingMode, resources. All defaults preserve today's static-scaling behaviour. The upscalingMode comment is corrected: Aggressive is an alias for Default, and Conservative caps pending worker pods at the number already connected.
• chart/templates/rayservice.yaml — conditional enableInTreeAutoscaling: true + autoscalerOptions block, gated on .Values.autoscaling.enabled. upscalingMode is quoted so the CRD enum is always a string.
• chart/templates/NOTES.txt — warns when autoscaling is on but the range is [N, N], since then nothing can scale and the sidecar's 500m / 512Mi is paid for nothing.
• chart/Chart.yaml — minor bump 0.4.1 → 0.5.0 for the new feature.
• README.md — four new rows in the values table, and the minReplicas / maxReplicas rows now say what the bounds do with and without autoscaling.
• docs/src/content/docs/scaling.md — new "Ray autoscaling" section: the three layers (Serve autoscaling_config or users' own tasks → Ray autoscaler → node autoscaler), the range as a capacity budget set once, idle timeout and upscaling mode guidance, the Argo CD ignoreDifferences interaction, and the finding below. Autoscaling removed from "What is not here".
• docs/src/content/docs/configuration.md — autoscaling values table; the "always set all three" caution and the min/max rows updated to match.

Worker count and range changes do not reach a running cluster

Found while testing this. The RayService controller leaves replicas, minReplicas and maxReplicas out of the spec hash it uses to detect drift, so a helm upgrade that changes only those fields updates the RayService and stops there — with autoscaling on (the range) or off (the static worker count). Upstream treats this as intended (kuberay #2331; an in-place fix was declined in kuberay #2333). This PR does not change that behaviour; it documents it and gives kubectl patch raycluster as the way to apply the change live. The chart-side fix is tracked in #41.

Backwards compatibility

Default autoscaling.enabled: false. Rendered manifests for any existing values file are unchanged — the new conditional emits nothing when disabled. Confirmed: helm template with default values against main differs only in the helm.sh/chart version label.

Real-world motivation

We hit this while sizing a Ray cluster for a Nebari deployment. The user needed 3+ CPUs on the Ray cluster (checkmaite reserves CPUs for its registry + per-job controller actors before any worker task can be placed). Static bump of worker.replicas unblocked them, but keeping 4 worker pods running 24/7 is wasteful when the cluster is idle. Autoscaling with minReplicas: 1, maxReplicas: 6 would let workers scale to zero-of-what's-needed when the cluster is quiet and up to 6 during active use.

Test plan

All on an EKS 1.34 cluster with KubeRay operator 1.3.0, scratch namespaces, operator subchart disabled.

• Deploy chart with autoscaling.enabled: false (default) → rendered RayService is byte-identical to pre-PR output. Confirmed via helm template diff against main; only the chart version label differs.
• Deploy chart with autoscaling.enabled: true, worker.minReplicas: 1, worker.maxReplicas: 4 → RayCluster spec has enableInTreeAutoscaling: true + autoscalerOptions; head pod is 2/2 with containers ray-head autoscaler; request_resources(num_cpus=5) scaled 1 → 3 workers in ~15 s; back to 1 after idleTimeoutSeconds.
• Sidecar resources without an override → requests and limits of 500m / 512Mi, as KubeRay hardcodes them.
• Override autoscaling.resources (250m / 256Mi for both) → RayCluster autoscalerOptions.resources and the live autoscaler container both show 250m / 256Mi requests and limits; head pod 2/2 ready in ~25 s; autoscaler loop running.
• Serve-config-only change (serve.proxyLocation) → applied in place, same RayCluster.
• helm upgrade changing only worker.maxReplicas (autoscaling on) or only worker.replicas (autoscaling off) → RayService spec updated, RayCluster unchanged after 4 min. kubectl patch raycluster with the same numbers → applied in place within 1 s, same head pod, not reverted. See the section above and #41.

Not run: Serve's per-deployment autoscaling_config driving the cluster autoscaler. The autoscaler's response to unschedulable demand is covered by the request_resources check above; Serve replicas are actors with resource requests and take the same path.

References

• KubeRay autoscaling guide
• Ray Serve K8s production guide — recommends enabling K8s + Ray autoscaling in production
• kuberay #2331 / kuberay #2333 — why count and range changes are not propagated
• #41 — chart-side fix

_{🤖 claude-fable-5-1 (high) · reviewed by @oldsj}

[oldsj]

The template is correct and the opt-in shape is right — I confirmed against the vendored KubeRay 1.3.0 CRD that every field name and the upscalingMode enum are valid, and a live install on my dev cluster scaled 1 → 3 workers on demand and back down after idleTimeoutSeconds. The autoscaler sidecar attaches, and KubeRay creates its RBAC on its own, so nothing else is needed in the chart. I also reproduced the backwards-compatibility claim: with autoscaling.enabled: false the rendered output against origin/main differs only in the helm.sh/chart version label.

Four things to fix before this merges, all of them about what surrounds the template rather than the template itself.

1. This needs a rebase — main now says the opposite of what the PR does. The branch is based on 224834b, and main has since landed #32 and #33, replacing Docusaurus with an Astro/Starlight site and writing real content into it. That content asserts in three places that this feature does not exist: docs/src/content/docs/scaling.md:44-47 ("The chart does not set enableInTreeAutoscaling, so KubeRay runs no Ray autoscaler sidecar"), scaling.md:209-210 (autoscaling listed under "What is not here"), and configuration.md:52-57, whose whole :::caution block is premised on "there is no Ray autoscaler". The values reference in configuration.md also documents every chart value and would be missing autoscaling.* entirely. Updating README.md alone was sufficient when this branch was written; it no longer is.

2. autoscaling.enabled: true on its own does nothing, and costs 500m/512Mi to do it. worker.minReplicas and worker.maxReplicas both default to 1, so the scaling range is 1..1 and the group is pinned — confirmed in KubeRay's GetWorkerGroupDesiredReplicas clamp and in Ray's autoscaling_config.py, which maps the group to min_workers == max_workers == 1. Meanwhile KubeRay injects the autoscaler sidecar with hardcoded requests and limits of 500m CPU / 512Mi memory (pod.go:523-531, and I confirmed it on a live pod), so the head pod's scheduling footprint goes from the chart's default 1 CPU / 2Gi to 1.5 CPU / 2.5Gi. Flipping the flag alone is therefore a pure regression: more resources, identical behaviour, head pod possibly unschedulable on a tight node — and for a RayService a Pending head means the service is down. Both the sidecar overhead and the "you must also widen worker.maxReplicas" requirement need to be stated where someone will see them.

3. The scaling range can only be set at install time. This is the part I'd most want documented, because it is genuinely surprising. The RayService controller deliberately strips replicas, minReplicas, and maxReplicas out of the spec hash it compares against the live RayCluster, so the autoscaler can own them. The side effect is that a helm upgrade changing only worker.maxReplicas never reaches the cluster. I watched it: I upgraded maxReplicas from 4 to 1, the RayService spec dutifully showed max=1, and the live RayCluster sat at max=4 indefinitely. The range only updates as a side effect of some other rayClusterConfig field changing — and when that happens you get a full zero-downtime cluster replacement, not an in-place edit. Changing autoscaling.idleTimeoutSeconds from 60 to 90 spun up a second RayCluster and reaped the first, restarting head and all workers. So tuning any autoscaler option on a live inference service costs a cluster rebuild. Worth a sentence in scaling.md.

4. The upscalingMode comment in values.yaml is wrong on two of three modes. Per raycluster_types.go:122-126 and the mapping in Ray's autoscaling_config.py: Aggressive is a literal alias for Default (both set upscaling_speed = 1000), not a faster tier; and Conservative is not "batches on a delay" — it sets upscaling_speed = 1, capping pending worker pods at the number currently connected.

One smaller thing: enabling this on an ArgoCD-managed cluster after the initial sync will silently not apply. enableInTreeAutoscaling renders under spec.rayClusterConfig, which is exactly the path the documented Application ignores under RespectIgnoreDifferences=true — the same failure class already written up for nebari-rayserve-pack#17. Worth mirroring that warning for this feature.

Nothing here needs a template change except the optional quoting nit below. Rebase, move the docs, and I think this is good.

---

Verification actually run, for the record: helm template diff of origin/main vs this head with default values (identical but for the chart label); field names and the upscalingMode enum checked against the vendored ray.io_rayservices.yaml CRD; and a live install on an isolated namespace of my jamesolds-dev EKS cluster covering sidecar attachment, scale-up via request_resources, scale-down after the idle timeout, sidecar resource inspection, auto-created RBAC, the maxReplicas propagation test, and the zero-downtime cluster replacement. Not covered: Ray Serve's own per-deployment autoscaling_config interaction.

_{🤖 claude-opus-5 (medium) · review-pr skill · reviewed by @oldsj}