From 8c308cc5f638bebba062c8b14f19ad7be62bb78a Mon Sep 17 00:00:00 2001 From: Oren Fromberg <145494346+oren-openteams@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:52:34 -0400 Subject: [PATCH 1/5] apply autoscaling changes to chart/values.yaml --- chart/values.yaml | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/chart/values.yaml b/chart/values.yaml index b25c4d7..4f22c95 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -222,6 +222,52 @@ worker: timeoutSeconds: 2 failureThreshold: 120 +# ============================================================================= +# Ray In-Tree Autoscaling +# ============================================================================= +# Enable Ray's in-tree autoscaler to dynamically scale worker pods based on +# task demand. When `autoscaling.enabled` is true, the chart: +# - Sets `enableInTreeAutoscaling: true` on the RayCluster spec, causing +# KubeRay to attach an autoscaler sidecar container to the head pod. +# - Uses `worker.minReplicas` and `worker.maxReplicas` as the scaling range +# (both fields are already exposed above; without autoscaling enabled +# they are ignored by KubeRay). +# +# Without autoscaling.enabled the chart is byte-identical to the pre-existing +# behaviour: worker.replicas is the deployed count, min/max are inert. +# +# See the KubeRay autoscaling guide for background: +# https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html +# +# Ray's docs explicitly note that autoscaling "adds node launch overheads +# and can be tricky to configure" — starting with a fixed replica count +# (autoscaling.enabled: false) is recommended for newcomers. +autoscaling: + enabled: false + + # Time in seconds an idle worker pod is kept alive before being scaled + # down. Lower values reclaim resources faster but can create pod churn if + # tasks arrive in bursts. + idleTimeoutSeconds: 60 + + # upscalingMode: + # - `Default` — standard responsiveness + # - `Conservative` — pending requests scale up in batches on a delay + # - `Aggressive` — immediately upscale to satisfy demand + upscalingMode: Default + + # Optional overrides for the autoscaler sidecar container's resources. + # KubeRay's own defaults apply when unset ({}); typical production + # override: + # resources: + # limits: + # cpu: "500m" + # memory: "512Mi" + # requests: + # cpu: "500m" + # memory: "512Mi" + resources: {} + # ============================================================================= # Overrides # ============================================================================= From 781c7bfabb7f6922ed7dc6387eb64d053ea0338b Mon Sep 17 00:00:00 2001 From: Oren Fromberg <145494346+oren-openteams@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:52:35 -0400 Subject: [PATCH 2/5] apply autoscaling changes to chart/templates/rayservice.yaml --- chart/templates/rayservice.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml index b2475ee..8cdb9be 100644 --- a/chart/templates/rayservice.yaml +++ b/chart/templates/rayservice.yaml @@ -19,6 +19,16 @@ spec: rayClusterConfig: rayVersion: {{ .Values.image.tag | quote }} + {{- if .Values.autoscaling.enabled }} + enableInTreeAutoscaling: true + autoscalerOptions: + idleTimeoutSeconds: {{ .Values.autoscaling.idleTimeoutSeconds }} + upscalingMode: {{ .Values.autoscaling.upscalingMode }} + {{- with .Values.autoscaling.resources }} + resources: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} headGroupSpec: rayStartParams: dashboard-host: "0.0.0.0" From 1500bbf1e4923f5728748007291872dddf44b359 Mon Sep 17 00:00:00 2001 From: Oren Fromberg <145494346+oren-openteams@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:52:36 -0400 Subject: [PATCH 3/5] apply autoscaling changes to chart/Chart.yaml --- chart/Chart.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/chart/Chart.yaml b/chart/Chart.yaml index ba07c8b..8f0c259 100644 --- a/chart/Chart.yaml +++ b/chart/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: nebari-rayserve-pack description: A Nebari Software Pack for Ray Serve type: application -version: 0.4.1 +version: 0.5.0 appVersion: "2.43.0" dependencies: - name: kuberay-operator From 81d7e06b4c271f86f57e177b518f3cb34f3023e4 Mon Sep 17 00:00:00 2001 From: Oren Fromberg <145494346+oren-openteams@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:52:37 -0400 Subject: [PATCH 4/5] apply autoscaling changes to README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 785a93f..3fe6b61 100644 --- a/README.md +++ b/README.md @@ -219,6 +219,10 @@ Key values in `chart/values.yaml`: | `worker.resources.requests.cpu` | `1` | Worker CPU request | | `worker.resources.requests.memory` | `2Gi` | Worker memory request | | `worker.runtimeClassName` | - | Runtime class for worker pods (e.g., `nvidia` for GPU) | +| `autoscaling.enabled` | `false` | Enable Ray in-tree autoscaling of worker pods | +| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down | +| `autoscaling.upscalingMode` | `Default` | `Default`, `Conservative`, or `Aggressive` | +| `autoscaling.resources` | `{}` | Optional resources for the autoscaler sidecar container | ### Serve Applications From 6cda8c70024d97aa9cd816e08e0b71c15f1273c5 Mon Sep 17 00:00:00 2001 From: James Olds <12104969+oldsj@users.noreply.github.com> Date: Thu, 10 Sep 2026 10:21:27 -0400 Subject: [PATCH 5/5] docs(scaling): rewrite autoscaling docs; state that count and range changes do not reach a running cluster MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The RayService controller excludes replicas/minReplicas/maxReplicas from its drift hash, so a helm upgrade changing only them never reaches the running RayCluster — with autoscaling on (the range) or off (the static worker count). Upstream treats this as intended (kuberay #2331, in-place fix declined in #2333). Say so in scaling.md, configuration.md and values.yaml, document `kubectl patch raycluster` as the way to apply the change live, and point at nebari-dev/rayserve-pack#41 for the chart-side fix. Rendered manifests with autoscaling off stay identical to main apart from the chart version label. Reframe the autoscaling docs around Serve autoscaling_config (and users' own tasks) as the demand layer and the cluster range as a capacity budget set once. Fix the upscalingMode comment, quote the rendered value, warn in NOTES when the range is N..N with autoscaling on, and make the README and configuration.md min/max rows agree. --- README.md | 8 +- chart/templates/NOTES.txt | 9 ++ chart/templates/rayservice.yaml | 2 +- chart/values.yaml | 54 +++++----- docs/src/content/docs/configuration.md | 29 +++++- docs/src/content/docs/scaling.md | 139 +++++++++++++++++++++++-- 6 files changed, 198 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index 3fe6b61..363b561 100644 --- a/README.md +++ b/README.md @@ -214,15 +214,15 @@ Key values in `chart/values.yaml`: | `head.resources.requests.memory` | `2Gi` | Head node memory request | | `head.runtimeClassName` | - | Runtime class for head pod (e.g., `nvidia` for GPU) | | `worker.replicas` | `1` | Number of worker nodes | -| `worker.minReplicas` | `1` | Min workers (for autoscaling) | -| `worker.maxReplicas` | `1` | Max workers (for autoscaling) | +| `worker.minReplicas` | `1` | Lower bound on workers; the autoscaler's floor when `autoscaling.enabled` | +| `worker.maxReplicas` | `1` | Upper bound on workers; the autoscaler's ceiling when `autoscaling.enabled`. Raise it or autoscaling does nothing | | `worker.resources.requests.cpu` | `1` | Worker CPU request | | `worker.resources.requests.memory` | `2Gi` | Worker memory request | | `worker.runtimeClassName` | - | Runtime class for worker pods (e.g., `nvidia` for GPU) | | `autoscaling.enabled` | `false` | Enable Ray in-tree autoscaling of worker pods | | `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down | -| `autoscaling.upscalingMode` | `Default` | `Default`, `Conservative`, or `Aggressive` | -| `autoscaling.resources` | `{}` | Optional resources for the autoscaler sidecar container | +| `autoscaling.upscalingMode` | `Default` | `Default` or `Conservative` (rate-limited); `Aggressive` is an alias for `Default` | +| `autoscaling.resources` | `{}` | Autoscaler sidecar resources; KubeRay's default is `500m` / `512Mi` for both requests and limits | ### Serve Applications diff --git a/chart/templates/NOTES.txt b/chart/templates/NOTES.txt index 4794381..ff26fa3 100644 --- a/chart/templates/NOTES.txt +++ b/chart/templates/NOTES.txt @@ -3,6 +3,15 @@ The RayService controller manages the Ray cluster and Serve proxy. Serve is pre-initialized with host 0.0.0.0 on port 8000. +{{- if .Values.autoscaling.enabled }} +{{- if not (gt (int .Values.worker.maxReplicas) (int .Values.worker.minReplicas)) }} + +WARNING: autoscaling.enabled is true but the worker range is + [{{ .Values.worker.minReplicas }}, {{ .Values.worker.maxReplicas }}], so nothing can scale. Raise worker.maxReplicas. + +{{- end }} +{{- end }} + {{- if .Values.nebariapp.enabled }} NebariApp is ENABLED. The nebari-operator will configure: diff --git a/chart/templates/rayservice.yaml b/chart/templates/rayservice.yaml index 8cdb9be..a08efdc 100644 --- a/chart/templates/rayservice.yaml +++ b/chart/templates/rayservice.yaml @@ -23,7 +23,7 @@ spec: enableInTreeAutoscaling: true autoscalerOptions: idleTimeoutSeconds: {{ .Values.autoscaling.idleTimeoutSeconds }} - upscalingMode: {{ .Values.autoscaling.upscalingMode }} + upscalingMode: {{ .Values.autoscaling.upscalingMode | quote }} {{- with .Values.autoscaling.resources }} resources: {{- toYaml . | nindent 8 }} diff --git a/chart/values.yaml b/chart/values.yaml index 4f22c95..6befea8 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -162,6 +162,9 @@ head: # Ray Workers # ============================================================================= worker: + # A values-only change to these does not reach a running RayCluster: KubeRay + # leaves them to the autoscaler. Patch the RayCluster in place instead (see + # docs, "Adding workers"). Chart-side fix tracked in rayserve-pack#41. replicas: 1 minReplicas: 1 maxReplicas: 1 @@ -225,40 +228,43 @@ worker: # ============================================================================= # Ray In-Tree Autoscaling # ============================================================================= -# Enable Ray's in-tree autoscaler to dynamically scale worker pods based on -# task demand. When `autoscaling.enabled` is true, the chart: -# - Sets `enableInTreeAutoscaling: true` on the RayCluster spec, causing -# KubeRay to attach an autoscaler sidecar container to the head pod. -# - Uses `worker.minReplicas` and `worker.maxReplicas` as the scaling range -# (both fields are already exposed above; without autoscaling enabled -# they are ignored by KubeRay). +# Off by default. When enabled, the chart sets `enableInTreeAutoscaling: true` and +# KubeRay attaches the Ray autoscaler as a sidecar on the head pod. It adds +# worker pods within [worker.minReplicas, worker.maxReplicas] when Ray tasks, +# actors, or Serve replicas have nowhere to run, and removes pods idle for +# idleTimeoutSeconds; worker.replicas is only the starting size. # -# Without autoscaling.enabled the chart is byte-identical to the pre-existing -# behaviour: worker.replicas is the deployed count, min/max are inert. +# Treat the range as a capacity budget set once, not a tuning knob: +# - Both bounds default to 1. Raise worker.maxReplicas or nothing scales. +# - Day-to-day behaviour belongs in each Serve deployment's +# `autoscaling_config` under serveApplications. RayService applies Serve +# config changes to the running cluster in place; any other change here +# rolls a new RayCluster (zero-downtime, but a full head and worker +# restart); a change to the range alone reaches nothing until the next +# roll — patch the RayCluster to apply it live. +# - The sidecar reserves 500m CPU / 512Mi memory on the head pod unless +# `autoscaling.resources` says otherwise. # -# See the KubeRay autoscaling guide for background: +# Full explanation in docs/src/content/docs/scaling.md ("Ray autoscaling"). +# KubeRay background: # https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/configuring-autoscaling.html -# -# Ray's docs explicitly note that autoscaling "adds node launch overheads -# and can be tricky to configure" — starting with a fixed replica count -# (autoscaling.enabled: false) is recommended for newcomers. autoscaling: enabled: false - # Time in seconds an idle worker pod is kept alive before being scaled - # down. Lower values reclaim resources faster but can create pod churn if - # tasks arrive in bursts. + # Seconds an idle worker pod is kept before scale-down. 60 is Ray's + # general-purpose default; inference workloads pay an image pull and a + # model load per new worker, so 300-600 usually churns less. idleTimeoutSeconds: 60 - # upscalingMode: - # - `Default` — standard responsiveness - # - `Conservative` — pending requests scale up in batches on a delay - # - `Aggressive` — immediately upscale to satisfy demand + # upscalingMode — read by the Ray autoscaler, not the KubeRay operator: + # - `Default` — upscaling is not rate-limited + # - `Conservative` — rate-limited: pending worker pods are capped at the + # number of workers already connected to the cluster + # - `Aggressive` — an alias for `Default`; behaviour is identical upscalingMode: Default - # Optional overrides for the autoscaler sidecar container's resources. - # KubeRay's own defaults apply when unset ({}); typical production - # override: + # Autoscaler sidecar resources. Left unset ({}), KubeRay applies requests + # AND limits of 500m CPU / 512Mi memory: # resources: # limits: # cpu: "500m" diff --git a/docs/src/content/docs/configuration.md b/docs/src/content/docs/configuration.md index e1f637a..976a40c 100644 --- a/docs/src/content/docs/configuration.md +++ b/docs/src/content/docs/configuration.md @@ -34,8 +34,8 @@ see [Connecting from Jupyter](/jupyter/#versions-must-match). | Value | Default | Purpose | |---|---|---| | `worker.replicas` | `1` | Worker pods. | -| `worker.minReplicas` | `1` | Lower clamp on `replicas`. Pinned to `1` in `values.yaml` — it does not follow `replicas`. | -| `worker.maxReplicas` | `1` | Upper clamp on `replicas`. Same caveat; leave it below `replicas` and you get `maxReplicas` workers. | +| `worker.minReplicas` | `1` | Lower clamp on `replicas`; the autoscaler's floor under [`autoscaling`](#autoscaling). Pinned to `1` in `values.yaml` — it does not follow `replicas`. | +| `worker.maxReplicas` | `1` | Upper clamp on `replicas`; the autoscaler's ceiling under [`autoscaling`](#autoscaling). Same caveat; leave it below `replicas` and you get `maxReplicas` workers. | | `worker.resources.requests` | `cpu: 1`, `memory: 2Gi` | — | | `worker.resources.limits` | `cpu: 2`, `memory: 4Gi` | — | | `worker.runtimeClassName` | unset | e.g. `nvidia`. | @@ -50,10 +50,29 @@ fall back to KubeRay's default. Full rationale in [Scaling and GPUs](/scaling/#p ::: :::caution[Always set `replicas`, `minReplicas`, and `maxReplicas` together] -The chart does not enable `enableInTreeAutoscaling`, so there is no Ray autoscaler and the -group size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay. -Since `values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See +With `autoscaling.enabled: false` (the default) there is no Ray autoscaler, and the group +size is exactly `replicas` — clamped into `[minReplicas, maxReplicas]` by KubeRay. Since +`values.yaml` pins both bounds to `1`, raising `replicas` alone changes nothing. See [Scaling and GPUs](/scaling/#adding-workers). + +With autoscaling on, `replicas` is only the starting size and the bounds are the range the +autoscaler works within — still `1..1` by default, so raise `maxReplicas`. +::: + +## `autoscaling` + +| Value | Default | Purpose | +|---|---|---| +| `autoscaling.enabled` | `false` | Emits `enableInTreeAutoscaling: true`, so KubeRay attaches a Ray autoscaler sidecar to the head pod. | +| `autoscaling.idleTimeoutSeconds` | `60` | Seconds an idle worker is kept before scale-down. | +| `autoscaling.upscalingMode` | `Default` | `Default` (not rate-limited), `Conservative` (pending pods capped at connected workers), or `Aggressive` (an alias for `Default`). | +| `autoscaling.resources` | `{}` | Overrides the sidecar's resources. Unset, KubeRay hardcodes requests **and** limits of `500m` CPU / `512Mi` memory. | + +:::note[The range is a budget, not a knob] +Raise `worker.maxReplicas` — both bounds default to `1`. Then leave the range alone: a +values-only change to it does not reach a running cluster at all, and changing any other +`autoscaling.*` value rolls a new RayCluster, whereas a Serve deployment's own +`autoscaling_config` applies in place. See [Ray autoscaling](/scaling/#ray-autoscaling). ::: ## `serve` and `serveApplications` diff --git a/docs/src/content/docs/scaling.md b/docs/src/content/docs/scaling.md index 331fe38..917485b 100644 --- a/docs/src/content/docs/scaling.md +++ b/docs/src/content/docs/scaling.md @@ -41,15 +41,139 @@ worker: replicas into the `[minReplicas, maxReplicas]` range. ::: -There is no autoscaler behind these bounds. The chart does not set -`enableInTreeAutoscaling`, so KubeRay runs no Ray autoscaler sidecar and the group size is -exactly `replicas`. Growing the pool means changing `replicas` and running `helm upgrade`; -`minReplicas` and `maxReplicas` are clamps, not a range something moves within. +By default there is no autoscaler behind these bounds: the group size is exactly `replicas`, +and growing the pool means changing `replicas` and running `helm upgrade`. `minReplicas` and +`maxReplicas` only become a range something moves within under +[Ray autoscaling](#ray-autoscaling). + +:::caution[Changing the worker count does not reach a running cluster] +A `helm upgrade` that changes only `replicas`, `minReplicas`, or `maxReplicas` updates the +RayService and stops there. The RayService controller leaves those three fields out of the +spec hash it uses to detect drift, so the running RayCluster keeps its old numbers — no event, +no rollout, no error. The change lands only when some other `rayClusterConfig` field changes in +the same upgrade, and then as a full cluster roll. To add workers to a running cluster, patch +the RayCluster directly and put the same numbers in your values file so the next rollout +carries them: + +```bash +kubectl -n rayserve patch raycluster --type json -p '[ + {"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":6}, + {"op":"replace","path":"/spec/workerGroupSpecs/0/minReplicas","value":6}, + {"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":6}]' +``` + +KubeRay adds the pods in place. See +[why a range change does not reach the cluster](#why-a-range-change-does-not-reach-the-cluster); +a chart-side fix is tracked in [#41](https://github.com/nebari-dev/rayserve-pack/issues/41). +::: Whether the new pods actually land is a separate question. On a cluster with a node autoscaler, asking for more than current nodes can hold triggers node scale-up; without one, the extra pods stay `Pending`. +## Ray autoscaling + +```yaml +autoscaling: + enabled: true + +worker: + minReplicas: 1 + maxReplicas: 6 + resources: + requests: { cpu: "4", memory: "16Gi" } + limits: { cpu: "4", memory: "16Gi" } + +serveApplications: + - name: my-model + import_path: myapp.model:app + deployments: + - name: MyModel + ray_actor_options: { num_cpus: 4 } + autoscaling_config: + min_replicas: 1 + max_replicas: 6 + target_ongoing_requests: 2 +``` + +Three layers, each reacting to the one above it: + +1. **Serve deployment autoscaling** (`autoscaling_config`) adds model replicas as request load + rises. Each replica needs the resources declared in `ray_actor_options`. +2. **The Ray autoscaler**, which `autoscaling.enabled` turns on, adds worker pods when those + replicas have nowhere to run, within `[worker.minReplicas, worker.maxReplicas]`, and removes + pods that have been idle for `autoscaling.idleTimeoutSeconds`. +3. **Your node autoscaler** adds nodes when the pods cannot be scheduled. + +The Ray autoscaler reacts to any unschedulable demand, so on a cluster that notebooks connect +to over Ray client, users' tasks and actors are the first layer as much as Serve is. For a +Serve-only cluster, a fixed `num_replicas` never generates demand and the autoscaler has +nothing to do; and without the second layer, `autoscaling_config` can only scale within the +resources the cluster already has. + +:::caution[Raise `maxReplicas`] +Both bounds default to `1`. `autoscaling.enabled: true` alone leaves the group pinned at one +worker and only adds the autoscaler sidecar to the head pod — `500m` CPU / `512Mi` memory, +requests and limits, unless `autoscaling.resources` overrides it. +::: + +### Tune Serve, set the cluster range once + +The two layers differ in how a change reaches a running service: + +- **Serve config** — `serveApplications`, including `autoscaling_config` — is applied in + place. The RayService controller resubmits it to the live cluster and replicas adjust in + seconds. +- **Cluster config** — everything under `rayClusterConfig`, which includes `autoscaling.*` — + is replaced, not edited. Any change rolls a new RayCluster: a second head and worker set + start, Serve comes up on them, traffic switches, the old cluster is deleted. Zero downtime, + but a full model reload. The worker range is the exception: on its own it changes nothing + until something else rolls the cluster. + +So treat `worker.minReplicas` / `worker.maxReplicas` as a capacity budget — the most workers +you will pay for, the fewest you want warm — set at install and rarely revisited. Put the +behaviour you expect to tune in `autoscaling_config`: `min_replicas` / `max_replicas`, +`target_ongoing_requests`, `upscale_delay_s` / `downscale_delay_s`. + +If you need a new ceiling on a live service, patch the RayCluster directly. The controller +leaves the range fields to the autoscaler and will not revert it; put the same value in your +values file so the next rollout carries it: + +```bash +kubectl -n rayserve patch raycluster --type json \ + -p '[{"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":8}]' +``` + +### Why a range change does not reach the cluster + +The RayService controller excludes `replicas`, `minReplicas`, and `maxReplicas` from the spec +hash it uses to detect drift, because the autoscaler writes `replicas` itself. So a +`helm upgrade` changing only those fields — the autoscaling range, or the static worker +count — never reaches the running cluster. Upstream considers this intended +([kuberay #2331](https://github.com/ray-project/kuberay/issues/2331)); a fix that propagated +the range in place was declined +([kuberay #2333](https://github.com/ray-project/kuberay/pull/2333)), with the guidance being to +edit the RayCluster directly, as above. A chart hook that applies the values to the live +RayCluster automatically is proposed in +[#41](https://github.com/nebari-dev/rayserve-pack/issues/41). + +### Idle timeout and upscaling mode + +`idleTimeoutSeconds: 60` is Ray's general-purpose default. A new worker pays an image pull and +a model load, so for inference a longer hold — 300 to 600 seconds — usually beats reclaiming a +pod that will be wanted again two minutes later. + +`upscalingMode: Default` is right for Serve. `Conservative` caps pending worker pods at the +number already connected, which only helps when node provisioning is slow and a burst of +`Pending` pods causes trouble. `Aggressive` is an alias for `Default`. + +### With Argo CD + +`enableInTreeAutoscaling` renders under `spec.rayClusterConfig` — the path the documented +Application ignores under `RespectIgnoreDifferences=true`. Enabling autoscaling on an +already-synced cluster is silently not applied unless that ignore rule is narrowed. Same issue +as [`orgCABundle`](/ca-bundle/). + ## GPUs Four things have to line up — and the fourth is the one people miss. @@ -206,12 +330,9 @@ head that starts OOM-killing takes the whole cluster with it, so it is worth hea ## What is not here -- **Ray cluster autoscaling** — the chart does not set `enableInTreeAutoscaling`, so the - worker group never grows on its own. - **Per-deployment autoscaling** — Ray Serve's own `autoscaling_config` goes in a - `serveApplications` deployment entry, not in the chart's values. It scales replicas within - the resources the cluster already has, which is all it can do without the cluster - autoscaler above. + `serveApplications` deployment entry, not in chart values. [Ray autoscaling](#ray-autoscaling) + covers how it pairs with the cluster autoscaler. - **Multiple worker groups** — the chart renders one `workerGroupSpecs` entry. Heterogeneous pools (CPU plus GPU) need a chart change or a second release. - **Node autoscaling** — that is your cluster autoscaler's job.