[Bug]: nvidia-vgpu-manager-daemonset excluded from all three of the driver DaemonSet's disruption-avoidance mechanisms, despite doing the same host-driver-install work

[yogeshbendre]

---

Describe the bug

How this was found: as a follow-up to #2709/#2717 (other vgpu-* components missing protections that every other DaemonSet gets), I checked whether nvidia-vgpu-manager-daemonset — which runs the exact same host-driver-install workload as nvidia-driver-daemonset — gets the same protection against being disruptively restarted while GPU/vGPU workloads are actively using the node. It doesn't, on three independent, unrelated mechanisms at once.

1. No updateStrategy in the manifest, so it silently defaults to RollingUpdate. assets/state-vgpu-manager/0500_daemonset.yaml has no updateStrategy key anywhere — confirmed directly against both v26.3.3 and current master. Contrast with assets/state-driver/0500_daemonset.yaml#L15-L17, which hardcodes:

updateStrategy:
  type: OnDelete

— documented in deployments/gpu-operator/values.yaml#L45-L47: "note that driver Daemonset is always set with OnDelete to avoid unintended disruptions."

2. The code-level guard that protects the driver from a user overriding this only checks one of the two constants meant to be a pair. controllers/object_controls.go#L3881-L3883 (applyUpdateStrategyConfig) refuses to apply a user-configured daemonsets.rollingUpdate.maxUnavailable to any DaemonSet whose name has prefix commonDriverDaemonsetName:

if strings.HasPrefix(obj.Name, commonDriverDaemonsetName) {
    // disallow setting RollingUpdate strategy with the driver container
    return nil
}

controllers/state_manager.go#L79-L80 defines commonDriverDaemonsetName ("nvidia-driver-daemonset") and commonVGPUManagerDaemonsetName ("nvidia-vgpu-manager-daemonset") together as a matched pair, used together everywhere else (cleanupAllDriverDaemonSets, per-kernel/per-RHCOS-version DaemonSet creation) — but this one guard function only tests the first constant.

3. The operator's own dedicated safe-upgrade controller doesn't watch vgpu-manager at all. Separately from the strategy field, the operator ships controllers/upgrade_controller.go, which performs a coordinated upgrade for driver DaemonSets — cordon the node, drain GPU-using workload pods off it first, then delete the old pod to trigger a controlled replacement. Its watch predicate is hardcoded to three label selectors, none of which match vgpu-manager:

// upgrade_controller.go:62-70
DriverLabelKey   = "app"
DriverLabelValue = "nvidia-driver-daemonset"
AppComponentLabelKey   = "app.kubernetes.io/component"
AppComponentLabelValue = "nvidia-driver"

(the third is the OpenShift DriverToolkit identification label). assets/state-vgpu-manager/0500_daemonset.yaml#L4-L5 labels the DaemonSet app: nvidia-vgpu-manager-daemonset — matching none of the three predicates at upgrade_controller.go#L304-L330. So this isn't just a missing strategy override guard — the entire coordinated cordon/drain-before-replace subsystem is blind to vgpu-manager's existence.

Supporting detail (weaker signal, noted for completeness): the two DaemonSets share the exact same k8s-driver-manager init-container binary (command: ["driver-manager"], args: ["uninstall_driver"]), which has its own built-in ENABLE_GPU_POD_EVICTION flag for evicting GPU-using pods before proceeding. The driver DaemonSet sets it "true" (assets/state-driver/0500_daemonset.yaml#L60-L61); vgpu-manager's identical init container sets it "false" (assets/state-vgpu-manager/0500_daemonset.yaml#L46-L47). This could plausibly be intentional for the VM-passthrough workload model (evicting a virt-launcher pod is a materially different operation than evicting an ordinary container pod), so it's included as color rather than a fourth independent claim — but it's consistent with the other two findings rather than contradicting them.

Both DaemonSets run the identical underlying command — command: ["nvidia-driver"], args: ["init"] — i.e. vgpu-manager is installing/managing a host-level driver exactly like the bare-metal driver DaemonSet, and carries the same disruption profile.

Why this matters: RollingUpdate lets the DaemonSet controller unilaterally kill and replace a pod the moment its template hash changes (new image, any field), bounded only by maxUnavailable (default 1) — with zero coordination. Concretely: an admin performs a routine gpu-operator upgrade unrelated to vGPU at all (say, a patch bump for an unrelated CVE), and the new chart version happens to bump the default vgpu-manager image tag. Because vgpu-manager defaults to RollingUpdate and is invisible to the upgrade-controller, Kubernetes begins killing and replacing its pods immediately, node by node — with no cordon, no drain, no coordination with whatever VMs are actively consuming a vGPU-sliced profile from that node's host driver at that moment. The identical version-bump scenario against the driver DaemonSet would instead go through the cordon/drain sequence first. Since vgpu-manager exists specifically for the VM-with-vGPU deployment model, the blast radius is every VM actively using vGPU on that node when the swap happens — triggered by what looks, from the admin's side, like an unrelated routine patch upgrade.

---

To Reproduce

This is a manifest/code-inspection finding — no cluster needed to verify the three claims above, and no functional GPU/vGPU hardware is involved in any case:

# 1. Confirm vgpu-manager's manifest has no updateStrategy override (defaults to RollingUpdate)
curl -sSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/assets/state-vgpu-manager/0500_daemonset.yaml | grep updateStrategy
# -> no output

# 2. Confirm the driver DaemonSet hardcodes OnDelete, for contrast
curl -sSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/assets/state-driver/0500_daemonset.yaml | grep -A1 updateStrategy
# -> type: OnDelete

# 3. Confirm the RollingUpdate override guard only tests the driver's name prefix
curl -sSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/controllers/object_controls.go | grep -A2 "disallow setting RollingUpdate"
# -> if strings.HasPrefix(obj.Name, commonDriverDaemonsetName) { ... }
#    (commonVGPUManagerDaemonsetName is never referenced in this function)

# 4. Confirm the upgrade-controller's watch predicate excludes vgpu-manager's own label
curl -sSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/controllers/upgrade_controller.go | grep -n 'DriverLabelValue ='
curl -sSL https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/assets/state-vgpu-manager/0500_daemonset.yaml | grep 'app:'
# -> DriverLabelValue = "nvidia-driver-daemonset"  vs.  app: nvidia-vgpu-manager-daemonset (no match)

On a live cluster, the effect is directly visible on any real install with zero customization:

kubectl get daemonset nvidia-vgpu-manager-daemonset -n gpu-operator -o jsonpath='{.spec.updateStrategy}'
# -> {} / null, meaning RollingUpdate with the k8s default maxUnavailable
kubectl get daemonset nvidia-driver-daemonset -n gpu-operator -o jsonpath='{.spec.updateStrategy.type}'
# -> OnDelete

Happy to build a full live upgrade-simulation repro (bump the vgpu-manager image via ClusterPolicy on a running operator install and observe the uncoordinated pod replacement directly) if that would help triage — held off since the three static findings above are unambiguous on their own.

---

Expected behavior

nvidia-vgpu-manager-daemonset should get the same disruption-avoidance protection as nvidia-driver-daemonset — either by hardcoding updateStrategy: OnDelete in its own manifest (matching the driver), extending the object_controls.go guard to also check commonVGPUManagerDaemonsetName, and/or extending upgrade_controller.go's watch predicate to include vgpu-manager's label — so a routine operator/chart upgrade can't silently trigger an uncoordinated host-driver restart on nodes actively serving vGPU workloads.

---

Environment (please provide the following information):

• GPU Operator Version: v26.3.3; reverified unfixed against current master at the time of filing on all three points — root cause is static (a missing manifest field, a single-constant guard, and a hardcoded label predicate), not version-specific behavior.
• No cluster/hardware needed to reproduce — see "To Reproduce."

---

Information to attach (optional if deemed irrelevant)

Not applicable — this is a manifest/code-completeness finding. Happy to provide a live upgrade-simulation repro if useful for triage (see note in "To Reproduce").