[Bug]: nvidia-vgpu-device-manager DaemonSet has no tolerations for nvidia.com/gpu, unlike every other GPU Operator DaemonSet — won't schedule on tainted GPU nodes on non-Helm installs

[yogeshbendre]

---

Describe the bug

This is the sibling gap to #2709 (missing priorityClassName on the same DaemonSet, fixed via #2712) — same manifest, same root-cause code shape, a different field, and not touched by that fix.

assets/state-vgpu-device-manager/0600_daemonset.yaml has no tolerations: key anywhere in the pod spec — confirmed directly against both v26.3.3 and current master. All 15 other DaemonSet-bearing components hardcode the same block, e.g. assets/state-driver/0500_daemonset.yaml#L29-L32:

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

The root cause is structurally identical to the priorityClassName bug — controllers/object_controls.go#L824-L827 (applyCommonDaemonsetConfig) only overrides tolerations, never defaults them:

// set tolerations if specified
if len(config.Daemonsets.Tolerations) > 0 {
    obj.Spec.Template.Spec.Tolerations = config.Daemonsets.Tolerations
}

This isn't compensated for elsewhere:

• api/nvidia/v1/clusterpolicy_types.go#L247 — the Tolerations field has no +kubebuilder:default marker.
• config/samples/v1_clusterpolicy.yaml#L23 — the sample/default ClusterPolicy CR (used by OLM subscriptions and the OpenShift catalog) sets daemonsets: {}, entirely empty, so Tolerations is nil at reconcile time.
• deployments/gpu-operator/values.yaml#L41-L44 sets a chart-wide default toleration — but that value only reaches the CR on Helm-templated installs, not OLM/OpenShift catalog installs which construct the CR independently.

Confirmed this isn't just a code-reading inference: installed the real nvidia/gpu-operator Helm chart (v26.3.3) and checked the live, reconciled DaemonSet —

$ kubectl get daemonset nvidia-vgpu-device-manager -n gpu-operator -o jsonpath='TOLERATIONS={.spec.template.spec.tolerations}{"\n"}PRIORITYCLASS={.spec.template.spec.priorityClassName}{"\n"}'
TOLERATIONS=[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Exists"}]
PRIORITYCLASS=system-node-critical

So the Helm path is confirmed fine — the gap below is specific to non-Helm-templated installs.

Why this matters: on any non-Helm-templated install (OLM subscription, OpenShift catalog, or a raw ClusterPolicy CR that leaves daemonsets.tolerations unset), nvidia-vgpu-device-manager pods get zero tolerations. If GPU nodes carry the standard nvidia.com/gpu:NoSchedule taint — the exact taint every sibling DaemonSet explicitly tolerates — the vgpu-device-manager pod never schedules on those nodes at all, silently breaking vGPU device/profile configuration cluster-wide, with no error pointing at the missing toleration as the cause.

Confirmed live, deploying the two real manifests (namespace/image/configmap-name placeholders substituted, no other changes) to a kind cluster, targeting the same tainted worker node for both:

$ kubectl get node worker -o jsonpath='{.spec.taints}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"present"}]

$ kubectl get daemonset nvidia-vgpu-device-manager -o jsonpath='{.status.desiredNumberScheduled}'
0
$ kubectl get pods -l app=nvidia-vgpu-device-manager
No resources found in default namespace.

$ kubectl get daemonset nvidia-driver-daemonset -o jsonpath='{.status.desiredNumberScheduled}'
1
$ kubectl get pods -l app=nvidia-driver-daemonset -o wide
NAME                            READY   STATUS                   NODE
nvidia-driver-daemonset-m7qfw   0/4     Init:RunContainerError   worker

nvidia-vgpu-device-manager isn't just left Pending with a FailedScheduling event a cluster operator could notice — the DaemonSet controller's own node-predicate filtering excludes the tainted node before a pod object is ever created (desiredNumberScheduled=0). There is nothing to see: no pod, no event, no signal that a missing toleration (rather than, say, a node-selector mismatch, or the node simply not existing yet) is the reason vGPU configuration never lands on that node. nvidia-driver-daemonset (has the toleration) is scheduled onto the identical node without issue — it fails afterward for unrelated reasons specific to running a privileged host-driver install in a kind/pause-image stand-in, which is expected and irrelevant to the scheduling behavior under test.

---

To Reproduce

No real GPU hardware or vGPU license is required — this is a manifest/scheduling-level check, not a functional GPU test.

# 1. Any Kubernetes cluster with 2+ worker nodes (a plain `kind` cluster is sufficient)
kind create cluster --name vgpu-dm-toleration-repro --config - <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
EOF

# 2. Taint a worker node the way GPU nodes typically are (matches the taint every sibling
#    DaemonSet explicitly tolerates), and label it to match both DaemonSets' nodeSelectors
kubectl taint node <worker> nvidia.com/gpu=present:NoSchedule
kubectl label node <worker> nvidia.com/gpu.deploy.vgpu-device-manager=true nvidia.com/gpu.deploy.driver=true

# 3. Fetch the actual, unmodified vgpu-device-manager manifest at the pinned tag
curl -sSL -o vgpu-dm.yaml \
  https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/assets/state-vgpu-device-manager/0600_daemonset.yaml

# 4. Substitute only the operator-templated placeholders (namespace, image, configmap name) —
#    tolerations is absent in the source file itself, this is the only edit needed
sed -i \
  -e 's#namespace: "FILLED BY THE OPERATOR"#namespace: default#' \
  -e 's#image: "FILLED BY THE OPERATOR"#image: "registry.k8s.io/pause:3.10"#g' \
  -e 's#name: "FILLED BY THE OPERATOR"#name: vgpu-config-dummy#' \
  vgpu-dm.yaml
kubectl create serviceaccount nvidia-vgpu-device-manager

# 5. Apply and check scheduling status
kubectl apply -f vgpu-dm.yaml
kubectl get daemonset nvidia-vgpu-device-manager -o jsonpath='{.status.desiredNumberScheduled}'
# -> 0   (no pod is ever created — the controller's own node-predicate filtering
#          excludes the tainted node before a Pod object exists)
kubectl get pods -l app=nvidia-vgpu-device-manager
# -> No resources found

# 6. For contrast, repeat with assets/state-driver/0500_daemonset.yaml (same substitutions,
#    plus stripping the OCP-only openshift-driver-toolkit-ctr sidecar container, which
#    references a volume the operator injects dynamically and isn't relevant here) —
#    it DOES hardcode the toleration and schedules normally onto the identical tainted node:
kubectl get daemonset nvidia-driver-daemonset -o jsonpath='{.status.desiredNumberScheduled}'
# -> 1
kubectl get pods -l app=nvidia-driver-daemonset -o wide
# -> pod placed on <worker> with an IP assigned (fails afterward for unrelated reasons
#      specific to a privileged host-driver install running against a pause-image stand-in)

---

Expected behavior

nvidia-vgpu-device-manager should tolerate nvidia.com/gpu:NoSchedule by default, consistent with every other DaemonSet the GPU Operator ships, so it schedules onto GPU nodes regardless of install method (Helm, OLM, or a hand-written ClusterPolicy CR).

---

Environment (please provide the following information):

• GPU Operator Version: v26.3.3; reverified unfixed against current master at the time of filing — root cause is a static manifest omission, not version-specific behavior.
• Reproduction environment: kind v1.32.2 cluster (control-plane + 3 workers), no cloud provider, no real GPU/vGPU hardware — the bug is at the pod-scheduling level, independent of the driver or vGPU functionality itself.

---

Information to attach (optional if deemed irrelevant)

Not applicable — this is a manifest-completeness finding confirmed via kubectl get daemonset -o jsonpath on desiredNumberScheduled and kubectl get pods. Happy to provide the full kubectl describe daemonset output for both DaemonSets used in the reproduction if useful for triage.