Do not ship GPUCluster CR as a helm chart resource

[rajathagasthya]

Description

Shipping the GPUCluster CR as a chart resource required a helm.sh/resource-policy: keep annotation so that CR deletion would be done by the pre-delete hook (that would use the finalizer) rather than helm. That workaround caused some issues such as helm uninstall printing a misleading "kept due to the resource policy" message for a CR deleted by the hook, and setting gpuCluster.deployCR=false during an upgrade leaving a stale CR behind that conflicts with a newly deployed ClusterPolicy.

This change stops shipping the CR in the chart. The chart now installs only the operator and CRDs. Users will need to create a GPUCluster CR directly (see config/samples/nvidia_v1alpha1_gpucluster.yaml).

Uninstall ordering is now the user's responsibility. A GPUCluster CR must be deleted while the operator is still running, since deleting it after helm uninstall leaves its finalizer unprocessed and the CR stuck in Terminating state. To avoid that from happening silently, the pre-delete hook is repurposed to fail the uninstall while GPUCluster CRs exist, and waits for CRs that are already terminating so an uninstall issued right after a delete still succeeds. The hook only lists CRs; it never deletes them.

Usage

Default install (ClusterPolicy, device-plugin stack) — unchanged:

helm install gpu-operator <chart> -n gpu-operator --create-namespace

Install in DRA mode (GPUCluster):

helm install gpu-operator <chart> -n gpu-operator --create-namespace \
  --set clusterPolicy.deployCR=false \
  --set driver.nvidiaDriverCRD.enabled=true \
  --set driver.nvidiaDriverCRD.deployDefaultCR=true
kubectl apply -f config/samples/nvidia_v1alpha1_gpucluster.yaml

Switching DRA → ClusterPolicy later: delete the GPUCluster CR first (while the operator is running, so its finalizer drains the DRA stack), then flip the flag:

kubectl delete gpucluster gpu-cluster
helm upgrade gpu-operator <chart> -n gpu-operator --set clusterPolicy.deployCR=true

The reverse flip (ClusterPolicy → DRA) has different behavior: the ClusterPolicy CR is chart-managed, so helm upgrade --set clusterPolicy.deployCR=false deletes it and its operands; then create the GPUCluster CR.

Uninstall in DRA mode: delete the GPUCluster CR first and wait for it to be gone (the finalizer needs the operator running), then uninstall:

kubectl delete gpucluster gpu-cluster
helm uninstall gpu-operator -n gpu-operator

Uninstall in ClusterPolicy mode is unchanged: just helm uninstall.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

[coveralls]

coverage: 38.367% (-0.02%) from 38.385% — remove-gpucluster-cr-from-chart into main

[cdesiniotis]

If we decide to do this, I assume we will maintain config/samples/nvidia_v1alpha1_gpucluster.yaml such that it always points to the latest DRA driver release?

[tariq1890]

I assume we will maintain config/samples/nvidia_v1alpha1_gpucluster.yaml such that it always points to the latest DRA driver release?

Yes, that is my understanding.

[rajathagasthya]

If we decide to do this, I assume we will maintain config/samples/nvidia_v1alpha1_gpucluster.yaml such that it always points to the latest DRA driver release?

Yes, I added a Renovate rule to always keep the DRA driver image current in the sample CR.

The sample CR can drift with the API fields over time, but we can be deliberate about it when making API changes. Perhaps even add a make target later that can fail when it's out of sync with the CRD.

[rahulait]

release-image-list and release-rc-assets workflows will also need an update to detect updated dra-driver image and generate assets accordingly.

[rajathagasthya]

We've decided to bundle the CR in the chart.