Skip to content

Mount ServiceAccount token when dcgm-exporter env needs API access - #2721

Open
rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:fix-dcgm-exporter-token-for-configmap-data
Open

Mount ServiceAccount token when dcgm-exporter env needs API access#2721
rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:fix-dcgm-exporter-token-for-configmap-data

Conversation

@rajathagasthya

@rajathagasthya rajathagasthya commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

The dcgm-exporter DaemonSet sets automountServiceAccountToken=false and re-enables it only when pod metadata enrichment is turned on through enablePodLabels/enablePodUID fields. Users who configure the same features through dcgmExporter.env are missed by that condition, and setting DCGM_EXPORTER_CONFIGMAP_DATA crashes the exporter on startup since it cannot read the ServiceAccount token. This worked before the token was made opt-in.

Mount the token when the user-provided env reads a ConfigMap or enables pod labels/pod UID. A DCGM_EXPORTER_CONFIGMAP_DATA value of none` is the exporter default meaning no ConfigMap is read, so it does not count. Configurations that make no API calls keep the default.

Apply the same env check to the cluster-scoped RBAC gate. Enrichment through raw env otherwise got a token but no matching permissions, leaving the pod informer with only the namespaced Role, which grants pod reads in the operator namespace without watch. The ConfigMap case keeps a separate condition since it needs no cluster-scoped RBAC: the namespaced Role already grants configmaps get/list and its RoleBinding is ungated.

Testing

Metrics are available when DCGM_EXPORTER_CONFIGMAP_DATA is set:

kubectl create configmap -n gpu-operator custom-metrics --from-literal=metrics=\
'DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).'

kubectl patch clusterpolicy cluster-policy --type merge -p \
'{"spec":{"dcgmExporter":{"enablePodLabels":false,"enablePodUID":false,"env":[{"name":"DCGM_EXPORTER_CONFIGMAP_DATA","value":"gpu-operator:custom-metrics"}]}}}'

Metrics are available when pod enrichment is set via env:

kubectl patch clusterpolicy cluster-policy --type merge -p \
'{"spec":{"dcgmExporter":{"env":[{"name":"DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS","value":"true"}]}}}'

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

@rajathagasthya
rajathagasthya force-pushed the fix-dcgm-exporter-token-for-configmap-data branch from eed3a44 to bccb920 Compare August 6, 2026 16:49
@rajathagasthya rajathagasthya self-assigned this Aug 6, 2026
@rajathagasthya
rajathagasthya marked this pull request as ready for review August 6, 2026 20:56
The dcgm-exporter DaemonSet sets automountServiceAccountToken=false and
re-enables it only when pod metadata enrichment is turned on through the
typed enablePodLabels/enablePodUID fields. Users who configure the same
features through raw dcgmExporter.env are missed by that condition, and
setting DCGM_EXPORTER_CONFIGMAP_DATA crashes the exporter on startup
since it cannot read the ServiceAccount token. This worked before the
token was made opt-in.

Mount the token when the user-provided env reads a ConfigMap or enables
pod labels / pod UID. A DCGM_EXPORTER_CONFIGMAP_DATA value of "none" is
the exporter default meaning no ConfigMap is read, so it does not count.
Configurations that make no API calls keep the default.

Apply the same env check to the cluster-scoped RBAC gate. Enrichment
through raw env otherwise got a token but no matching permissions,
leaving the pod informer with only the namespaced Role, which grants pod
reads in the operator namespace without watch. The ConfigMap case keeps
a separate condition since it needs no cluster-scoped RBAC: the
namespaced Role already grants configmaps get/list and its RoleBinding
is ungated.

The daemonset test compared gpuv1.EnvVar against the container's
corev1.EnvVar, which can never be equal. Both render as v1.EnvVar in
failure output, so the mismatch stayed invisible until a test case set
env on the ClusterPolicy.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
@rajathagasthya
rajathagasthya force-pushed the fix-dcgm-exporter-token-for-configmap-data branch from bccb920 to 0cc3160 Compare August 7, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant