GPU health checking

The NVMLDeviceHealthCheck feature gate enables continuous GPU health monitoring through the NVIDIA Management Library (NVML). When a GPU enters an error state, the driver applies a device taint to the corresponding ResourceSlice, signaling the Kubernetes scheduler to avoid placing new workloads on the affected device.

With this feature enabled, unhealthy devices are tainted in the ResourceSlice so the scheduler stops placing new workloads on them.

Feature status

NVMLDeviceHealthCheck is an Alpha feature gate, disabled by default.

NVMLDeviceHealthCheck is mutually exclusive with the DynamicMIG, PassthroughSupport, and MPSSupport feature gates. Refer to the feature gate constraints documentation for more details.

Prerequisites

• The DRADeviceTaints Kubernetes feature gate must be enabled on the kube-apiserver, kube-controller-manager, and kube-scheduler. In Kubernetes v1.34 and 1.35, DRADeviceTaints is disabled by default and must be explicitly enabled. In Kubernetes v1.36, the feature gate is enabled by default.
• NVIDIA DRA driver v0.4.0 or later installed via Helm.

How it works

When enabled, the GPU kubelet plugin starts an NVML event monitor. When a health event occurs on a GPU, the driver updates the ResourceSlice for the affected device with a device taint.

The monitor tracks three event categories:

Taint effects

The driver applies the following Kubernetes device taint effects:

• NoSchedule: The Kubernetes scheduler does not allocate the device to new workloads. Existing workloads that already hold a claim to the device are not evicted.
• None: This taint is informational only. Scheduling is not affected, but the taint is visible in the ResourceSlice.

The driver only applies the None and NoSchedule effects. The driver never evicts running workloads with NoExecute. If you want to implement eviction rules when a device is tainted, create a DeviceTaintRule with effect: NoExecute. Follow the Kubernetes documentation for taints set by an admin for details.

XID errors

XID codes are NVIDIA-defined error identifiers for GPU hardware and firmware conditions. The driver classifies XIDs as fatal or non-fatal. Fatal XIDs produce a NoSchedule taint and non-fatal XIDs produce a None taint. Refer to the NVIDIA XID Errors documentation for information about XID errors and codes.

By default, the driver classifies the following XID errors as non-fatal because they indicate application-level failures rather than hardware degradation. The following table identifies these errors:

You can classify additional XID errors as non-fatal by specifying a comma-separated list in the --additional-xids-to-ignore CLI argument or the ADDITIONAL_XIDS_TO_IGNORE environment variable.

Enabling the feature

Add the following to your Helm values:

featureGates:
  NVMLDeviceHealthCheck: true

Then apply the change with helm upgrade:

helm upgrade dra-driver-nvidia-gpu oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
    --namespace dra-driver-nvidia-gpu \
    --reuse-values \
    --set featureGates.NVMLDeviceHealthCheck=true

View taints on ResourceSlice

Each taint appears on an individual device entry in ResourceSlice.spec.devices, not on the ResourceSlice itself. Use jq to show only device entries that have taints:

kubectl get resourceslices -o json | jq '
  .items[].spec.devices[]
  | select((.taints // []) | length > 0)
  | {
      device: .name,
      taints: [.taints[] | {key, value, effect, timeAdded}]
    }
'

The following output shows an example non-fatal XID event:

    {
  "device": "gpu-0-mig-1g12gb-19-0",
  "taints": [
    {
      "key": "gpu.nvidia.com/xid",
      "value": "43",
      "effect": "None",
      "timeAdded": "2026-07-22T02:24:46Z"
    }
  ]
}

The response includes the following details:

• The device field identifies the affected device entry in the ResourceSlice.
• The key field identifies the health event category, and gpu.nvidia.com/xid indicates an XID error.
• The value field contains the decimal XID code reported by NVML, which is 43 in this example.
• The effect field is None because the driver classifies XID 43 as non-fatal by default, so this taint records the event without preventing new allocations. For fatal XID codes, the effect is NoSchedule, which prevents new allocations that do not tolerate the taint.
• The timeAdded field records when the API server added the taint. The GPU kubelet plugin leaves this field unset when it adds or changes a taint so that the API server assigns the timestamp.

Recovering from an unhealthy device

Device taints persist until the GPU kubelet plugin restarts. There is no automated taint removal in the current release.

To clear taints after a hardware issue is resolved:

1. Confirm the hardware error is resolved. Use dmesg to check the kernal logs.
2. Restart the GPU kubelet plugin by rolling its kubelet-plugin DaemonSet:

kubectl rollout restart daemonset/dra-driver-nvidia-gpu-kubelet-plugin -n dra-driver-nvidia-gpu

On restart, the GPU kubelet plugin re-evaluates device health. Devices with no active NVML health events will not receive taints.

Kubernetes administrators can use DeviceTaintRule objects to manually remove or override device taints without restarting the driver.

Limitations and considerations

• No automated recovery: Taint removal requires a driver restart or a manual DeviceTaintRule override. The driver does not clear taints when hardware recovers.
• One taint per key per device: Each device holds at most one taint per taint key. If multiple XID events occur on the same device, only the most recent value is retained.
• Mutually exclusive feature gates: Cannot be used with DynamicMIG, PassthroughSupport, or MPSSupport.
• Publish failure handling: If the driver fails to update the ResourceSlice after a health event (for example, due to a transient API server error), the failure is logged but not retried. The ResourceSlice may remain stale until the next successful publish or driver restart.