Diagnose Driver Install

Diagnose an NVIDIA driver install

Work through these in order; most "driver failures" are one of the first

three and need no reinstall.

1. Are you being fooled by GPU hiding?

On DeepOps Slurm nodes, GPUs are hidden from ordinary SSH sessions by

design. Bare nvidia-smi over SSH reporting No devices were found on an

otherwise healthy node is expected.

srun --gpus=1 nvidia-smi          # the authoritative test on Slurm nodes

If the srun job sees the GPU, the driver is fine. Stop here.

2. Is the hardware visible at all?

lspci | grep -i nvidia

No output → not a driver problem. The GPU is absent, unseated, or bound by

VFIO passthrough or platform firmware; escalate to hardware support before

touching software.

3. Open vs proprietary kernel modules

nvidia-smi: No devices were found immediately after a clean install is the

classic symptom of the wrong module flavor for the GPU generation:

• Turing and newer (T4, A100, H100, RTX PRO Blackwell): open kernel modules

supported — DeepOps default nvidia_driver_ubuntu_use_open_kernel_modules: true

is correct.

• Pascal and older (P100, GTX 10xx): open modules are not supported —

set nvidia_driver_ubuntu_use_open_kernel_modules: false in

config/group_vars/all.yml and rerun the driver play.

Check what is loaded: modinfo nvidia | grep -i license (open modules say

MIT/GPL, proprietary says NVIDIA).

4. Did the kernel module actually build and load?

dkms status                        # driver module state per kernel
dmesg | grep -iE 'nvidia|nvrm' | tail -20
lsmod | grep nvidia

• DKMS shows an error for the running kernel → usually missing headers

(linux-headers-$(uname -r)) or a kernel updated after the driver

install. Install headers or reboot into the matching kernel, then rerun

the driver play.

• Module loaded but nvidia-smi fails → check dmesg for RmInitAdapter or

fallen-off-the-bus errors; these are hardware/firmware territory.

5. Rerun the play, don't hand-fix

After correcting configuration, converge with the playbook rather than

manual package surgery, then validate:

ansible-playbook -l <host> playbooks/nvidia-software/nvidia-driver.yml
python3 scripts/validation/validate_slurm.py --json    # or validate_k8s.py

Reruns are idempotent; a healthy converged rerun reports changed=0.

Kubernetes-specific notes

GPU Operator crash-looping driver pods: `kubectl logs -n <gpu-operator

namespace> <driver-pod>` usually names the same root causes — missing

headers, wrong module flavor, or a node reboot needed. Fix via the operator

values or node state, not by installing drivers by hand on the host (the

operator owns the driver on Kubernetes nodes unless DeepOps was configured

for host drivers).