Deploy K8s Gpu Cluster

Deploy a Kubernetes GPU cluster

Preconditions

• Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver

installs may reboot them; no active users or workloads).

• SSH access from the provisioning machine to every host as a sudo-capable

user.

• submodules/kubespray initialized — Kubernetes playbooks fail on missing

kubespray_defaults role imports without it.

• Run everything from the repository root.

Procedure

• Prepare the environment and verify it:

   git submodule update --init --recursive
   ./scripts/setup.sh
   cp -r config.example config

• Edit config/inventory: control plane nodes under

[kube_control_plane] and [etcd], workers under [kube_node] (a

single machine can hold all three roles).

• Preflight — must pass before deploying:

   python3 scripts/validation/deepops_doctor.py --remote --json

• Deploy:

   ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml

This runs Kubespray and installs the NVIDIA GPU Operator. Expect

roughly 45–90 minutes on a first run.

• Validate — the success signal is this, not the play recap:

   python3 scripts/validation/validate_k8s.py --json --cuda-smoke

Require "ok": true with nodes_ready == nodes_total,

gpus_allocatable > 0, and cuda_smoke_ok: true.

Failure branches

• Playbook fails on a transient error: rerun the same playbook;

Kubespray is rerun-safe. A converged rerun reports changed=0.

• Syntax/import error mentioning kubespray_defaults: the submodule is

not initialized; run git submodule update --init --recursive.

• gpus_allocatable: 0: the GPU Operator stack is not ready. Check

kubectl get pods -A | grep -i nvidia — the driver DaemonSet can take

10+ minutes on first deploy; if pods are crash-looping, follow

skills/diagnose-driver-install/.

• CUDA smoke pod stuck Pending: `kubectl -n deepops-validate

describe pod deepops-validate-cuda` — usually no allocatable GPU

(see above) or an image pull problem on airgapped networks (use

--cuda-image to point at a mirrored image).

• Single-node clusters: control plane taints are handled by the

playbook for the single-node case; if pods stay Pending on a multi-role

node, check taints with kubectl describe node <name> | grep -i taint.