Dgx Diagnose Agent Skill

Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM/SGLang container bugs, MIG state problems, NVLink/Fabric Manager errors, X/Vulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.

872 tokens

context cost

the whole folder, loaded on every use

files

instructions only

copies elsewhere

how many repositories repackaged it

1211

stars on the repo

on the repository, not the skill itself

Install

one command, takes just this skill from the repository

npx skills add https://github.com/NVIDIA/dgx-spark-playbooks --skill dgx-diagnose

The instruction itself

15 sections, as written by the author

DGX Station Diagnostics

Diagnose common DGX Station issues. Run through the checks below to identify the problem.

Step 1. Gather system state

Run these commands and analyze the output:

# GPU status
nvidia-smi

# GPU device list with indices
nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader

# Driver version
nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1

# MIG state
nvidia-smi -i 1 -q 2>/dev/null | grep -i "MIG Mode" || echo "Could not query MIG on device 1"

# Fabric Manager
systemctl is-active nvidia-fabricmanager

# GPU processes
sudo fuser -v /dev/nvidia* 2>/dev/null || echo "No GPU processes found"

# Docker containers using GPUs
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" 2>/dev/null

Step 2. Match symptoms to known issues

Based on the gathered state and the user's reported problem, check for these known issues:

CUDA crashes with `--gpus all`

Cause: Mixed coherency — GB300 (ATS) and RTX PRO (non-ATS) cannot share a CUDA context.

Fix: Use --gpus '"device=N"' targeting only the GB300.

Model running on wrong GPU (RTX PRO instead of GB300)

Check: The device index in the docker command vs actual GPU indices.

Fix: Verify with nvidia-smi --query-gpu=index,name --format=csv,noheader and correct the --gpus flag.

vLLM crash / FlashInfer buffer overflow

Check: Container version — docker inspect vllm-server | grep Image

Fix: Use nvcr.io/nvidia/vllm:26.01-py3. Version 25.10 has a known FlashInfer bug on DGX Station.

SGLang CUDA errors

Check: Container tag — must be cu130 for Blackwell SM103.

Fix: Use lmsysorg/sglang:latest-cu130.

CUDA OOM despite 279 GB HBM

Check: --max-model-len / --context-length and memory utilization settings.

Fix: Reduce context length or lower --gpu-memory-utilization / --mem-fraction-static.

`nvidia-smi -mig 1` returns "In use by another client"

Check: sudo fuser -v /dev/nvidia* — GPU processes must be stopped first.

Fix: Stop all GPU workloads, then retry.

NVLink errors after disabling MIG

Check: systemctl is-active nvidia-fabricmanager

Fix: sudo systemctl start nvidia-fabricmanager

X server crash after nvidia-xconfig -a

Fix: sudo cp /etc/X11/xorg.conf.nvidia-xconfig-original /etc/X11/xorg.conf

Vulkan VK_ERROR_INITIALIZATION_FAILED

Cause: CUDA initialized before Vulkan, binding to GB300.

Fix: Run CUDA and Vulkan workloads in separate processes. For Vulkan apps: __GL_DeviceModalityPreference=2 ./your_app

HuggingFace 401 / token errors

Fix: Pass token inline: -e HF_TOKEN="hf_...". Don't rely on shell export for background Docker tasks.

Port already in use

Check: lsof -i :<PORT>

Fix: Stop the conflicting process or use a different host port: -p 8001:8000.

Step 3. Report findings

Tell the user:

What the issue is
Why it happens (root cause)
The specific command to fix it
How to verify the fix worked

Other skills for the same job

different authors, same section of the catalogue

Datacommons Client

by christophacham

×3

Work with Data Commons, a platform providing programmatic access to public statistical data from global sources. Use this skill when working with demographic data, economic indicators, health statistics, environmental data, or any public datasets available through Data Commons. Applicable for querying population statistics, GDP figures, unemployment rates, disease prevalence, geographic entity resolution, and exploring relationships between statistical entities.

Neuropixels neural recording analysis. Load SpikeGLX/OpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen/IBL curation, AI-assisted visual analysis, for Neuropixels 1.0/2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

20k tokens

Senior Data Scientist

by ComeOnOliver

×3

World-class data science skill for statistical modeling, experimentation, causal inference, and advanced analytics. Expertise in Python (NumPy, Pandas, Scikit-learn), R, SQL, statistical methods, A/B testing, time series, and business intelligence. Includes experiment design, feature engineering, model evaluation, and stakeholder communication. Use when designing experiments, building predictive models, performing causal analysis, or driving data-driven decisions.

Python interface to OpenMS for mass spectrometry data analysis. Use for LC-MS/MS proteomics and metabolomics workflows including file handling (mzML, mzXML, mzTab, FASTA, pepXML, protXML, mzIdentML), signal processing, feature detection, peptide identification, and quantitative analysis. Apply when working with mass spectrometry data, analyzing proteomics experiments, or processing metabolomics datasets.

Parallel/distributed computing. Scale pandas/NumPy beyond memory, parallel DataFrames/Arrays, multi-file processing, task graphs, for larger-than-RAM datasets and parallel workflows.

35k tokens

Tensorboard

by Orchestra-Research

×1

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit

15k tokens