Fault Injection Loop

Skill: fault-injection-loop

Iterative closed-loop skill that runs a prioritized fault-injection experiment pool

2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates

gaps across the matrix, and proposes targeted improvements to attribution modules.

Overview

┌───────────────────────────────────────────────────────────────────────┐
│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │
│               GPU faults first, then crash, Python-hang, signal       │
│                                                                        │
│  repeat until pool exhausted:                                          │
│  1. SUBMIT   → sbatch 2 jobs from pool head                            │
│  2. WAIT     → poll until both jobs leave RUNNING/PENDING              │
│                                                                        │
│  after all jobs done:                                                  │
│  3. ANALYZE  → watch_and_analyze.sh: /log-analysis + /fr-analysis     │
│               per completed job, streaming as jobs finish              │
│  4. SCORE    → compare attribution output vs injected ground truth     │
│  5. AGGREGATE→ build results table; identify systematic failure modes  │
│  6. IMPROVE  → patch log_analyzer/nvrx_logsage.py                     │
│  7. LOOP     → re-run same pool with updated attribution code          │
└───────────────────────────────────────────────────────────────────────┘

Step 0 — Fault Pool Design

The pool is defined as an ordered list of FAULT_TYPE:RANK:ITER:NODES entries

inside scripts/prepare_node_alloc.sh. Default pool (34 experiments, 17 batches):

# GPU hangs — highest priority; full rank sweep across all node counts
GPU_SLEEP:1:5:2   GPU_SLEEP:0:5:2      # 2-node: rank-1, rank-0
GPU_SLEEP:4:5:2   GPU_SLEEP:7:5:2      # 2-node: mid-rank, last-rank
GPU_SLEEP:1:5:4   GPU_SLEEP:0:5:4      # 4-node: rank-1, rank-0
GPU_SLEEP:8:5:4   GPU_SLEEP:15:5:4     # 4-node: mid, last
GPU_SLEEP:1:5:8   GPU_SLEEP:0:5:8      # 8-node: rank-1, rank-0
GPU_SLEEP:16:5:8  GPU_SLEEP:31:5:8     # 8-node: mid, last

# GPU errors — high priority; rank-0 and rank-1 across all node counts
GPU_ERROR:1:5:2   GPU_ERROR:0:5:2
GPU_ERROR:1:5:4   GPU_ERROR:0:5:4
GPU_ERROR:1:5:8   GPU_ERROR:0:5:8

# Crash faults
SIGKILL:1:5:2     SIGKILL:0:5:2
SIGKILL:1:5:4     SIGKILL:1:5:8
SEGFAULT:1:5:2    SEGFAULT:0:5:2
SEGFAULT:1:5:4    OS_ABORT:1:5:2

# Python-level hangs
LOCK_GIL:1:5:2    LOCK_GIL:0:5:2
WORKLOAD_EXC:1:5:2  ASYNC_EXC:1:5:2

# Signals
SIGTERM:1:5:2     SIGINT:1:5:2
SIGSTOP:1:5:2     SIGNAL_EXC:1:5:2

Rank coverage per node count (4 GPUs/node):

| Nodes | Total ranks | rank-0 | rank-1 | mid | last |

|-------|-------------|--------|--------|-----|------|

| 2 | 8 | 0 | 1 | 4 | 7 |

| 4 | 16 | 0 | 1 | 8 | 15 |

| 8 | 32 | 0 | 1 | 16 | 31 |

To run a custom subset, override POOL before calling the script:

POOL="GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2" bash scripts/prepare_node_alloc.sh

Local User Config

Start from the tracked template:

cp scripts/user.env.example scripts/user.env

Then edit scripts/user.env with cluster-specific settings. This file is

sourced by run_session.sh, prepare_node_alloc.sh, watch_and_analyze.sh,

l4_gb200_reduced.sh, and n3_super_gb200_fi.sh. It is required for this skill

to run and is intended to stay local and untracked.

Recommended contents:

PARTITION=gb-nvl-134-135
BASE_EXPERIMENTS_DIR="${HOME}/nvrx-attr-experiments"
MEGATRON_REPO_HOST_PATH="${HOME}/megatron-lm-main"
SHARED_TMP_BASE_DIR="${HOME}/tmp"
WORKSPACE_HOST_PATH="${HOME}/tmp"
CONTAINER_IMAGE="nvcr.io/nvidia/nemo:26.04"
LLM_API_KEY_FILE="${HOME}/.llm_api_key"
JUDGE_API_KEY_FILE="${HOME}/.llm_api_key"
NVRX_LLM_MODEL="nvidia/nemotron-3-super-120b-a12b"
NVRX_LLM_BASE_URL="https://integrate.api.nvidia.com/v1"
JUDGE_MODEL="qwen/qwen3.5-397b-a17b"
JUDGE_BASE_URL="https://integrate.api.nvidia.com/v1"
FR_SEGMENT_SIZE=32

Use user.env for stable site defaults such as partition, container image, and

host paths, plus local LLM credentials and endpoint settings for log-analysis

and the judge. Use per-run environment overrides for experiment-specific

controls such as POOL, WORKLOAD, BATCH_SIZE, FAULT_TYPE,

FAULT_AT_ITER, or FAULT_DELAY.

If you use local Triton/Inductor cache staging, set the cache variables in

scripts/user.env. See scripts/user.env.example for the supported

ENABLE_NFS_CACHE_STAGING, NFS_TRITON_CACHE, and NFS_INDUCTOR_CACHE

entries and workload-specific path examples.

Environment variables:

| Variable | Default | Description |

|---|---|---|

| WORKLOAD | llama4_scout | Select a registered workload by name (see scripts/workloads.conf) |

| ACCOUNT | _(cluster default or scripts/user.env)_ | SLURM account |

| PARTITION | _(cluster default or scripts/user.env)_ | SLURM partition |

| GPUS_PER_NODE | 4 | GPUs per node |

| TIME | 00:30:00 | Per-job wall-clock limit |

| BATCH_SIZE | 2 | Jobs submitted per round |

| POLL_INTERVAL | 30 | Seconds between queue polls |

| BASE_EXPERIMENTS_DIR | ${HOME}/nvrx-attr-experiments | Root for all output |

| MEGATRON_REPO_HOST_PATH | ${HOME}/megatron-lm-main | Host path to the Megatron checkout mounted into the container |

| SHARED_TMP_BASE_DIR | ${HOME}/tmp | Shared filesystem path used for cross-step coordination |

| WORKSPACE_HOST_PATH | ${HOME}/tmp | Host path mounted at /workspace inside the container |

| CONTAINER_IMAGE | nvcr.io/nvidia/nemo:26.04 | Container image used by the workload script |

| LLM_API_KEY_FILE | _unset_ | File containing the log-analysis API key |

| JUDGE_API_KEY_FILE | _unset_ | File containing the judge API key |

| NVRX_LLM_MODEL | nvidia/nemotron-3-super-120b-a12b | Model for log-analysis |

| NVRX_LLM_BASE_URL | https://integrate.api.nvidia.com/v1 | Base URL for log-analysis |

| JUDGE_MODEL | qwen/qwen3.5-397b-a17b | Model for judge scoring |

| JUDGE_BASE_URL | https://integrate.api.nvidia.com/v1 | Base URL for judge scoring |

| FR_SEGMENT_SIZE | 32 | Ranks per segment for coarse FR scoring |

| SBATCH_SCRIPT | scripts/l4_gb200_reduced.sh | Job script to submit |

| POOL | _(default pool above)_ | Space-separated experiment triplets |

Registered workloads (scripts/workloads.conf)

| Name | Script | Base dir | Description |

|---|---|---|---|

| llama4_scout | l4_gb200_reduced.sh | ${HOME}/nvrx-attr-experiments | Llama4-Scout (reduced layers) on GB200; minimum supported size is 2 nodes |

| n3_super | n3_super_gb200_fi.sh | ${HOME}/nvrx-attr-experiments | Nemotron3-Super on GB200; minimum supported size is 8 nodes |

Workload note:

• llama4_scout requires at least 2 nodes.
• n3_super requires at least 8 nodes. Its default registered pool contains only 8-node experiments.

# Run the full pool against the validated example workload
bash scripts/prepare_node_alloc.sh

# Run a custom subset against llama4_scout
POOL="GPU_SLEEP:1:5:2 SIGKILL:1:5:2" WORKLOAD=llama4_scout bash scripts/prepare_node_alloc.sh

Step 1 & 2 — Batched Submission + Wait (automated)

bash scripts/prepare_node_alloc.sh

The script loops: submit 2 jobs → poll squeue every 30 s until both finish →

submit next 2. Progress is printed inline:

>>> Batch 1: experiments 1–2 of 34
  submitted: GPU_SLEEP rank=1  iter=5 nodes=2 -> job=1850
  submitted: GPU_SLEEP rank=0  iter=5 nodes=2 -> job=1851
  waiting for GPU_SLEEP:1:5:2 GPU_SLEEP:0:5:2 (1850,1851) ... 30s 60s done.
>>> Batch 2: experiments 3–4 of 34
  ...

A session directory and TSV tracking file are created at launch time:

${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/
  experiments.tsv                              ← tracking file (all job IDs + paths)
  n<N>_<FAULT>_r<R>_i<I>/                     ← one subdir per experiment
    logs/slurm/<JOB_ID>.launch.out
    logs/slurm/<JOB_ID>.*.1.main_workload.log  ← log-analysis input
    checkpoints/                               ← fr-analysis input (FR dumps)
    tensorboard/
  experiments_report.md                        ← generated by watch_and_analyze.sh

Tracking file columns: JOB_ID FAULT_TYPE RANK ITER NODES EXPERIMENT_DIR

Step 3 — Analyze All Experiments

Run the watcher/analyzer — it reads the tracking file and processes each experiment

as its job state leaves RUNNING/PENDING (works whether jobs are still running or

already done):

bash scripts/watch_and_analyze.sh \
    ${BASE_EXPERIMENTS_DIR}/fault_injection/<YYYYMMDD_HHMMSS>/experiments.tsv

The watcher:

• Reads each row from the tracking TSV
• Calls nvrx_logsage.py --exclude_nvrx_logs and parses the text output to get

restart_decision and attribution_text

• Calls FR analysis as python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*" and passes the raw table output to the judge
• Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)
• Appends a scored row to <session>_report.md as a markdown table row
• Repeats until all experiments are analyzed

To also run the sub-skills interactively for a single experiment:

/log-analysis --log-path "${EXPERIMENT_DIR}/logs/slurm/${JOB_ID}.*.1.main_workload.log"
/fr-analysis  --fr-path "${EXPERIMENT_DIR}/checkpoints" -p "_dump_*"

Step 4 — Score Each Experiment

Scoring is performed by scripts/score_attribution.py, an LLM judge that

receives the ground truth, the filtered raw log, the logsage attribution output, and the FR

analysis output, then returns structured JSON scores with a reasoning note.

| Column | Values | Meaning |

|---|---|---|

| restart_correct | true / false / N/A | Restart decision matches expected for this fault type |

| rank_primary | true / false / partial | Injected rank is the primary root-cause in attribution |

| rank_any | true / false | Injected rank mentioned anywhere in attribution |

| fault_described | true / false / partial | Fault nature (hang/crash/signal/exception) correctly described |

| fr_rank_correct | rank / node / segment / false / no_dumps | FR analysis narrows correctly to the injected rank, exactly one GPUS_PER_NODE rank block containing that rank, the configured FR_SEGMENT_SIZE rank block containing the injected rank, or fails to narrow usefully |

| judge_notes | string | One-sentence summary of the main gap or confirmation |

The judge is given:

• Ground truth: fault_type, rank, iter, nodes
• Expected restart decision + rationale (derived from score_attribution.py:_RESTART_TABLE)
• Filtered raw log (last 400 lines, same exclude_nvrx_logs filtering as logsage)
• Raw logsage stdout (5-field text format)
• Raw FR analysis table output from fr_attribution.py --fr-path ... -p "_dump_*"
• GPUS_PER_NODE and FR_SEGMENT_SIZE to map the injected rank to exact node-sized and segment-sized scopes for FR scoring

Default judge model: qwen/qwen3.5-397b-a17b. Override with --model in score_attribution.py.

Default segment size for FR scope scoring: 32 ranks. Override with FR_SEGMENT_SIZE.

Step 5 — Aggregate Results

The canonical output of the loop is the markdown table in <session>_report.md.

When summarizing results for users, prefer linking to that file and reproducing the

same table shape rather than flattening the results into plain prose.

The report markdown table from watch_and_analyze.sh gives a matrix view. Look for

patterns across rows:

| FAULT_TYPE | NODES | RANK | restart_correct | rank_primary | rank_any | fault_described | fr_rank_correct | judge_notes |
|------------|-------|------|-----------------|--------------|----------|-----------------|-----------------|-------------|
| GPU_SLEEP  |   2   |  0   |      true       |    false     |   true   |      true       |      true       | rank-0 identified only in secondary issues |
| GPU_SLEEP  |   2   |  1   |      true       |     true     |   true   |      true       |      true       | correct on all dimensions |
| GPU_ERROR  |   2   |  1   |      false      |    false     |  false   |     partial     |      true       | LLM issued RESTART; rank not mentioned |
| SIGKILL    |   2   |  0   |      true       |    false     |  false   |     false       |      true       | attribution describes timeout not kill signal |

Common failure mode patterns and their meaning:

| Pattern | Interpretation |

|---|---|

| rank_primary=false, rank_any=true | Rank detected but treated as collateral; logsage putting it in secondary issues |

| rank_any=false for rank-0 | Rank-0 hang silences watchdog on other ranks; logsage lacks rank-0 signal |

| fault_described=partial for crash types | Crash keywords present but fault type not specifically named |

| restart_correct=false for GPU_ERROR | LLM conflating hardware error with recoverable hang |

| fr_rank_correct=no_dumps | NCCL watchdog did not fire before job ended — adjust TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC |

| fr_rank_correct=node | FR isolated exactly one GPUS_PER_NODE rank block containing the injected rank, but not the exact rank |

| fr_rank_correct=segment | FR isolated the configured FR_SEGMENT_SIZE rank block containing the injected rank, but not the exact node/rank |

Step 6 — Identify and Apply Improvements

FR analysis

Deterministic graph algorithm — do not modify automatically.

Note misidentifications and escalate to the team.

Log analysis (safe to modify)

| Observation | Target location | Suggested fix |

|---|---|---|

| Wrong restart for hang | nvrx_logsage.py fast-path | Strengthen NCCL timeout → RESTART IMMEDIATE mapping |

| Missing rank in attr text | nvrx_logsage.py prompt | Extract rank from NCCL watchdog lines; add regex |

| Crash misclassified as hang | nvrx_logsage.py | Add SIGKILL/SEGFAULT/GPU_ERROR keyword patterns |

| ERRORS NOT FOUND when errors exist | return_application_errors config | Loosen error extraction filter |

| rank-0 not detected | prompt or fast-path | Add explicit rank-0 hang heuristic (other ranks silent) |

| attr off by many iters | prompt context | Increase weight of iteration-stamped log lines |

| LLM wrong on GPU_ERROR | prompt | Distinguish cudaError → crash from NCCL timeout → hang |

Editable file: attribution/log_analyzer/nvrx_logsage.py

After each patch, re-run the same pool subset that previously failed:

POOL="GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2" bash scripts/prepare_node_alloc.sh

Step 7 — Loop

Increment experiment counter. Suggested sweep order across code-change iterations:

• Iteration 1: full default pool (34 experiments)
• Iteration 2: targeted re-run of all failing cells from iteration 1
• Iteration 3: expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps
• Iteration 4: add SEGFAULT and LOCK_GIL 4-node/8-node coverage

Stop condition: all cells pass all four scoring dimensions for two consecutive

code-change iterations.

Adapting A SLURM Script For The Feedback Loop

The feedback loop is not tied to l4_gb200_reduced.sh, but your sbatch script must

match a small contract so the loop can submit, analyze, and score each run.

Required changes for a custom workload script:

• Accept these exported variables from prepare_node_alloc.sh:

FAULT_TYPE, FAULT_RANK, FAULT_AT_ITER, EXPERIMENT_DIR, BASE_EXPERIMENTS_DIR,

and GPUS_PER_NODE.

• Write the main training log to:

${EXPERIMENT_DIR}/logs/slurm/${SLURM_JOB_ID}.*.1.main_workload.log

so watch_and_analyze.sh can find it.

• Write NCCL flight-recorder dumps under ${EXPERIMENT_DIR}/checkpoints/.
• Emit a fault-injection marker when the fault is injected.

watch_and_analyze.sh uses this to decide whether the run reached the injection point.

• Preserve the per-experiment directory layout:

logs/slurm/, checkpoints/, and tensorboard/.

This has only been validated with Megatron-LM because the current run-valid check and

fault markers depend on Megatron's debug_fault_injection.py behavior. If you adapt the

loop to another framework, update both the sbatch script and watch_and_analyze.sh.

Appendix A: SBATCH_SCRIPT fault parameters

The example SBATCH_SCRIPT reads these env vars from prepare_node_alloc.sh via --export:

| Variable | Default | Description |

|---|---|---|

| FAULT_AT_ITER | 5 | Training iteration at which to inject |

| FAULT_DELAY | 15 | Delay in seconds before fault injection after the iteration anchor |

| FAULT_RANK | 1 | Global rank to inject [0, total_ranks) |

| FAULT_TYPE | GPU_SLEEP | Megatron fault type enum name |

| GPUS_PER_NODE | 4 | GPUs per node (used to compute TOTAL_TASKS) |

| EXPERIMENT_DIR | ${BASE_EXPERIMENTS_DIR}/fault_injection/n${SLURM_NNODES}_${FAULT_TYPE}_r${FAULT_RANK}_i${FAULT_AT_ITER} | Per-experiment output root |

| BASE_EXPERIMENTS_DIR | ${HOME}/nvrx-attr-experiments | Shared root (datacache, triton/inductor caches) |

Valid FAULT_TYPE values:

GPU_ERROR, GPU_SLEEP, WORKLOAD_EXC, ASYNC_EXC, SIGNAL_EXC, OS_ABORT,

LOCK_GIL, SEGFAULT, SIGINT, SIGKILL, SIGTERM, SIGSTOP

Appendix B: Single-experiment manual run

# Manual runs land under fault_injection/manual/ by default (no session dir needed)
EXPERIMENT_DIR=${HOME}/nvrx-attr-experiments/fault_injection/manual/n2_GPU_SLEEP_r1_i5
mkdir -p ${EXPERIMENT_DIR}/logs/slurm ${EXPERIMENT_DIR}/checkpoints ${EXPERIMENT_DIR}/tensorboard

sbatch \
    --nodes=2 \
    --output=${EXPERIMENT_DIR}/logs/slurm/%j.launch.out \
    --error=${EXPERIMENT_DIR}/logs/slurm/%j.launch.err \
    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=${EXPERIMENT_DIR} \
    scripts/l4_gb200_reduced.sh

Optional site-specific cleanup:

export CONTAINER_CLEANUP_CMD='
ENROOT_DIR="/var/lib/enroot/data/$(id -u)"
rm -rf "${ENROOT_DIR:?}"/* 2>/dev/null || true
echo "$(hostname): / $(df -h / | tail -1 | awk "{print \$3\" used, \"\$4\" avail\"}")"
'