Miles Rl Training Agent Skill

miles: Enterprise-Grade RL for Large-Scale Model Training

miles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.

When to Use miles

Choose miles when you need:

• Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)
• FP8 or INT4 quantization-aware training
• Bit-wise identical train-inference alignment
• Speculative RL for maximum throughput
• Production stability with enterprise support

Consider alternatives when:

• You want the research-grade original → use slime
• You need flexible backend swapping → use verl
• You want PyTorch-native abstractions → use torchforge

Key Features

Low-Precision Training

• Unified FP8: End-to-end FP8 for both inference and training
• INT4 QAT: 1TB models on single-machine VRAM (H200)
• Rollout Routing Replay (R3): Bit-wise expert alignment for MoE

Performance Optimizations

• Speculative RL: 25%+ rollout speedup with online SFT draft models
• Zero-Copy Weight Sync: CUDA IPC zero-copy mapping
• Partial Rollout: Recycle half-finished trajectories

Train-Inference Alignment

• TIS/MIS: Truncated/Masked Importance Sampling for off-policy correction
• Kernel-level optimization: FlashAttention-3, DeepGEMM integration

Installation

# Recommended: Docker
docker pull radixark/miles:latest
docker run --rm --gpus all --ipc=host --shm-size=16g \
  -it radixark/miles:latest /bin/bash

# From source
git clone https://github.com/radixark/miles.git
cd miles
pip install -r requirements.txt
pip install -e .

Quick Start

miles inherits slime's configuration system. Basic training:

python train.py \
    --advantage-estimator grpo \
    --model-name qwen3-30b-a3b \
    --hf-checkpoint /path/to/qwen3-30b-a3b-hf \
    --rollout-batch-size 512 \
    --n-samples-per-prompt 8

Workflow 1: Large MoE Training

Use this workflow for training large MoE models like DeepSeek V3 or Qwen3-MoE.

Prerequisites Checklist

• [ ] H100/H200 GPUs with FP8 support
• [ ] MoE model (DeepSeek V3, Qwen3-MoE)
• [ ] Docker environment with miles

Step 1: Environment Setup

# FP8 block scaling (recommended for stability)
export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1
export CUDA_DEVICE_MAX_CONNECTIONS=1

Step 2: Configure Training

python train.py \
    --actor-num-gpus-per-node 8 \
    --rollout-num-gpus 8 \
    --hf-checkpoint /path/to/deepseek-v3 \
    --advantage-estimator grpo \
    --tensor-model-parallel-size 8 \
    --expert-model-parallel-size 4 \
    --prompt-data /path/to/data.jsonl \
    --num-rollout 3000

Verification Checklist

• [ ] Model loads without errors
• [ ] Routing decisions are consistent
• [ ] No NaN/Inf in loss values

Workflow 2: Speculative RL Training

Use this workflow for maximum rollout throughput with EAGLE speculative decoding.

How Speculative RL Works

• Small draft model generates candidate tokens
• Target model verifies in parallel
• Draft model updated via online SFT to track policy

Step 1: Enable Speculative Decoding

miles supports EAGLE speculative decoding via SGLang:

python train.py \
    --actor-num-gpus-per-node 8 \
    --hf-checkpoint /path/to/target-model \
    --sglang-speculative-algorithm EAGLE \
    --sglang-speculative-num-steps 3 \
    --sglang-speculative-eagle-topk 1 \
    --sglang-speculative-num-draft-tokens 4 \
    --sglang-speculative-draft-model-path /path/to/draft-model \
    --advantage-estimator grpo \
    --prompt-data /path/to/data.jsonl

Step 2: Enable Online MTP Training (Optional)

For online SFT of draft model during training:

--mtp-num-layers 1 \
--enable-mtp-training \
--mtp-loss-scaling-factor 0.2

Note: Online MTP training requires a torch dist checkpoint with MTP weights. Add --mtp-num-layers 1 during checkpoint conversion from HuggingFace.

Expected Speedup

• Standard rollout: Baseline
• Speculative RL: 25-40% faster rollout
• With partial rollout: Additional 10-15% throughput

Configuration Reference

miles inherits all slime arguments. See slime API Reference for the complete list.

Cluster Resources (from slime)

--actor-num-nodes 1
--actor-num-gpus-per-node 8
--rollout-num-gpus 8
--rollout-num-gpus-per-engine 2
--colocate

Megatron Parallelism (from slime)

--tensor-model-parallel-size 8
--pipeline-model-parallel-size 2
--expert-model-parallel-size 4    # MoE expert parallelism

Speculative Decoding (miles-specific)

--sglang-speculative-algorithm EAGLE
--sglang-speculative-num-steps 3
--sglang-speculative-eagle-topk 1
--sglang-speculative-num-draft-tokens 4
--sglang-enable-draft-weights-cpu-backup
--sglang-speculative-draft-model-path /your/draft/model/path

Online MTP Training (miles-specific)

--mtp-num-layers 1
--enable-mtp-training
--mtp-loss-scaling-factor 0.2

Key Features (Conceptual)

The following features are documented in miles but specific CLI flags may vary. Consult the miles repository for latest configuration.

Unified FP8 Pipeline

End-to-end FP8 sampling and training that eliminates quantization-induced discrepancy causing RL collapse in MoE models.

Rollout Routing Replay (R3)

Records expert routing decisions during SGLang inference and replays them during Megatron training for bit-wise expert alignment.

How R3 Works:

• During SGLang inference, expert routing decisions are recorded
• Routing decisions stored in sample.rollout_routed_experts
• During Megatron training, routing is replayed instead of recomputed
• Ensures identical expert selection between train and inference

INT4 Quantization-Aware Training

Enables single-machine deployment of 1TB+ models (e.g., on H200).

Memory Savings with INT4:

| Model Size | BF16 VRAM | INT4 VRAM | Reduction |

|------------|-----------|-----------|-----------|

| 70B | 140GB | 45GB | 3.1x |

| 235B | 470GB | 150GB | 3.1x |

| 671B | 1.3TB | 420GB | 3.1x |

Train-Inference Alignment

miles achieves "exactly 0 KL divergence" between training and inference through:

• Flash Attention 3
• DeepGEMM
• Batch-invariant kernels from Thinking Machines Lab
• torch.compile integration

Sample Data Structure

miles uses the same Sample dataclass as slime with the rollout_routed_experts field for MoE routing replay:

@dataclass
class Sample:
    prompt: str | list[dict]
    tokens: list[int]
    response: str
    reward: float | dict
    loss_mask: list[int]
    status: Status
    metadata: dict
    rollout_log_probs: list[float]
    rollout_routed_experts: list[list[int]]  # MoE routing for R3

See slime API Reference for the complete Sample definition.

Common Issues and Solutions

Issue: FP8 Training Collapse

Symptoms: Loss explodes, NaN values

Solutions:

• Use block scaling: export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1
• Reduce learning rate: --lr 5e-7
• Ensure MoE routing is consistent between train/inference

Issue: Speculative Draft Drift

Symptoms: Low acceptance rate over time

Solutions:

• Enable online MTP training to keep draft model aligned
• Reduce speculative steps: --sglang-speculative-num-steps 2
• Use CPU backup: --sglang-enable-draft-weights-cpu-backup

Issue: Train-Inference Mismatch

Symptoms: Policy divergence, reward collapse

Solutions:

• Use TIS for off-policy correction: --use-tis --tis-threshold 0.9
• Verify log probs match between SGLang and Megatron
• Enable R3 for MoE models

Supported Models

| Family | Models | MoE Support |

|--------|--------|-------------|

| DeepSeek | R1, V3, V3.2 | Full |

| Qwen | 2, 2.5, 3 (including MoE) | Full |

| Llama | 3, 3.1, 3.3, 4 | Dense only |

| Gemma | 2, 3, 3N | Dense only |

| GLM | 4.5, 4.6, 4.7 | Dense only |

| MiniMax | M2, M2.1 | Full |

Resources

• GitHub: https://github.com/radixark/miles
• Introduction Blog: https://lmsys.org/blog/2025-11-19-miles/
• Slime (upstream): https://github.com/THUDM/slime
• SGLang: https://github.com/sgl-project/sglang