Nightly Sync

Nightly Sync: Main to Dev

This skill is read by the automated sync bot during the nightly-sync-main-to-dev

workflow. It contains all domain knowledge for merging main into dev, resolving

conflicts, iterating on CI, and shipping the PR.

Phase 1: Create the Sync Branch and Merge

Branch Setup

• Create branch $BRANCH from origin/dev
• Merge: git merge origin/main --no-edit
• Resolve conflicts surgically. Do NOT use global -X theirs, and do NOT

blanket-checkout main's version of a shared file. Main's version may be

taken wholesale only for files in "Files to Override from Main" below, or

when you have identified a specific main commit that intentionally removes

the dev-only code. For all other conflicts, combine both sides so recent

dev-only additions remain present.

Preserving Dev-Only Additions

Do NOT blanket-override all shared files with main's version. Dev has features

not yet in main (new classes, new modules, new tests). The merge preserves both

sides' non-conflicting additions — only intervene where there is an actual

conflict.

Squash-Merge Chain Detection

Dev often develops features as a chain of PRs (PR1 → PR2 → PR3) where each

builds on the last. When PR1 is squash-merged to main, git sees main's squashed

version and dev's original commits as unrelated changes. A conflict resolution

that blindly picks main can silently discard PR2/PR3's improvements on dev.

After the merge, check for this pattern:

• For each conflicted file, run git log --oneline origin/dev -- <file> to

see if dev has commits that came AFTER the code main is bringing in.

• If dev has follow-up commits (bug fixes, refactors, extensions), **favor

dev's version** for those sections.

• If the conflict is just main bringing in a clean copy of what dev already

has (no follow-ups), main's version is fine.

Practical check: run git diff origin/dev -- <file> on conflicted files. If

dev's code was removed or reverted, investigate whether dev's version is the

more evolved one.

Real examples from PR #4291:

• emerging_optimizers.py: Main's version was MORE complete — it squash-merged

dev's PRs plus added more. Taking main for that section was correct.

• distrib_optimizer.py: Main overwrote dev's GroupedQuantizedTensor support.

Had to restore _is_distopt_quantized_param and the expanded

_expand_quantized_param_shard_for_cast loop while keeping main's NVFP4

additions. This required a surgical merge combining sections from both.

Key insight: squash-merge chains can go in EITHER direction. Sometimes main

is ahead (it squash-merged dev's work + more), sometimes dev is ahead (it has

follow-up PRs). Always diff both ways before deciding which version to favor.

Real example from PR #4882 / PR #4318:

• transformer_engine.py: main had unrelated TEFusedMLP refactors, while dev

had the new TEFusedDenseMLP class. The sync kept the config flag and test

but dropped the class and gpt_layer_specs.py selection. That is a merge

accident: restore the dev class and selection while preserving main's

TEFusedMLP.as_mlp_submodule refactor.

Files to Override from Main

These files have known semantic conflicts where dev's versions reference args

or APIs that main removed or renamed. Take main's version with

git checkout origin/main -- <file>:

• megatron/training/training.py — references dev-only args
• megatron/training/initialize.py — references dev-only args
• megatron/training/utils.py — references dev-only args
• megatron/training/datasets/data_samplers.py — references dev-only args
• megatron/core/optimizer/layer_wise_optimizer.py — constructor signature

Caveat for ALL overrides: After taking main's version of any file, you

MUST run the API Mismatch Detection procedure (see below) on that file.

Taking main's caller code while keeping dev's callee implementations is the

1 source of sync bugs.

**IMPORTANT: Do NOT take main's pyproject.toml, uv.lock, or

docker/Dockerfile.ci.dev.** These three files are a tightly coupled

triple — the Dockerfile's uv sync command must match the dependency

groups in pyproject.toml, and uv.lock must be consistent with both.

Main's versions are missing dev-only dependencies (e.g.

fast-hadamard-transform, correct TransformerEngine revision) and the

--group no_pypi_wheels flag needed to install them. Keep dev's versions

of all three files.

**IMPORTANT: .github/CODEOWNERS must NEVER be modified by the sync

bot under any circumstances.** Dev's CODEOWNERS is intentionally

different from main's — do not take main's version, do not merge them,

do not touch the file. If the merge produces a conflict or a non-zero

diff against origin/dev on this path, restore dev's version verbatim:

git checkout origin/dev -- .github/CODEOWNERS

Then verify with git diff origin/dev -- .github/CODEOWNERS — output

must be empty. Modifying CODEOWNERS triggers spurious reviewer

requests and conflicts with the dev team's governance; rolling back a

CODEOWNERS change after the PR lands is painful.

NEVER manually edit uv.lock. It is a machine-generated lockfile. If

it needs to change, it must be regenerated with uv lock inside a CUDA

container (see .claude/skills/build-and-test/SKILL.md).

Git Source Reconciliation (pyproject.toml)

After keeping dev's pyproject.toml, check whether main has added NEW git

sources to [tool.uv.sources] that don't exist in dev's version. Main's

merged code may import from packages only available at specific git revisions.

• Diff the [tool.uv.sources] sections:

git show origin/main:pyproject.toml vs git show origin/dev:pyproject.toml

• For each git source in main but not dev, add it to dev's pyproject.toml
• For sources in both but at different revisions, check whether dev's revision

works. If dev's revision is broken (TOML parse errors, missing classes main's

code imports), take main's revision instead.

Real examples from PR #4291:

• nvidia-resiliency-ext: Main's torch.py imports get_write_results_queue

which only existed in main's pinned git revision, not on PyPI. Had to add

main's git source to dev's pyproject.toml.

• nemo-run: Dev's pinned revision had a TOML parse error with uv 0.7.2.

Had to swap to main's revision.

After any changes to pyproject.toml, regenerate uv.lock inside a CUDA

container:

docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \
  bash -c "pip install uv==0.7.2 && cd /workspace && \
  uv venv .venv --system-site-packages && uv sync --only-group build && uv lock"
# Clean up root-owned .venv:
docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \
  bash -c "rm -rf /workspace/.venv"

API Mismatch Detection (Post-Merge Audit)

The merge can create "Frankenstein" code where main's callers use dev's

implementations (or vice versa) with different method signatures. This

compiles fine but fails at runtime.

After the merge, audit cross-boundary call sites:

• Identify files where main's version was taken (-X theirs or explicit

git checkout origin/main)

• For each, find all external call sites: classes it instantiates, methods

it calls on imported objects, functions from other modules it invokes

• Verify method names, parameter counts, and signatures match between the

caller and the implementation in the merged tree

• Pay special attention to "interface" modules (files defining base classes)

— if main and dev evolved the interface differently, every caller and

implementer must agree

Real examples from PR #4291:

• multi_latent_attention.py (main) called off_interface.group_commit()

but dev's interface only had group_offload() — method renamed

• mamba_model.py (main) called init_chunk_handler(3 params) but dev's

interface required 6 params — signature expanded on dev

• mamba_model.py called mark_not_offloadable() but dev had

mark_not_offload() — method renamed

• bulk_offload() did .remove() after bulk_offload_group() already

.pop()d the same item — double-removal from a list

Practical detection:

# For each file taken from main, find what it imports and calls
grep -rn "from <module> import\|<module>\." megatron/
# Cross-reference with the actual implementations in the merged tree

File-Specific Merge Lessons

These lessons were learned from PR #4291. They may recur if the same files

continue to diverge:

• gated_delta_net.py: If the merge creates code calling non-existent helper

methods (e.g. _resolve_cu_seqlens), take dev's version wholesale.

• model_chunk_schedule_plan.py: Watch for missing imports (e.g.

CudaGraphScope) silently dropped during conflict resolution.

• fine_grained_activation_offload.py: Critical interface file used by many

callers. If main and dev have divergent method names/signatures, prefer

dev's implementation and patch main-originated callers to match.

• distrib_optimizer.py: Dev may have broader type abstractions (e.g.

_is_distopt_quantized_param covering both FP8 and GroupedQuantizedTensor).

Main may simplify to explicit type checks. Restore dev's abstractions.

Special Handling: data_schedule.py

Main and dev have completely different classes in this file:

• Main: HybridCPDataLoaderWrapper (imported by main's training.py)
• Dev: BasePackingScheduler, DpBalancedScheduler,

DefaultDynamicCPScheduler, wrap_data_iterator,

get_batch_on_this_rank_for_sequence_packing (imported by pretrain_gpt.py

and tests)

Do NOT take either version wholesale. Keep dev's file and append main's

HybridCPDataLoaderWrapper class (plus any missing imports like

BalancedCPScheduler, Any, List) at the end.

Restore Deleted Files

Compare git ls-tree between origin/main and HEAD to find files in main

that are missing from the merged tree. For each:

• Restore if main's code imports/references it and would break without it

(e.g. hybrid_cp_schedule.py if data_schedule.py imports from it)

• Do NOT restore if dev intentionally deleted it — check

git log origin/dev -- <file> for the deletion commit to understand intent

• When in doubt, check whether any file in the merged tree imports from the

missing file. If nothing imports it, skip it.

Formatting

Run on ALL changed Python files (relative to origin/dev), in this order:

• black (version 24, --config pyproject.toml)
• isort
• Order matters: black first, then isort — reverse order can undo isort's work
• pylint on changed megatron/core/ files — fix missing-docstring and

line-too-long violations before pushing

Pre-push advisory checks

Before every git push in this workflow (the initial push in Phase 1

AND every fix-push in Phase 3), run these bash checks as guidance. They

must never block the push. Review every finding: fix genuine merge accidents

and document intentional main removals or formatting/reordering false positives

in the PR body.

set +e
(
set -euo pipefail

MERGE_COMMIT=$(git rev-list --min-parents=2 --max-count=1 HEAD || true)
if [ -n "$MERGE_COMMIT" ]; then
  DEV_REF="${MERGE_COMMIT}^1"
  MAIN_REF="${MERGE_COMMIT}^2"
else
  DEV_REF="origin/dev"
  MAIN_REF="origin/main"
fi

# 1. CODEOWNERS must be identical to dev's.
if ! git diff --quiet "$DEV_REF" HEAD -- .github/CODEOWNERS; then
  echo "WARNING: .github/CODEOWNERS differs from dev. Restore with:"
  echo "  git checkout $DEV_REF -- .github/CODEOWNERS"
fi

# 2. Dependency-management triple must be identical to dev's.
for f in pyproject.toml uv.lock docker/Dockerfile.ci.dev; do
  if ! git diff --quiet "$DEV_REF" HEAD -- "$f"; then
    # pyproject.toml is allowed to differ ONLY for git source reconciliation
    # (new [tool.uv.sources] entries from main). If you intentionally edited
    # it for that reason, document the reconciliation in the PR body.
    echo "WARNING: $f differs from dev"
  fi
done

# 3. Dev-feature preservation audit.
#
# The most common sync regression is silently dropping a dev-only feature
# that main does not have yet. Pattern:
#   T0: a feature lands on dev
#   T1 > T0: the same feature lands on main (possibly reformatted)
#   The sync runs between T0 and T1. Blindly resolving a conflict in
#   main's favour drops dev's addition wherever main happened to touch a
#   nearby line for an unrelated reason.
#
# For each file the sync touched (modulo skill-sanctioned overrides and
# the dependency triple), find every line that satisfies ALL of:
#   line is on origin/dev        (dev had it)
#   line is NOT on origin/main   (main never owned it)
#   line is NOT in the merged tree  (the merge dropped it)
# Filter out whitespace-only lines and bracket-only lines (they
# frequently differ for cosmetic reasons).
#
# Files in the "Files to Override from Main" list (training.py,
# initialize.py, utils.py, data_samplers.py, layer_wise_optimizer.py)
# are exempt by skill convention — main may legitimately win there.
# CODEOWNERS and the dep triple are checked above; skip them here.

INTENTIONAL_OVERRIDE_REGEX='^(megatron/training/training\.py|megatron/training/initialize\.py|megatron/training/utils\.py|megatron/training/datasets/data_samplers\.py|megatron/core/optimizer/layer_wise_optimizer\.py)$'
SKIP_REGEX='^(pyproject\.toml|uv\.lock|docker/Dockerfile\.ci\.dev|\.github/CODEOWNERS)$'

FINDINGS=0
for f in $(git diff --name-only "$DEV_REF"..HEAD \
            -- '*.py' '*.md' '*.yaml' '*.yml' '*.toml' \
               '*.sh' '*.cpp' '*.cu' '*.h' \
            | sort -u); do
  [[ "$f" =~ $SKIP_REGEX ]] && continue
  [[ "$f" =~ $INTENTIONAL_OVERRIDE_REGEX ]] && continue
  git cat-file -e "HEAD:$f" 2>/dev/null || continue

  missing=$(comm -23 \
              <(git show "$DEV_REF:$f"  2>/dev/null | sort -u) \
              <(git show "$MAIN_REF:$f" 2>/dev/null | sort -u) \
            | comm -23 - <(git show "HEAD:$f" 2>/dev/null | sort -u) \
            | grep -E '[[:alnum:]_]' \
            || true)

  if [ -n "$missing" ]; then
    echo "=== $f ==="
    printf '%s\n' "$missing"
    FINDINGS=$((FINDINGS + $(printf '%s\n' "$missing" | grep -c .)))
  fi
done

if [ "$FINDINGS" -gt 0 ]; then
  echo "WARNING: $FINDINGS potential dev-only line removal(s) detected. For each:"
  echo "  (a) MAIN INTENTIONALLY REMOVED — find the specific commit in"
  echo "      'git log origin/main -- <file>' that removed it; document the"
  echo "      SHA in the PR body, then the drop is acceptable."
  echo "  (b) MERGE ACCIDENT — main never explicitly touched that line."
  echo "      RESTORE the dev line (Edit/Write to put it back)."
  echo "Default to (b); only declare (a) with a specific main commit as evidence."
  echo "This audit is advisory; continue the push after reviewing the findings."
fi

echo "nightly-sync pre-push guidance complete"
)
GUIDANCE_STATUS=$?
if [ "$GUIDANCE_STATUS" -ne 0 ]; then
  echo "WARNING: nightly-sync pre-push guidance failed with status $GUIDANCE_STATUS; allowing the push to continue."
fi
exit 0

All pre-push findings are advisory. The hook must return success even when it

finds a CODEOWNERS difference, potential dev-feature removal, dependency-triple

difference, or an internal audit error. The underlying policies still apply:

restore accidental changes, preserve dev-only features, and document exact main

commits for intentional removals. A warning by itself is never a reason to stop

the workflow or request authorization to continue.

Recent regressions the dev-feature audit would have flagged (all

"merge accident" type from #4659 and #4716):

• transformer_layer.py lost _forward_mlp_router(input_ids=None)
• token_dispatcher.py lost the

num_sms_preprocessing_api=... kwarg on the _HybridEPManager call

• moe_layer.py lost self._maybe_record_overload_factor(...)
• gpt_dynamic_inference_with_coordinator.py lost

from megatron.training.arguments import parse_and_validate_args

• datasets/readme.md lost the dev-only "Packing Scheduler" section
• PR #4882 / PR #4318 dropped the TEFusedDenseMLP implementation and

gpt_layer_specs.py selection while leaving the config flag and unit test

• data_samplers.py / utils.py / training.py kept main's

args.hybrid_context_parallel instead of dev's

args.dynamic_context_parallel (counts as a MERGE ACCIDENT — dev's

reference is present, main's is the deprecated alias that's False

when callers pass --dynamic-context-parallel). These files are on

the override list so the audit treats them as "advisory", but you

should still rename args.hybrid_context_parallel →

args.dynamic_context_parallel on every reference after taking

main's version of these files.

Commit and Push

Phase 1 produces a single commit on the sync branch. The merge itself

creates the merge commit; fold any post-merge work (formatting,

conflict surgery, restored files, regenerated uv.lock) into it

rather than stacking a second commit:

git add -A
git commit --amend --no-edit  # rewrites the merge commit's tree;
                              # parents are preserved.
git push -u origin "$BRANCH"  # only non-force push of the run.

Once pushed, this commit is immutable for the rest of the run.

Phase 3 fixes go into a separate rolling fix commit on top (see

Phase 3 step 4 and the two-commit policy in Rules).

Phase 2: Create the Draft PR

• Title: chore: nightly sync main into dev ($DATE)
• Create as draft: gh pr create --draft
• Body should include:
• Summary of what was synced (number of commits from main)
• Python-only line-change stats, so reviewers can gauge the real

code surface (excluding golden-value JSON, uv.lock, etc.). Compute

with:

     git diff --numstat origin/dev...HEAD -- '*.py' \
       | awk 'BEGIN{a=0;d=0} {a+=$1; d+=$2} END{
           printf "Python lines: +%d / -%d across %d files\n", a, d, NR
         }'

Include the exact line (e.g. Python lines: +1234 / -567 across 42 files)

in the PR body so reviewers see it at a glance.

• List of files where main's version was taken over the merge
• List of files that were deleted in dev but restored (and why)
• Disposition of every pre-push advisory finding, including CODEOWNERS or

dependency-triple differences and potential dev-feature removals. Record

whether each was corrected, intentional (with the exact commit or reason),

or a formatting/reordering false positive. State explicitly if there were

no findings.

• The remerge-diff output (git show --remerge-diff HEAD on the merge

commit) so reviewers can inspect ONLY the conflict resolutions. If the

output is very long, summarize conflicts by file and put the full diff

in a collapsed <details> block. If git is too old for --remerge-diff,

note the git version and describe the merge strategy used instead.

• Save the PR number for later phases
• Add the Run functional tests and Run MBridge tests labels to the

PR immediately after creation. The Run functional tests label ensures

/ok to test triggers the full CI suite (unit tests + functional/

integration tests with 100-step training and golden value comparison).

The Run MBridge tests label triggers the MBridge test suite. Without

these labels, only a lightweight subset runs.

  gh pr edit <PR_NUMBER> --repo $REPO \
    --add-label "Run functional tests" \
    --add-label "Run MBridge tests"

Phase 3: CI Iteration

CI Architecture

• Nemo_CICD_Test is a downstream gate job aggregating unit test,

integration test, and other results. If it fails, investigate the upstream

jobs it depends on — do NOT debug the gate itself.

• Integration tests (H100, GB200) may be skipped for non-maintainer PRs.

This is expected; the Nemo_CICD_Test gate will fail as a result.

• tests/unit_tests/conftest.py imports from megatron.training.training,

so a broken import in training.py (or anything it transitively imports)

cascades to fail ALL test suites. If every test job fails with ImportError,

check the training.py import chain first.

Execution model: one step, no background

You run inside ONE GitHub Actions step. The moment you stop emitting

tool calls, the step ends and the runner container is destroyed. Any

background process you started dies with it. There is NO persistent

session and NO future wakeup. See the workflow prompt's "NO background

tasks" block for the full ban list.

Practical rule: every wait for CI to resolve is a SINGLE foreground Bash

tool call that blocks inline until the wait is resolved.

The Fix-Then-Retrigger Loop

Two nested loops. Do NOT conflate them:

• The outer loop is YOUR sequence of tool calls (each iteration: one

/ok to test, one blocking poll, maybe one fix-and-push). It is NOT a

Bash loop. It advances because you make new tool calls.

• The inner loop is a single blocking Bash tool call using

while true; do ... sleep 120; done. It runs during one iteration of

the outer loop and ends when CI reaches a terminal state for that

iteration.

The outer loop terminates ONLY when Phase 4's gate is satisfied.

Source of truth: gh pr view <PR_NUMBER> --repo $REPO --json statusCheckRollup.

This lists every required check, including external status contexts

(GitLab CI, copy-pr-bot, etc.) that gh api .../actions/runs/.../jobs

does NOT show.

Outer-loop iteration (each iteration is a few tool calls):

• latest_sha=$(git rev-parse HEAD) (one Bash call).
• Post /ok to test $latest_sha on the PR:

gh pr comment <PR_NUMBER> --repo $REPO --body "/ok to test $latest_sha"

• ONE blocking Bash tool call. This is the inner loop. Copy this

template verbatim, only changing REPO and PR:

   REPO='NVIDIA/Megatron-LM'
   PR='<PR_NUMBER>'
   # Names matched case-insensitively, anchored to the START of the name.
   EXEMPT='copy-pr-bot|is-not-external-contributor|greptile|coderabbit|codeowners|.*review|.*approval|codecov|coverage|build-docs|doc-build|readthedocs|sphinx'
   # Sentinel check that tells us CI has fully run. Update this if the
   # aggregate gate job is renamed.
   SENTINEL='Nemo_CICD_Test'

   while true; do
     # Normalize both CheckRun (.status / .conclusion) and StatusContext
     # (.state) entries into the same {name, status, conclusion} shape.
     rollup=$(gh pr view "$PR" --repo "$REPO" --json statusCheckRollup --jq '
       .statusCheckRollup[] | [
         (.name // .context // "?"),
         (if .__typename == "StatusContext" then
            (if (.state == "PENDING" or .state == "EXPECTED") then "IN_PROGRESS"
             else "COMPLETED" end)
          else (.status // "UNKNOWN") end),
         (if .__typename == "StatusContext" then
            (if .state == "SUCCESS" then "SUCCESS"
             elif (.state == "FAILURE" or .state == "ERROR") then "FAILURE"
             else "NEUTRAL" end)
          else (.conclusion // "UNKNOWN") end)
       ] | @tsv')

     # Sentinel: do NOT declare green until the CI aggregate gate has
     # reached a terminal state. Before /ok to test triggers the run,
     # the sentinel is absent; while CI is running, it's IN_PROGRESS.
     sentinel_line=$(printf '%s\n' "$rollup" | awk -F'\t' -v s="$SENTINEL" '$1 == s')
     sentinel_status=$(printf '%s\n' "$sentinel_line" | awk -F'\t' 'NR==1 {print $2}')
     if [ "$sentinel_status" != "COMPLETED" ]; then
       echo "=== $(date -u) waiting for $SENTINEL (status: ${sentinel_status:-absent}) ==="
       sleep 120
       continue
     fi

     # Classify non-exempt checks (exempt list applied to the NAME only).
     non_exempt=$(printf '%s\n' "$rollup" | awk -F'\t' -v p="^($EXEMPT)" 'tolower($1) !~ tolower(p)')
     failed=$(printf '%s\n' "$non_exempt" | awk -F'\t' '$2 == "COMPLETED" && $3 !~ /^(SUCCESS|SKIPPED|NEUTRAL)$/')
     pending=$(printf '%s\n' "$non_exempt" | awk -F'\t' '$2 != "COMPLETED"')

     if [ -n "$failed" ]; then
       echo "=== NON-EXEMPT FAILURES ==="
       printf '%s\n' "$failed"
       echo "RESULT=FAILURE"
       exit 0
     fi
     if [ -n "$pending" ]; then
       # Sentinel is COMPLETED but a non-exempt check is still pending —
       # rare but possible. Keep waiting; do NOT ship.
       echo "=== $(date -u) sentinel done but non-exempt checks still pending ==="
       printf '%s\n' "$pending"
       sleep 120
       continue
     fi

     echo "=== ALL NON-EXEMPT CHECKS COMPLETED GREEN ==="
     printf '%s\n' "$non_exempt"
     echo "RESULT=GREEN"
     exit 0
   done

This Bash call blocks for as long as CI takes (minutes to hours). Do

NOT split it into many short polls interleaved with other tool calls

— that wastes --max-turns and creates windows where you could lose

track of the loop state.

• Read the tool output:
• If RESULT=FAILURE: diagnose via

gh api repos/$REPO/actions/jobs/<JOB_ID>/logs (or the

external-context equivalent) and fix the code. The Phase 1

commit is immutable; fixes accumulate in a single rolling fix

commit on top of it:

     git add -A
     if git rev-parse --verify HEAD^2 >/dev/null 2>&1; then
       # HEAD has two parents → still the Phase 1 merge commit.
       # First failure of this run: create the fix commit.
       git commit -m "fix: post-CI corrections"
       git push origin "$BRANCH"
     else
       # HEAD is the existing fix commit → amend it.
       git commit --amend --no-edit
       git push --force-with-lease origin "$BRANCH"
     fi

--force-with-lease (not --force): if a human pushed onto the

branch since the bot last fetched, the lease aborts the push

instead of clobbering them — fetch and decide what to do.

Start a new outer-loop iteration at step 1 with the new HEAD SHA.

• If RESULT=GREEN: outer loop is done. Proceed to Phase 4.

Why not wait-for-run-to-register first? gh pr comment with

/ok to test <sha> is handled by copy-pr-bot, which takes a few

seconds to trigger the CI run. The statusCheckRollup poll in step 3

will initially show checks in PENDING / QUEUED; that's fine — the

inner loop treats those as "keep waiting" and will see them advance as

CI progresses. No separate registration poll needed.

Anti-Patterns (what went wrong on run 24800621116)

• **Do NOT classify a queued/in-progress job as "infrastructure-

blocked" and ship.** A stuck queue drains eventually — wait. If the

job eventually passes, great; if it fails, go fix it.

• **Do NOT mark ready while any required check is PENDING /

QUEUED / IN_PROGRESS on the HEAD SHA.** A push is not a pass;

only a COMPLETED + green status is.

• Do NOT declare an untested job "pre-existing." Pre-existing

means the test ran to completion and failed the same way on recent

dev CI. A job that never ran on your PR cannot be pre-existing.

• Do NOT use gh api .../actions/runs/.../jobs alone as the gate

signal. External status contexts (GitLab CI pipelines, copy-pr-bot

status, etc.) do NOT appear there. Use statusCheckRollup.

• Do NOT start any background process. No &, no nohup, no

run_in_background: true, no ScheduleWakeup. The GitHub Actions

step owns your shell; when the step ends, every background process

is killed and cannot resume.

• Do NOT push directly to pull-request/<PR_NUMBER> branches.

The community bot manages those branches when it processes

/ok to test. Pushing to them directly breaks the CI trigger

mechanism. Always push to your own sync branch (e.g.

main2dev/<DATE>) instead.

• **Do NOT forget the Run functional tests and Run MBridge tests

labels.** Without Run functional tests, the internal GitLab

functional tests do not run; without Run MBridge tests, the

MBridge test suite does not run.

Failure Investigation

• Fetch logs: gh api repos/$REPO/actions/jobs/<JOB_ID>/logs
• Grep for: ImportError, ModuleNotFoundError, FAILED,

would reformat, line-too-long, Traceback

• Read the error, understand root cause, fix the code

Common Issues

• ImportError for a class/module: Dev test imports a class from a file

where we took main's version. Restore only the missing class/function —

not the entire file. If a file's classes are completely different between

main and dev, keep both sets of code.

• Formatting failures (black/pylint): Run black --config pyproject.toml

on offending files. For pylint long-line or missing-docstring, edit directly.

• Circular imports: isort can reorder imports in a way that introduces

circular dependencies (e.g. megatron/legacy/model/__init__.py). Check

git diff on __init__.py files to see if import order changed.

• Dependency version mismatches: Taking main's pyproject.toml/uv.lock

can change library versions in the CI container. Dev-only code may depend on

newer versions (e.g. TransformerEngine's single_grouped_weight). If failures

trace to missing kwargs or changed APIs in third-party libs, this is the cause.

• API mismatch (AttributeError / TypeError at runtime): Main's callers

reference methods that don't exist (or have different signatures) in dev's

implementations. See "API Mismatch Detection" in Phase 1. Fix by adding

shims, renaming methods, or adjusting call signatures.

• Infrastructure / network failures (apt-get, pip download): Errors like

archive.ubuntu.com unreachable or Connection timed out during package

installation are transient CI infrastructure issues, not code problems.

Retry CI with the same SHA. Do not investigate as code failures.

Pre-Existing Failure Verification

You MUST empirically verify before classifying any failure as pre-existing.

• gh pr list --repo $REPO --base dev --state merged --limit 3
• gh pr checks <PR_NUMBER> --repo $REPO on a recently merged dev PR
• If the same test bucket passes on recent dev CI → the failure is

sync-caused. You must fix it.

• Only if the test also fails on recent dev CI can you classify it as

pre-existing. Document with the dev PR number and CI run as evidence.

Internal GitLab Functional Tests

GitHub CI covers unit tests and some integration tests. Internal GitLab

(gitlab-master.nvidia.com) runs additional functional tests on

H100/GB200 hardware that may reveal issues GitHub CI does not catch.

These surface in statusCheckRollup as external status contexts (the

bash template already handles them via the __typename == "StatusContext"

branch).

• Fine-grained activation offloading failures, for example, only showed

up in GitLab functional tests during PR #4291

• If GitHub CI passes but a reviewer reports GitLab failures,

investigate with the same rigor as GitHub CI failures

• The sync PR should ideally pass both GitHub and GitLab CI before

merge, but GitHub CI passing (i.e. the Phase 4 gate above) is the

minimum before gh pr ready

Phase 4: Mark PR Ready — Strict Gate

Run gh pr ready ONLY when every non-exempt required check on the latest

CI run (against the current HEAD SHA) satisfies BOTH:

• status == "completed" — NOT queued, in_progress, pending,

waiting, or requested.

• conclusion ∈ {"success", "skipped", "neutral"}.

If a non-exempt check is pending/queued/in-progress: keep polling; do not

run gh pr ready. If it fails: go back to Phase 3's loop.

The exempt list (approval/coverage/docs) is defined in Phase 3; only those

checks may be ignored.

A pre-existing failure (same test failing identically on recent dev CI)

may be accepted, but ONLY after it has fully run, been empirically

verified against dev, and documented in the PR body with evidence (dev PR

number + CI run URL).

gh pr ready <PR_NUMBER> --repo $REPO

Then comment on the PR confirming it is ready for human review. The

comment should include:

• Which non-exempt checks passed (summary from the bash template's

final ALL NON-EXEMPT CHECKS COMPLETED GREEN output)

• Any documented pre-existing failures with evidence (dev PR number +

CI run URL showing the same failure on recent dev CI)

• Which files were taken from main vs. merged manually
• Any API mismatches detected and fixed
• Any pyproject.toml git source reconciliation performed
• Links to the CI runs that validated the fixes

Rules

• Do not globally prioritize main over dev. Preserve recent dev-only additions

unless a specific main commit intentionally removed them. Main may win

wholesale only for the explicit override list above; otherwise resolve

conflicts by combining main's incoming changes with dev's still-unmerged

features.

• Two-commit policy: the PR contains at most two bot-authored

commits — the Phase 1 merge commit (immutable once pushed) and a

single rolling fix commit on top. The fix commit is created on

the first Phase 3 failure (normal push) and amended on every

subsequent failure (git commit --amend --no-edit +

git push --force-with-lease). Never modify the Phase 1 commit

after pushing it; never let the fix-commit count exceed one.

• CI triggers via comment: /ok to test <sha>
• CI runs appear on branch pull-request/<PR_NUMBER>
• Git committer identity: svcnvidia-nemo-ci
• After editing imports, run isort on those files
• Push directly to NVIDIA/Megatron-LM (not a fork). The bot uses a PAT

with write access. CLAUDE.md says "never push directly" but that rule is

for human contributors — the sync bot is an exception.