Cuda Cutlass Fmha Incremental Rebuild Agent Skill

Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels

> The general false-green principles (stale binary, wrong-artifact mtime) are summarised

> in the ort-test skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific

> detail.

The gotcha (verification-integrity bug)

nvcc-generated depfiles do not track the CUTLASS fused-MHA headers under

onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/ (e.g. kernel_forward.h,

fmha_launch_template.h). These headers are #included by the fmha_sm*.cu

translation units, but the build system does not record that dependency.

Consequence: after you edit one of those headers, an incremental build.sh:

• does not recompile fmha_sm*.cu,
• reports [100%] Built target ... and exits 0,
• leaves the recompiled artifacts — the fmha_sm*.cu.o objects and the

libonnxruntime_providers_cuda.so they link into — unchanged (same mtime as

the pre-edit build).

(Do not use the gtest test-exe mtime as the stale symptom: in the shared-provider

build the exe dlopens the .so and is not relinked, so its mtime stays old even

after a *correct* rebuild — see "How to confirm" below. The reliable diagnostic signal

is the fmha_sm*.cu.o / .so mtime.)

So your "successful" rebuild is running the old kernel. Tests that should now

pass (or fail) reflect the previous code, not your edit. This silently invalidates

any FAIL→PASS / PASS→FAIL verification.

The fix — force recompile the .cu units

Before rebuilding after editing any cutlass_fmha/*.h header:

touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu

Then run the normal build command. This forces the fmha_sm*.cu translation units

(and downstream binaries) to recompile against your header change.

How to confirm the rebuild was real (don't trust "[100%] Built")

Confirm that the artifact which actually links the recompiled fmha_sm*.cu.o

is newer than your header edit.

⚠️ **Do NOT just check the test EXE mtime — it can falsely flag a good build as

stale.** In the shared-provider build configuration (the default here), the CUDA

execution provider is a shared module: the recompiled fmha_sm*.cu.o link into

libonnxruntime_providers_cuda.so, and the onnxruntime_provider_test executable

dlopens that .so — it is not relinked. So after a *correct* rebuild the

test exe mtime stays old while the .so advances. Checking the exe alone

would wrongly conclude the build was stale.

Check the right artifact for your link mode:

• Shared-provider build (default): the .so that links the recompiled .o —

build/<dir>/<cfg>/libonnxruntime_providers_cuda.so

• Statically-linked provider: the test exe itself (onnxruntime_provider_test)

Safest check — stat both the recompiled object and the .so, and confirm BOTH

are newer than the header edit:

stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
# in your build dir, e.g. build/Debug_quickbuild/Debug/:
stat -c '%y %n' libonnxruntime_providers_cuda.so
# and the actual recompiled object (path varies by build dir):
find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +

If the .so (and the fmha_sm*.cu.o) timestamps are older than (or equal to) the

header edit, the build was stale — touch the .cu files and rebuild. The most

reliable signal of all is behavioral: a test that was failing now passes (a stale

binary cannot flip its result).

Related: pick the right test binary

This is the CUDA/CUTLASS instance of false-green mode 1 (zero-match / wrong binary) —

see the ort-test skill's "False-green taxonomy" for the general principle. In short:

attention/MEA/Flash boundary gtests (e.g. FlashStructuralEmptyRows*,

Attention_Causal_NonPadKVSeqLen_MEA_*) live in onnxruntime_provider_test, which CI

runs; onnxruntime_test_all does not contain them and gives a false green. Verify the

MEA/Flash boundary fix against onnxruntime_provider_test.

Related: disk frugality on shared GPU dev boxes

Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach

tens of GB). On a shared box, /home filling to 100% makes builds fail in

non-obvious places — e.g. git submodule sync reporting No space left on device

or a config.lock error, not an obvious "disk full" at the compile step.

Before a big rebuild, check free space and clean only clearly-stale, regenerable

build directories (old dated experiment dirs). Never delete another agent's active

build dir or anything ambiguous:

df -h /home
du -sh build/* | sort -h