Perf Torch Sync Free

Writing Sync-Free PyTorch Code

Sync-free code means the CPU continuously queues work to the GPU without

waiting for GPU operations to complete. When host-device synchronizations

are eliminated, the GPU works continuously without idle stalls.

Every host-device synchronization ultimately calls one of three CUDA driver

APIs that block the CPU thread:

• cuEventSynchronize -- CPU waits until a specific GPU event completes
• cuStreamSynchronize -- CPU waits until all work on a stream finishes
• cuCtxSynchronize -- CPU waits until all work across all streams finishes

When to Use

Reach for this skill when you encounter:

• Triggers: User wants to remove host-device synchronizations, eliminate

CPU stalls from GPU waits, make code async/sync-free, remove .item() or

.cpu() calls that block the CPU, or understand why specific PyTorch

operations cause synchronization

• Symptoms: Frequent cudaStreamSynchronize in nsys profiles,

warnings from torch.cuda.set_sync_debug_mode, training throughput

limited by CPU-GPU round-trips, .item() or .cpu() calls in hot loops

• Keywords: "sync-free", "synchronization", ".item()", ".cpu()",

"host-device sync", "eliminate syncs", "CPU stall", "non_blocking",

"set_sync_debug_mode", "cudaStreamSynchronize", "cudaEventSynchronize",

"remove syncs", "async GPU", "CPU waiting on GPU"

Do NOT use this skill for:

• Applying CUDA Graphs or reducing kernel launch overhead (use

perf-torch-cuda-graphs instead)

• Profiling GPU kernels, system timelines, or finding GPU idle time (use

perf-nsight-compute-analysis or perf-nsight-systems)

• Kernel optimization or code generation (use kernel-triton-writing)
• Optimizing NCCL communication or distributed training collective

operations

• Reducing GPU memory usage or gradient checkpointing
• General model compilation with torch.compile

Requirements

| Dependency | Version | Notes |

|------------|---------|-------|

| PyTorch | >=2.0 | With CUDA support |

| NVIDIA GPU | Any | CUDA-capable |

| Nsight Systems | Optional | For comprehensive sync detection via nsys |

Workflow

Step 1: Detect Synchronizations

Use one or both methods to find sync points in the code.

Quick detection -- PyTorch sync debug mode prints a warning with stack

trace on every synchronization:

import torch

# Enable at the start of the region you want to check
torch.cuda.set_sync_debug_mode('warn')   # prints warning + stack trace
# torch.cuda.set_sync_debug_mode('error')  # raises exception on sync

# Run your training step / forward pass here
train_step(model, batch)

torch.cuda.set_sync_debug_mode(0)  # disable

This mode only detects syncs going through PyTorch's wrapped

cuStreamSynchronize. Third-party libraries calling CUDA sync APIs

directly are not detected.

Comprehensive detection -- Nsight Systems captures all sync calls

including those from extensions and libraries:

nsys profile --capture-range=cudaProfilerApi \
             --python-sampling=true \
             --backtrace=dwarf \
             python your_script.py

In the Nsight Systems GUI, check the CUDA API timeline row and search

for cudaStreamSynchronize, cudaEventSynchronize, or

cudaDeviceSynchronize. The call stack panel shows which Python line

triggered each sync.

Step 2: Classify -- False vs True Dependencies

After detecting syncs, classify each one before deciding how to fix it.

False dependencies (avoidable) -- CPU does not actually need the GPU

result. These can be eliminated without changing program logic:

• Debug prints left in hot paths (print(loss.item()))
• Unnecessary .item() calls for logging that could be deferred
• Using .cuda() instead of .to('cuda', non_blocking=True)
• Using .type(torch.LongTensor) instead of .type(torch.long)
• Creating tensors from Python objects directly on CUDA

True dependencies (require restructuring) -- CPU genuinely needs the

GPU value to proceed:

• Control flow dependency: if loss.item() > threshold: -- CPU

branches on a GPU-computed value

• Dynamic memory allocation: output = x[mask] -- output size depends

on GPU computation

• CPU computation using GPU values: computing statistics for logging,

updating learning rates from metrics

True dependencies require restructuring: move logic to GPU

(torch.where()), delay to end of iteration, or accept that those parts

stay outside any CUDA Graph capture region.

Step 3: Eliminate Systematically

Apply fixes in order of increasing difficulty. Start with easy wins.

1. Remove redundancy -- Delete operations that do not need to happen:

• Remove debug prints and logging from hot loops
• Delete unnecessary .item() calls
• Eliminate duplicate synchronizations

2. Use non_blocking=True -- Make transfers async where CPU does not

immediately use the result:

# Before (syncs)
x_gpu = x_cpu.cuda()
x_cpu = x_gpu.cpu()

# After (async, no sync)
x_gpu = x_cpu.to('cuda', non_blocking=True)
x_cpu = x_gpu.to('cpu', non_blocking=True)   # only if CPU does not use x_cpu immediately

Only use non_blocking=True for GPU-to-CPU when the CPU does not

immediately read the result. Otherwise the CPU may operate on incomplete

data.

3. Switch to sync-free API alternatives -- See the Quick Reference

Table below for a condensed mapping of common patterns.

4. Delay synchronization to end of iteration -- Move logging and

validation to after the optimizer step rather than mid-forward/backward:

# Before: sync mid-iteration
loss = model(batch)
print(f"Loss: {loss.item()}")    # cuStreamSynchronize
loss.backward()

# After: delay to end of iteration
loss = model(batch)
loss.backward()
optimizer.step()
print(f"Loss: {loss.item()}")    # sync is outside the hot path

5. Coalesce multiple syncs into one -- If you need several GPU values

on CPU, gather them and transfer once:

# Before: 3 separate syncs
loss_val = loss.item()           # cuStreamSynchronize
acc_val = accuracy.item()        # cuStreamSynchronize
gnorm_val = grad_norm.item()     # cuStreamSynchronize

# After: 1 sync
metrics = torch.stack([loss, accuracy, grad_norm])
vals = metrics.cpu()             # single cuStreamSynchronize
loss_val, acc_val, gnorm_val = vals.tolist()

6. Offload logic to GPU -- Replace CPU-side logic with GPU-native ops:

# Before: CPU control flow (syncs)
if loss.item() > threshold:
    result = a
else:
    result = b

# After: GPU-side selection (no sync)
result = torch.where(loss > threshold, a, b)

# Before: Python max (syncs)
val = max(x_gpu[0, 0], x_gpu[0, 1])

# After: torch.max (no sync)
val = torch.max(x_gpu[0, 0], x_gpu[0, 1])

7. Exclude unavoidable syncs from capture range (last resort) -- If a

sync cannot be eliminated, keep it outside the CUDA Graph capture region

and graph only the sync-free sections. Partial graphing is better than no

graphing.

Step 4: Verify

Re-run detection to confirm syncs are eliminated:

torch.cuda.set_sync_debug_mode('error')  # will raise if any sync remains
train_step(model, batch)
torch.cuda.set_sync_debug_mode(0)

Or re-profile with Nsight Systems and confirm no cudaStreamSynchronize /

cudaEventSynchronize / cudaDeviceSynchronize calls appear in the

target region.

Quick Reference Table

| Sync-Inducing Pattern | Sync-Free Alternative |

|----------------------|----------------------|

| Device Transfers | |

| .cpu() or .to('cpu') | .to('cpu', non_blocking=True) (fire-and-forget only) |

| .cuda() or .to('cuda') | .to('cuda', non_blocking=True) |

| .type(torch.LongTensor) | .type(torch.long) (dtype conversion, stays on GPU) |

| Tensor Creation | |

| torch.tensor(obj, device='cuda') | Create on CPU, then .to('cuda', non_blocking=True) |

| torch.tensor(0, device='cuda') | torch.zeros(1, device='cuda', dtype=...).squeeze() |

| torch.as_tensor(arr, device='cuda') | Create on CPU, then .to('cuda', non_blocking=True) |

| torch.cuda.BoolTensor(list) | torch.tensor(list, device='cpu').to('cuda', non_blocking=True) |

| Control Flow | |

| .item() in conditionals | torch.where() or move outside critical region |

| if gpu_tensor: | Keep logic on GPU with torch.where() |

| Python max(a, b) on GPU tensors | torch.max(a, b) |

| torch.is_nonzero(t) | Avoid; use GPU-side comparisons |

| Indexing | |

| x_gpu[idx_cpu] or x_gpu[idx_list] | x_gpu[idx_gpu] (keep indices on same device) |

| x_gpu[idx] = 0 (scalar assignment) | x_gpu[idx] = zero_gpu (GPU tensor value) |

| x[i:j] with CUDA tensor bounds | x[:, s] with s = torch.arange(i, j, device='cuda') |

| Dynamic Shapes | |

| x_gpu[mask_gpu] (masked selection) | torch.where(mask_gpu, x_gpu, 0) (fixed shape) |

| torch.nonzero(mask) | torch.where() or move outside critical region |

| torch.masked_select(x, mask) | torch.where(mask, x, 0) |

| torch.unique(x) | Avoid in hot path; precompute if possible |

| torch.repeat_interleave(x, r) | Specify output_size=N if known |

Finding More Information

• Tier 1 (this file): Workflow, classification, elimination strategies,

and quick reference table

• Tier 2 (references/sync-patterns.md): Comprehensive pattern catalog

with 9 categories, full code examples showing sync-inducing and sync-free

versions, and the specific CUDA driver API triggered by each pattern