Ncu Report

Nsight Compute Report Analysis

This skill handles two modes:

• Analyze an existing report: The user provides a path to an .ncu-rep file (or one exists under examples/). Use the ncu CLI to extract and present metrics.

• Generate a new report: The user specifies a script or kernel to profile but does NOT provide a .ncu-rep file. In this case, set up profiling using tilus.utils.ncu_utils.ncu_run(), run it, then analyze the resulting report.

Generating a Report

Tilus provides tilus.utils.ncu_utils.ncu_run() to profile kernels with full metrics and source correlation.

API

from tilus.utils.ncu_utils import ncu_run

# ncu_run(func, *args, kernel_regex=".*", **kwargs) -> NsightComputeReport
report = ncu_run(main, bench=False, kernel_regex="tilus|nvjet")

• func: a callable (typically a main() function) that runs the kernel(s) to profile
• *args, **kwargs: passed through to func
• kernel_regex: regex to filter which kernels to profile (default ".*")
• Returns NsightComputeReport with .report_path pointing to the generated .ncu-rep file

What it does

• Runs the function under ncu with --set full, all rules enabled, and --import-source yes
• Saves the report to ncu-reports/reportN.ncu-rep next to the script (auto-increments N)
• Uses the system Python and the ncu binary at /usr/local/cuda/bin/ncu

How to generate a report for the user

IMPORTANT: ncu_run() must be called inside a if __name__ == "__main__": block. It works by re-importing the script as a subprocess under ncu — if ncu_run() is at module level, the subprocess will call ncu_run() again, causing infinite recursion and a runtime error.

Step 1: Read the script — find the example script the user specified and read it to understand how the kernel is invoked.

Step 2: Set up profiling — choose one of these approaches:

• If the script already has a __main__ block with ncu_run(): just run it directly.
• If the script has a __main__ block but no ncu_run(): edit the __main__ block to add ncu_run(). For example, if the block calls main(), change it to call ncu_run(main, bench=False, kernel_regex="tilus").
• If the script has no __main__ block or is hard to modify (e.g., it's a test file, or the kernel launch is deeply nested): create a new script next to it (e.g., profile_<name>.py) that imports and calls the kernel under ncu_run().

Example of editing an existing __main__ block:

if __name__ == "__main__":
    from tilus.utils.ncu_utils import ncu_run
    ncu_run(main, bench=False, kernel_regex="tilus")

Example of creating a new profiling script:

from tilus.utils.ncu_utils import ncu_run
from matmul_v9 import main

if __name__ == "__main__":
    ncu_run(main, bench=False, kernel_regex="tilus")

Step 3: Run the script — python <script_path>. The report will be saved to <script_dir>/ncu-reports/reportN.ncu-rep.

Step 4: Analyze — proceed to the Analysis Workflow below with the generated report.

Note: ncu profiling requires sudo or appropriate permissions (CAP_SYS_ADMIN). If the command fails with permission errors, suggest running with sudo.

Analysis Workflow

Follow this sequence. Skip steps the user doesn't need, but always start with Step 1.

Step 1: Overview — List kernels and session info

Run these in parallel:

# List all kernels with timing
ncu -i <REPORT> --page raw --csv --metrics gpu__time_duration.sum 2>&1

# Session/device info
ncu -i <REPORT> --page session --csv 2>&1

Present a summary table:

• Kernel name (shortened), Block Size, Grid Size, Duration (ms)
• Device name, compute capability, CUDA version

Step 2: Speed of Light — Top-level throughput

ncu -i <REPORT> --page details --csv --section SpeedOfLight 2>&1

Key metrics to highlight per kernel:

• Duration (ms)
• Compute (SM) Throughput (%) — how busy the SMs are
• Memory Throughput (%) — overall memory utilization
• DRAM Throughput (%) — HBM bandwidth utilization
• L1/TEX Cache Throughput (%)
• L2 Cache Throughput (%)
• SOLBottleneck rule — check the Rule Description column for bottleneck guidance

Step 3: Compute & Memory Workload Analysis

# Compute workload
ncu -i <REPORT> --page details --csv --section ComputeWorkloadAnalysis 2>&1

# Memory workload
ncu -i <REPORT> --page details --csv --section MemoryWorkloadAnalysis 2>&1

Key compute metrics: Executed IPC Active, SM Busy %, Issue Slots Busy %

Key memory metrics: Mem Busy %, Max Bandwidth %, L1/L2 hit rates

Step 4: Occupancy

ncu -i <REPORT> --page details --csv --section Occupancy 2>&1

Report: Theoretical Occupancy, Achieved Occupancy, and limiters (registers, shared memory, block size).

Step 5: Detailed metrics (on demand)

To extract specific raw metrics:

ncu -i <REPORT> --page raw --csv --metrics <metric1>,<metric2>,... 2>&1

To filter by kernel:

ncu -i <REPORT> --page raw --csv --metrics <metrics> --kernel-name regex:<pattern> 2>&1

Step 6: Source-level analysis (on demand)

SASS-only (default, always available):

ncu -i <REPORT> --page source --csv --kernel-name regex:<pattern> 2>&1

CUDA source correlated with SASS (requires --import-source yes during profiling):

ncu -i <REPORT> --page source --csv --print-source cuda,sass --kernel-name regex:<pattern> 2>&1

Source output columns include per-instruction: Warp Stall Sampling, Instructions Executed, Thread Instructions Executed, stall reasons (stall_barrier, stall_math, stall_wait, etc.), shared memory conflicts, and more.

Step 7: Rules / automated analysis (on demand)

Rules are included in the details page output. Look for non-empty "Rule Name" column entries.

ncu -i <REPORT> --page details --csv --print-rule-details 2>&1 | grep -v '^"[0-9]' | head -5  # header

To see all rule results with descriptions:

ncu -i <REPORT> --page details --csv --print-rule-details 2>&1

Filter for rows where column 17 (Rule Name) is non-empty.

Reference: ncu CLI Options for Report Analysis

Pages (--page)

| Page | Description |

|------|-------------|

| details | Sections with metrics organized by section name + rule results |

| raw | All collected metrics as flat columns (one row per kernel) |

| source | Per-instruction source code with correlated metrics |

| session | Session info, device attributes, launch settings |

Key Flags

| Flag | Description |

|------|-------------|

| --csv | Output as CSV (essential for parsing) |

| --metrics <m1>,<m2> | Filter specific metrics (for raw page) |

| --section <id> | Filter by section identifier (for details page) |

| --kernel-name regex:<pat> | Filter kernels by name regex |

| --kernel-name <exact> | Filter by exact kernel name |

| --print-source sass\|ptx\|cuda\|cuda,sass | Select source view for source page |

| --print-details header\|body\|all | Control detail level: header (default), body (charts/tables), all |

| --print-metric-name name | Show internal metric names instead of display labels |

| --print-metric-name label-name | Show both label and internal name |

| --print-units base | Show metrics in base units (no auto-scaling) |

| --print-summary per-kernel | Aggregate across invocations per kernel (min/max/avg) |

| --print-rule-details | Include additional rule tables and KPI metrics |

Section Identifiers

| Identifier | Display Name |

|------------|-------------|

| SpeedOfLight | GPU Speed Of Light Throughput |

| ComputeWorkloadAnalysis | Compute Workload Analysis |

| MemoryWorkloadAnalysis | Memory Workload Analysis |

| MemoryWorkloadAnalysis_Tables | Memory Workload Analysis Tables |

| Occupancy | Occupancy |

| LaunchStats | Launch Statistics |

| SchedulerStats | Scheduler Statistics |

| WarpStateStats | Warp State Statistics |

| InstructionStats | Instruction Statistics |

| SourceCounters | Source Counters |

| WorkloadDistribution | GPU and Memory Workload Distribution |

| NumaAffinity | NUMA Affinity |

| SpeedOfLight_RooflineChart | GPU Speed Of Light Roofline Chart |

| SpeedOfLight_HierarchicalTensorRooflineChart | Roofline Chart (Tensor Core) |

| SpeedOfLight_HierarchicalHalfRooflineChart | Roofline Chart (Half Precision) |

Section Sets (used during profiling with --set)

| Set | Sections | Est. Metrics |

|-----|----------|-------------|

| basic | LaunchStats, Occupancy, SpeedOfLight, WorkloadDistribution | 213 |

| detailed | basic + ComputeWorkloadAnalysis, MemoryWorkloadAnalysis, SourceCounters, Roofline | 906 |

| full | All sections including Instruction/Scheduler/WarpState stats, all Rooflines | 7794 |

Commonly Used Raw Metrics

| Metric | Description |

|--------|-------------|

| gpu__time_duration.sum | Kernel wall-clock duration |

| sm__throughput.avg.pct_of_peak_sustained_elapsed | SM throughput % |

| dram__throughput.avg.pct_of_peak_sustained_elapsed | DRAM throughput % |

| sm__warps_active.avg.pct_of_peak_sustained_active | Active warps % |

| launch__occupancy_limit_registers | Occupancy limiter: registers |

| launch__occupancy_limit_shared_mem | Occupancy limiter: shared memory |

| launch__occupancy_limit_blocks | Occupancy limiter: blocks |

| launch__occupancy_limit_warps | Occupancy limiter: warps |

| sm__sass_thread_inst_executed_op_* | Per-opcode instruction counts |

| l1tex__t_sector_hit_rate.pct | L1 cache hit rate |

| lts__t_sector_hit_rate.pct | L2 cache hit rate |

Available Rules (used during profiling with --rule)

| Rule ID | Description |

|---------|-------------|

| SOLBottleneck | High-level bottleneck detection |

| SOLFPRoofline | Floating Point Roofline Analysis |

| CPIStall | Warp stall analysis |

| Occupancy | Achieved Occupancy analysis |

| LaunchConfiguration | Kernel launch config analysis |

| HighPipeUtilization | High pipe utilization bottleneck |

| IssueSlotUtilization | Scheduler issue analysis |

| SharedMemoryConflicts | Shared memory bank conflicts |

| ThreadDivergence | Warp/thread divergence |

| UncoalescedGlobalAccess | Uncoalesced global memory |

| UncoalescedSharedAccess | Uncoalesced shared memory |

| SlowPipeLimiter | Slow pipe limiting compute |

| FPInstructions | FP instruction analysis |

| PCSamplingData | PC sampling data |

Comparing Kernels

When the report contains multiple kernels (e.g., a reference nvjet kernel and a tilus kernel), always present metrics side-by-side for comparison. Highlight:

• Duration difference (which is faster, by how much)
• Throughput differences (compute vs memory bound)
• Occupancy differences
• Any rule findings that differ

Tips

• The raw page has one row per kernel with all metrics as columns — good for extracting specific values.
• The details page organizes metrics by section — good for browsing all metrics in a section.
• The source page is per-instruction — good for hotspot analysis. Output can be very large; pipe through head or filter with grep.
• Use --print-units base with --csv for consistent numeric parsing.
• Use --print-metric-name name to get programmatic metric names instead of human labels.
• Source analysis with cuda,sass view shows CUDA source lines interleaved with their SASS instructions — extremely useful for correlating high-level code with assembly hotspots.