Doca Gpunetio Ib Write Lat Agent Skill

DOCA GPUNetIO ib_write_lat

Where to start: This is a tool skill for the GPUNetIO-

flavored ib_write_lat benchmark shipped under

doca/tools/gpunetio_ib_write_lat/ (a client + server pair,

built from source against the installed DOCA via meson).

It measures the latency of an RDMA WRITE work request when

the WR is posted **from a CUDA kernel through the

doca-gpunetio device-side surface**, in a ping-pong cadence.

Open TASKS.md and start at

## configure for the GPU-NIC pairing

precondition and the build pattern; jump to

## run for the single-iteration smoke

flow. Open CAPABILITIES.md when the

question is *what this tool actually measures*, *how it

differs from the GPI sister tool on the same physical

operation*, or *how to interpret the half-iter / full-iter

/ CUDA-side usec output and the median / p99 / jitter

characterization*. If DOCA is not installed yet, route to

doca-setup first; if the

user is still deciding between GPUNetIO and GPI as a

programming surface, the picture in

../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes

and

../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes

is the first stop.

Example questions this skill answers well

The CLASSES of doca-gpunetio-ib-write-lat questions this

skill is built to answer, each with one worked example. The

class is the load-bearing piece; the worked example is one

instance.

• **"What GPU-init RDMA-WRITE latency / jitter can the

GPUNetIO path deliver for a real-time / control-loop

workload?"** — worked example: *"measure per-iteration

WRITE latency between two hosts with an H100 +

ConnectX-7 on each side, target the median and the p99

separately"*. Answered by the GPU-NIC pairing

precondition in

CAPABILITIES.md ## Capabilities and modes

+ the bring-up flow in

TASKS.md ## configure +

TASKS.md ## run.

• **"This is the GPUNetIO tool — how does the latency

number differ from the GPI programming surface?"** —

worked example: *"the team is using GPI; should I expect

GPUNetIO to beat / tie / lose vs GPI?"*. Answered by the

*"same physical operation, different runtime framework"*

rule in

CAPABILITIES.md ## Capabilities and modes

+ the cross-link to the GPI library skill

../../libs/doca-gpi/CAPABILITIES.md

(note: doca/tools/ ships no GPI ib_write_lat

benchmark binary — GPI is a programming surface, not a

shipped benchmark tool).

• **"Median vs p99 vs jitter — which one is the actual

answer for a real-time control loop?"** — worked

example: *"my control loop has a deadline; the median

is well under the budget but p99 spikes; do I quote

the median or the p99?"*. Answered by the

median-vs-p99-vs-jitter rule in

CAPABILITIES.md ## Observability

+ the eval-loop overlay in

TASKS.md ## test.

• **"What is the latency-vs-batching trade-off specific

to GPU-init RDMA?"** — worked example: *"my CUDA kernel

could batch multiple WRs to amortize the GPU-side

overhead; what does that buy me on latency vs what does

it cost me?"*. Answered by the

latency-vs-batching trade-off in

CAPABILITIES.md ## Capabilities and modes.

• **"What version of DOCA + CUDA Toolkit do I need for

this binary to build and run?"** — worked example: *"my

install has DOCA at one semver and CUDA at another;

will the ToT-shipped gpunetio_ib_write_lat even

link?"*. Answered by the version overlay in

CAPABILITIES.md ## Version compatibility.

• **"How do I read the half-iter / full-iter / CUDA-side

usec columns?"** — worked example: *"the binary printed

half-iter, full-iter, and a CUDA-side number — what is

the right column to quote for one-way latency vs

round-trip vs cross-check?"*. Answered by the column-

semantics rule in

CAPABILITIES.md ## Observability.

Audience

This skill serves **external developers and performance

engineers who need a reproducible measurement of the

latency of an RDMA WRITE WR when the WR is posted from a

CUDA kernel through doca-gpunetio**, on the user's actual

install and GPU-NIC pair. Concretely:

• A developer designing a GPU-resident real-time control

loop and deciding whether the GPUNetIO path's tail

latency fits the deadline.

• A platform operator validating a tuning change (NUMA

pinning, GPU PCIe placement, IB device choice, GID

index, NIC firmware burn) by re-running this benchmark

against the new state.

• An SRE / performance engineer producing a *"this is the

GPUNetIO-driven WRITE latency on this GPU-NIC pair

today, with median + p99 + jitter"* artifact downstream

consumers can cite.

• An AI agent answering *"is the doca-gpunetio latency

budget acceptable for this real-time workload class"*

honestly — with measured numbers, the build +

invocation that produced them, and the GPU + NIC +

DOCA version that scopes them — rather than guessing.

It is not for users debugging the doca-gpunetio

library itself (route to

../../libs/doca-gpunetio/SKILL.md),

and not a substitute for the perftest upstream

ib_write_lat (which measures CPU-initiated WRITE

latency).

Language scope

The doca-gpunetio-ib-write-lat tool is shipped as **C

plus CUDA .cu translation units** under

doca/tools/gpunetio_ib_write_lat/, split into a

client/ subtree, a server/ subtree, and a common/

subtree shared between them (per the verified file layout:

client/{main.c,perftest.{c,h},meson.build},

server/{main.c,perftest.{c,h},meson.build},

common/{common.c,common.h,kernel.cu}). The host-side

build is meson against the installed DOCA pkg-config

modules (doca-gpunetio, doca-rdma, doca-common,

plus the CUDA Toolkit dependency); the device-side build

is nvcc against the DOCA GPU NetIO device-side header

set. There is no Python / Rust / Go binding — the tool is

a pair of CLI binaries.

When to load this skill

Load this skill when the user is — or the agent needs to

— build and run the gpunetio_ib_write_lat client +

server on real hosts with DOCA installed plus a CUDA

Toolkit matched to the DOCA install, and a GPU + IB device

pair on each host's PCIe topology. Concretely:

• Measuring kernel-initiated RDMA WRITE latency between

two hosts (or a host and a BlueField DPU) with the

GPUNetIO surface.

• Characterizing tail latency (p99 / p99.9) and jitter

for a real-time / control-loop workload class.

• Deciding whether the GPUNetIO path is the right runtime

surface for a class of workload vs the GPI programming

surface (the doca-gpi

library — doca/tools/ ships no GPI benchmark binary)

or the classic CPU-initiated perftest path.

• Capturing a documented baseline (build + invocation +

DOCA version + GPU + NIC + as-deployed environment +

numbers) for later regression hunts.

• Diagnosing a build / link / run failure that surfaces

the GPUNetIO + RDMA bring-up sequence under this tool's

shipped scaffolding.

Do not load this skill for general DOCA orientation,

library API work, or installation. For those, use

doca-public-knowledge-map,

../../libs/doca-gpunetio/SKILL.md,

or doca-setup. Do not load

it for *application-level* real-time deadline analysis —

this benchmark measures the WR latency through GPUNetIO,

not the user's full pipeline.

What this skill provides

This is a thin loader. Substantive material lives in

two companion files:

• CAPABILITIES.md — what the tool measures (the

ping-pong WRITE latency primitive driven by both sides'

CUDA kernels through doca-gpunetio), the

runtime-surface selection rule (GPUNetIO vs GPI vs

CPU-initiated), the GPU-NIC pairing precondition, the

latency-vs-batching trade-off intrinsic to GPU-init

RDMA, the median / p99 / jitter reporting taxonomy,

the version overlay (DOCA .pc PLUS CUDA Toolkit),

the layered error taxonomy, the observability surface

(stdout report including the timeout knob the

gpunetio_rdma_write_lat_* kernel functions surface

per the verified common.h), and the safety overlay.

• TASKS.md — step-by-step workflows for the in-scope

task verbs: install, configure, build, modify,

run (smoke-before-bulk; single-iteration verification;

reading the report columns), test (the eval loop —

median / p99 / jitter / steady-state), debug (walk

the error taxonomy layer by layer), use (how a

latency result feeds a real-time class-of-workload

decision), plus a Deferred task verbs block.

The skill assumes a host where DOCA is already installed,

a CUDA Toolkit matched to the install is present, and the

operator has whatever privileges the public install

profile expects for binding a doca_dev, a doca_gpu,

and an OOB TCP socket.

What this skill deliberately does not ship

This skill is agent guidance, not a samples or

scripts bundle. It deliberately does not contain — and

pull requests should not add:

• Specific flag strings or expected latency numbers

beyond what the tool's shipped --help and main.c

ARGP registration establish. The flag surface is small

(device name, GPU PCIe address, GID index, server IP

on the client side); the agent re-reads the binary's

--help on the installed version.

• **Pre-written DOCA GPUNetIO or CUDA kernel source

code** that would compete with the shipped tool tree.

The shipped client/, server/, and common/

subtrees are the verified worked example.

• Wrappers, parsers, or scripts in any language that

consume the tool's stdout. The output format is small

and documented in

CAPABILITIES.md ## Observability.

• A samples/, bindings/, or reference/ subtree.

This is a thin loader for a shipped tool tree.

Loading order

• Read this SKILL.md first to confirm the user's

question is in scope (the user actually wants to

measure kernel-initiated WRITE latency through

GPUNetIO, not the GPI variant, not the CPU-initiated

variant, and not a library API question).

• **For what the tool measures, the surface-selection

rule against the GPI sister tool and the CPU-initiated

perftest, the latency-vs-batching trade-off, the

median / p99 / jitter reporting taxonomy, the version

overlay, the error taxonomy, the observability

surface, and the safety overlay, see

CAPABILITIES.md.**

• **For step-by-step workflows — install,

configure, build, modify, run, test,

debug, use — see TASKS.md.**

Related skills

• ../../libs/doca-gpunetio/SKILL.md —

the library this tool wraps. The per-GPU doca_gpu

context, the GPU-visible RDMA handles, the CUDA-side

persistent-kernel pattern, the dual capability-

discovery rule (DOCA cap-query AND

cudaGetDeviceProperties), and the env preconditions

(nvidia_peermem loaded, CUDA buffers registered

with DOCA) live there.

• ../../libs/doca-rdma/SKILL.md —

the underlying RDMA library. The RDMA queue this tool

binds is created and connected via doca-rdma; the

queue lifecycle, the transport type (RC vs UC vs UD),

the permission matrix, and the connection method are

owned there.

• ../../libs/doca-verbs/SKILL.md —

the raw-verbs escape hatch beneath doca-rdma /

doca-gpunetio. This tool stays on the higher-level

surfaces.

• ../doca-gpunetio-ib-write-bw/SKILL.md —

bandwidth analog of this tool on the same runtime

framework. Same physical operation; different metric

class (latency vs BW). The two together carry the full

GPUNetIO-side latency / throughput picture.

• doca-gpi — the GPI

programming surface (CUDA-kernel-initiated RDMA). The

alternative runtime framework for the same physical

operation; doca/tools/ ships no GPI ib_write_lat

benchmark binary, so the GPI comparison is against the

library surface, not a sibling tool. The selection rule

in

CAPABILITIES.md ## Capabilities and modes

is the decision aid; the agent's job is to teach when

to pick which.

• doca-version — the

canonical version-detection chain, four-way match

rule. The ## Version compatibility section here is a

thin overlay.

• doca-setup — env

preparation, install verification, GPU + CUDA Toolkit

pairing, nvidia_peermem load, hugepages, NUMA, and

the NGC DOCA container path.

• doca-public-knowledge-map —

routing to the public DOCA documentation set and the

CUDA Toolkit pointer.

• doca-debug — the

cross-cutting debug ladder.

• doca-hardware-safety —

the bundle-wide hardware-safety meta-policy.