Trt Cpp Runtime Quickstart

TensorRT C++ Runtime Deploy

Load a serialized TensorRT engine from disk and run inference from C++ using only the modern Runtime API. Produces a minimal, copy-pasteable deploy harness that drops next to any .plan / .engine file and extends to production.

Reference samples to open before writing new code:

• quickstart/SemanticSegmentation/tutorial-runtime.cpp — cleanest minimal load-and-run example. Mirrors Steps 1–7 below.
• samples/sampleOnnxMNIST/sampleOnnxMNIST.cpp — end-to-end sample that also builds the engine; the runtime portion shows realistic I/O wiring.
• Public headers: include/NvInferRuntime.h — read IRuntime, ICudaEngine, IExecutionContext, IStreamReaderV2.

When to Use

| Situation | Use this skill? |

|-------------------------------------------------------------------------------------------|-----------------|

| You have a .plan/.engine and need to run it from a C++ binary | Yes |

| You need a minimal harness that uses enqueueV3 + setTensorAddress | Yes |

| You want to load an engine from a std::istream or large file via IStreamReaderV2 | Yes |

| You need to wire dynamic shapes (setInputShape) before inference | Yes |

| You are *building* / optimizing the engine (calibration, INT8, sparsity, builder configs) | No - use trtexec or IBuilder directly |

| You are deploying in Python | No - use tensorrt Python bindings |

| You are writing a plugin (IPluginV3) or custom layer | No - separate plugin skill |

| You need multi-GPU, MPS, MIG, or process-level orchestration | No - out of scope |

Prerequisites

• TensorRT installed. Verify NvInferRuntime.h is on the include path

and libnvinfer.so is on the link path. On a TRT dev container these are

in /usr/include/x86_64-linux-gnu/ and /usr/lib/x86_64-linux-gnu/ (or

/opt/tensorrt/... for tarball installs).

• CUDA toolkit available. cuda_runtime_api.h and libcudart.so must

be reachable; nvcc --version should match the CUDA version the engine

was built against.

• A serialized engine. A .plan/.engine file built **on the same

major TRT version and the same GPU architecture (compute capability) you

will deploy on**. Engines are not portable across major TRT versions or

across SMs unless the builder was given --hardwareCompatibilityLevel.

• The engine's I/O tensor names. Inspect with:

   trtexec --loadEngine=model.plan --verbose 2>&1 | grep -E 'Input|Output'

• A C++17 compiler (g++ >= 9 or clang++ >= 10).

Step 1: Create the IRuntime

The runtime owns engine deserialization and must outlive every

ICudaEngine it creates. Construct one per process for typical deployments.

class Logger : public nvinfer1::ILogger {
public:
    void log(Severity severity, char const* msg) noexcept override {
        if (severity <= Severity::kWARNING) {
            std::cerr << msg << std::endl;
        }
    }
};

Logger gLogger;
std::unique_ptr<nvinfer1::IRuntime> runtime{
    nvinfer1::createInferRuntime(gLogger)};
if (!runtime) throw std::runtime_error("createInferRuntime failed");

A custom logger is mandatory - TensorRT does not log internally. Keep it

process-global so deserialization warnings (version skew, calibrator

mismatch) are not lost.

Step 2: Read the Plan into Memory

For small/medium engines (< ~1 GiB) read the whole file into a

std::vector<char> and hand the pointer to

IRuntime::deserializeCudaEngine(blob, size). This is what the

SemanticSegmentation tutorial does and the simplest correct path:

std::ifstream f(planPath, std::ios::binary);
if (!f) throw std::runtime_error("cannot open " + planPath);
f.seekg(0, std::ios::end);
auto size = static_cast<size_t>(f.tellg());
f.seekg(0, std::ios::beg);
std::vector<char> blob(size);
if (!f.read(blob.data(), size))
    throw std::runtime_error("short read on " + planPath);

For very large engines, or when the bytes live behind a stream (HTTP,

mmap'd archive, encrypted store), implement an IStreamReaderV2 - see

Step 3.

Step 3 (optional): Use IStreamReaderV2 for Streaming Loads

IStreamReader (v1) is deprecated in TensorRT 11.0. Always use

IStreamReaderV2: it reads into both host and device memory and is the

only stream-reader form guaranteed for new code. Subclass and implement

read(...) and seek(...):

class FileStreamReader : public nvinfer1::IStreamReaderV2 {
public:
    explicit FileStreamReader(std::string const& path)
        : mFile(path, std::ios::binary) {
        if (!mFile) throw std::runtime_error("open failed: " + path);
    }
    int64_t read(void* dst, int64_t n,
                 cudaStream_t /*stream*/) noexcept override {
        mFile.read(static_cast<char*>(dst), n);
        return mFile.gcount();
    }
    bool seek(int64_t off, nvinfer1::SeekPosition where) noexcept override {
        auto dir = (where == nvinfer1::SeekPosition::kSET) ? std::ios::beg
                 : (where == nvinfer1::SeekPosition::kCUR) ? std::ios::cur
                 : std::ios::end;
        mFile.clear();
        mFile.seekg(off, dir);
        return static_cast<bool>(mFile);
    }
private:
    std::ifstream mFile;
};

FileStreamReader rd{planPath};
std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(rd)};

Step 4: Deserialize and Create an Execution Context

ICudaEngine is thread-safe for read-only queries; IExecutionContext

is not - allocate one per inference thread.

std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(blob.data(), blob.size())};
if (!engine) throw std::runtime_error("deserializeCudaEngine failed");

std::unique_ptr<nvinfer1::IExecutionContext> ctx{
    engine->createExecutionContext()};
if (!ctx) throw std::runtime_error("createExecutionContext failed");

Step 5: Wire Tensors with setTensorAddress

Enumerate I/O tensors via getNbIOTensors() + getIOTensorName(i). Use

getTensorIOMode, getTensorDataType, and getTensorShape to size and

allocate buffers. Set every tensor address before enqueueV3 - the

modern API has no implicit binding-index map.

for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto mode  = engine->getTensorIOMode(name);
    auto shape = engine->getTensorShape(name);   // -1 = dynamic dim
    if (mode == nvinfer1::TensorIOMode::kINPUT && hasDynamic(shape)) {
        // Fill in concrete shape, e.g. batch=1
        shape.d[0] = 1;
        ctx->setInputShape(name, shape);
    }
}
// After setInputShape on all dynamic inputs, query output shapes.
for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto bytes = elementCount(ctx->getTensorShape(name))
               * dtypeSize(engine->getTensorDataType(name));
    void* dev = nullptr;
    cudaMalloc(&dev, bytes);
    ctx->setTensorAddress(name, dev);
}

Always call setInputShape for dynamic inputs before querying output

shapes - the latter depends on the former.

Step 6: Run enqueueV3

enqueueV3(stream) is the only non-deprecated enqueue API;

enqueueV2/execute* are gone in modern flows.

cudaStream_t stream{};
cudaStreamCreate(&stream);

cudaMemcpyAsync(devInput, hostInput, inBytes,
                cudaMemcpyHostToDevice, stream);
if (!ctx->enqueueV3(stream))
    throw std::runtime_error("enqueueV3 failed");
cudaMemcpyAsync(hostOutput, devOutput, outBytes,
                cudaMemcpyDeviceToHost, stream);
cudaStreamSynchronize(stream);

If you reuse buffers across iterations, skip the per-call

setTensorAddress - addresses persist on the context until overwritten.

Step 7: Shutdown Order

Destroy in reverse construction order: contexts -> engines -> runtime,

then free CUDA memory and destroy the stream. With std::unique_ptr this

is automatic as long as the context is declared *after* the engine, and

the engine *after* the runtime. Free cudaMalloc allocations explicitly

(RAII wrapper recommended).

Build

Wire the steps above into your application's build system. For a standalone smoke test, a minimal build is:

g++ -std=c++17 runtime.cpp -o run -lnvinfer -lcudart   # adjust CUDA/TRT include + lib paths
./run model.plan

Common Errors

| Symptom | Likely cause |

|----------------------------------------------------------------------|------------------------------------------------------------------------------|

| deserializeCudaEngine returns nullptr, log says "version tag" | Engine built on a different TRT major version. Rebuild on the deploy version |

| nullptr with "engine plan file is generated on an incompatible device" | SM mismatch. Rebuild on the target SM or use --hardwareCompatibilityLevel |

| enqueueV3 returns false, log mentions "Tensor X has no address" | Forgot setTensorAddress for one of the I/O tensors |

| enqueueV3 false, "shape" in message | Forgot setInputShape for a dynamic input, or supplied an out-of-profile shape |

| cudaErrorIllegalAddress on H->D / D->H copy | Mismatched element count / dtype between host buffer and engine tensor |

| Process crashes inside TRT during destruction | Wrong destruction order - context outlived engine, or engine outlived runtime |

| cudaErrorMemoryAllocation during context creation | Workspace too big for the device; rebuild with smaller workspace |

Pitfalls

• Do not use IStreamReader v1. Deprecated in TRT 11.0. Use

IStreamReaderV2 (note cudaStream_t parameter on read).

• Do not use enqueueV2 / execute / binding indices. These are

legacy paths; the only stable modern path is name-based

setTensorAddress + enqueueV3.

• One IExecutionContext per thread. Sharing contexts across threads

is undefined behavior; sharing the engine is fine.

• Stream lifetime. The CUDA stream passed to enqueueV3 must outlive

the inference. Destroying it while work is in flight crashes or corrupts

output.

• Async vs sync copies. Mixing synchronous cudaMemcpy with

enqueueV3 on a stream serializes the GPU; always pair enqueueV3

with cudaMemcpyAsync on the same stream.

• Engine portability. A .plan is tied to (TRT major version, GPU SM,

CUDA major version). Never check engines into a repo without recording

these three facts.

• Logger lifetime. The logger passed to createInferRuntime must

outlive the runtime; a stack-local logger in main is fine, a function-

scope local is a use-after-free.

• Refit / weight streaming. Engines built with refit or weight

streaming enabled need extra setup calls (setWeightStreamingBudgetV2,

IRefitter); out of scope here.