Trt Strong Typing Migration

Migrate Weakly-Typed to Strongly-Typed TensorRT Build

Strong typing is the recommended build mode for TensorRT 11.x; weak typing was

deprecated in 10.12 and is scheduled for removal. This skill migrates a weakly-typed

build flow to a strongly-typed equivalent, one path per entry point (Python network,

trtexec, C++ builder), plus the ModelOpt AutoCast pre-step needed when the source

is an ONNX file that must run in mixed precision.

Version behavior. On TensorRT 11.x strong typing is unconditional — every

network is strongly typed, the kSTRONGLY_TYPED creation flag is accepted but

ignored, and trtexec --stronglyTyped is a no-op (default-on). On 10.12–10.x

you opt in explicitly via that flag / --stronglyTyped. Setting it is therefore

always safe: required on 10.x, harmless on 11.x. Either way the migration is the

same — remove the precision *hints* (gone in 11.x) and move precision into the graph.

When to Use

| Scenario | Use this skill? |

|----------|-----------------|

| Existing Python build calls config.set_flag(trt.BuilderFlag.FP16 / INT8 / TF32) | Yes |

| Existing C++ build calls config->setFlag(BuilderFlag::kFP16 / kINT8 / kTF32) | Yes |

| Existing trtexec line uses --fp16, --int8, --best, --bf16 and the developer wants strong typing | Yes |

| Build emits "weak typing is deprecated, use STRONGLY_TYPED" warning at runtime | Yes |

| Build fails with "flag X is incompatible with strongly typed networks" | Yes |

| Developer has a FP32 ONNX file and wants a strongly-typed mixed FP16 engine | Yes — start at Step 1 (AutoCast) |

| Developer is importing a brand-new ONNX file with no existing build flow | No — use trt-onnx-quickstart first; come back here to switch the flag |

| Developer's engine produces wrong outputs *after* migration | No — use trt-quantization-accuracy |

| Developer is on TensorRT 10.11 or earlier | No — strong typing is fully available from 10.12 onward; upgrade before migrating |

Prerequisites

• TensorRT 10.12 or later (11.x recommended). Check:

   python3 -c "import tensorrt; print(tensorrt.__version__)"
   trtexec --help 2>&1 | grep -i stronglyTyped

The version check is the determinant — strong typing is available from 10.12

on. The trtexec probe just confirms the CLI flag for the trtexec path; a

missing --stronglyTyped means a pre-10.12 install.

• ModelOpt installed when the source is an ONNX file requiring mixed precision.

   pip install nvidia-modelopt onnx onnxruntime

If the ONNX file is already strongly typed (contains explicit Cast nodes

and FP16 initializers), AutoCast is not needed — skip directly to Step 2.

• A working baseline. Capture the weakly-typed engine's outputs on a known

input set before migrating. Strong typing is stricter, and any silent precision

substitutions weak typing was making will become observable.

• The verifier script. scripts/verify.sh runs migrate.py against a

committed sample and asserts the rewrite is well-formed. Run it before editing

anything to confirm the helper works.

Migration paths

There are three migration paths — Python builder, trtexec, C++ builder.

**Infer the path from the query when signals are clear; ask only if genuinely

uncertain.** Match the strongest signal:

| Signal in the request | Path | Entry point |

|------------------------|------|-------------|

| Python tokens: trt.BuilderFlag, set_flag(, create_network(, .py | A | Python INetworkDefinition builder |

| Standalone trtexec --fp16 / --int8 / --best / --bf16 | B | trtexec CLI |

| C++ tokens: BuilderFlag::, createNetworkV2, ->setFlag, .cpp / .h | C | C++ builder |

| Multiple signals | — | do each, most-specific first |

| No usable signal (generic "how do I migrate") | — | ask which entry point, or proceed with Python and say you assumed it |

The migration preserves existing precisions (kFP16 → FP16 cast, kINT8 → Q/DQ) and is

identical whether the network came from a parser or was hand-built — only Step 1

(AutoCast) is ONNX-specific, so source and precision goal rarely need a question.

Inferring the path is a low-stakes default — it does not extend to destructive

operations, which always need confirmation. migrate.py defaults to dry-run

(prints a diff); the --write flag is the human-in-the-loop confirmation. Treat

any other destructive step (overwriting a .plan, replacing an ONNX) the same way:

show, confirm, then act.

For a strongly-typed INT8 model straight from an ONNX file (not a migration of

existing builder code), jump to Step 1's ModelOpt quantization recipe.

Step 1 — (ONNX source only) AutoCast the model to mixed precision

Skip if the source is not ONNX, or if the ONNX file is already strongly typed.

A FP32 ONNX file built for a weakly-typed --fp16 flow carries no precision

information — TensorRT was free to pick FP16 or FP32 per layer. Strong typing

requires the graph itself to declare precision via Cast operators and typed

initializers. ModelOpt's AutoCast inserts these.

Use the ModelOpt CLI — it's the supported entry point and takes .npz calibration

directly. Use --calibration_data for both tools (autocast's canonical flag;

quantization's abbreviation of --calibration_data_path):

# FP16 / BF16 mixed-precision AutoCast
python -m modelopt.onnx.autocast \
    --onnx_path model.fp32.onnx \
    --output_path model.mixed.onnx \
    --low_precision_type fp16 \
    --calibration_data calib.npz

# INT8 / FP8 strongly-typed quantization (produces ONNX with explicit Q/DQ)
python -m modelopt.onnx.quantization \
    --onnx_path model.fp32.onnx \
    --output_path model.int8.onnx \
    --quantize_mode int8 \
    --high_precision_dtype fp16 \
    --calibration_data calib.npz

Keep accuracy-sensitive ops in FP32 via AutoCast's node/op-type exclusion lists

(e.g. exclude MatMul/LayerNormalization on transformer-heavy networks), and keep

FP32 graph I/O when downstream stages expect it. See the

Model Optimizer docs for the exact

module and flags for your ModelOpt version.

Verify the converted model in ONNX Runtime before passing it to TRT:

import onnxruntime, numpy as np
sess = onnxruntime.InferenceSession("model.mixed.onnx", providers=["CPUExecutionProvider"])
# Compare a few outputs against the FP32 model with np.allclose(..., rtol=5e-3, atol=5e-3).

If parity fails here, stop — the conversion is wrong and no amount of TRT

engine work will recover it. Re-run with tighter exclusion lists.

Next step — build the strongly-typed plan from the converted ONNX:

# For the INT8 case (modelopt.onnx.quantization output)
trtexec --onnx=model.int8.onnx --stronglyTyped --saveEngine=model.int8.plan

# For the mixed-precision case (modelopt.onnx.autocast output)
trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan

Critical: do NOT pass --int8 or --fp16 alongside --stronglyTyped. Those flags are removed in TRT 11, and on TRT 10.12+ they conflict with strong typing. The precision is already encoded in the ONNX (via explicit Cast / Q / DQ nodes); the builder honors it.

The reference recipe for the end-to-end INT8 flow is samples/python/strongly_type_autocast/sample.py in tensorrt-oss.

Step 2A — Migrate a Python INetworkDefinition builder

Mechanical replacements:

| Weakly typed | Strongly typed |

|--------------|----------------|

| builder.create_network(0) or builder.create_network(1 << int(NetworkDefinitionCreationFlag.EXPLICIT_BATCH)) | builder.create_network(1 << int(NetworkDefinitionCreationFlag.STRONGLY_TYPED)) |

| config.set_flag(trt.BuilderFlag.FP16) | Remove. Types come from the network. |

| config.set_flag(trt.BuilderFlag.BF16) | Remove. |

| config.set_flag(trt.BuilderFlag.INT8) | Remove. Quantization expressed via Q/DQ nodes in the graph. |

| config.set_flag(trt.BuilderFlag.TF32) | Keep. kTF32 is retained in 11.x — it allows (not requires) TF32 for FP32 tensors and is orthogonal to typing. |

| layer.precision = trt.float16 | Remove. Cast the input tensor instead (network.add_cast). |

| layer.set_output_type(0, trt.float16) | Remove. Output type follows the network; for cast/quantize/dequantize/fill layers use set_to_type instead. |

| builder.platform_has_fast_int8 / has_fast_fp16 checks gating the above | Remove the gating logic along with the flag. |

Worked rewrite (the MNIST builder from samples/python/network_api_pytorch_mnist/sample.py

already uses the strongly-typed form — note NetworkDefinitionCreationFlag.STRONGLY_TYPED

and the absence of any BuilderFlag.FP16 call):

builder = trt.Builder(TRT_LOGGER)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, common.GiB(1))
# ... populate network ...
plan = builder.build_serialized_network(network, config)

If the network is built via the ONNX parser, the parser respects strong typing

automatically — no code change is needed beyond the flag.

Where a removed precision / set_output_type hint actually mattered, encode it

in the graph: cast a tensor's dtype with add_cast, or insert an explicit

Quantize/Dequantize pair for INT8 (the scale is a build-time constant; ModelOpt

derives it from calibration, or supply it by hand):

# kFP16 hint -> explicit cast on the tensor:
cast = network.add_cast(some_tensor, trt.float16)

# kINT8 hint -> explicit Q/DQ pair (per-tensor scalar scale). NOTE: trt.Weights does
# not copy the numpy buffer, so q_scale must stay alive until the engine is built.
q_scale = np.array(1.0 / 127.0, dtype=np.float32).reshape(())          # rank-0 scalar
scale = network.add_constant(shape=(), weights=trt.Weights(q_scale)).get_output(0)
q  = network.add_quantize(some_tensor, scale, trt.int8).get_output(0)
dq = network.add_dequantize(q, scale, trt.float32).get_output(0)
# Feed dq into the consumer; TensorRT fuses the Q/DQ into it so it runs in INT8.

Drive the rewrite from scripts/migrate.py:

python3 scripts/migrate.py path/to/build.py            # dry-run diff
python3 scripts/migrate.py path/to/build.py --write    # rewrite in place

The script edits AST nodes (not regexes) — it understands create_network and

set_flag call shapes, so it is safe on files that mix conventions.

Step 2B — Migrate a trtexec command line

Replacements:

| Weakly typed flag | Strongly typed equivalent |

|-------------------|---------------------------|

| --fp16 | Remove. Precision comes from the ONNX. See the --stronglyTyped note below. |

| --bf16 | Remove. Same as --fp16. |

| --int8 | Remove. Q/DQ must be present in the ONNX. |

| --best | Remove. Not allowed with --stronglyTyped on 10.12–10.x (see Common Errors). |

| --stronglyTyped | Version-gated. Required on 10.12–10.x to opt into strong typing; a no-op (default-on) on 11.x — drop it or leave it, your choice. |

| --noTF32 | Keep. kTF32 is retained in 11.x; --noTF32 still disables TF32 and is orthogonal to typing. |

| --precisionConstraints=obey / prefer | Remove. Constraints are expressed in the graph. |

| --layerPrecisions=... | Remove. Use ONNX Cast nodes / AutoCast. |

| --layerOutputTypes=... | Remove. |

Worked rewrite:

# Before (weakly typed)
trtexec --onnx=model.onnx --fp16 --saveEngine=model.fp16.plan \
        --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \
        --maxShapes=input:16x3x224x224

# After (strongly typed; ONNX has been AutoCast'd to mixed FP16)
# On 11.x --stronglyTyped is the default and a no-op — the line below is equivalent
# with or without it. On 10.12–10.x it is REQUIRED, or the build stays weakly typed.
trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan \
        --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \
        --maxShapes=input:16x3x224x224

The trtexec README's Example 6

is the canonical reference.

Step 2C — Migrate a C++ builder

Replacements mirror the Python path:

| Weakly typed (C++) | Strongly typed (C++) |

|---|---|

| builder->createNetworkV2(0) | builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)) |

| config->setFlag(BuilderFlag::kFP16) | Remove. |

| config->setFlag(BuilderFlag::kBF16) | Remove. |

| config->setFlag(BuilderFlag::kINT8) | Remove. |

| config->setFlag(BuilderFlag::kTF32) | Keep. kTF32 is retained in 11.x — orthogonal to typing. |

| config->setFlag(BuilderFlag::kOBEY_PRECISION_CONSTRAINTS / kPREFER_PRECISION_CONSTRAINTS) | Remove. No per-layer precision exists to constrain. |

| layer->setPrecision(DataType::kHALF) | Remove. Cast the input tensor instead (addCast). |

| layer->setComputePrecision(...) | Remove. Deprecated 10.16, removed; compute precision follows input dtypes. |

| layer->setOutputType(0, DataType::kHALF) | Remove. For cast/quantize/dequantize/fill layers use setToType() instead. |

Worked rewrite:

// Before (10.x weakly typed) — the builder picks precision from these hints:
auto network = std::unique_ptr<INetworkDefinition>(
    builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH)));
auto config = std::unique_ptr<IBuilderConfig>(builder->createBuilderConfig());
config->setFlag(BuilderFlag::kFP16);                        // hint: builder may use FP16
config->setFlag(BuilderFlag::kINT8);                        // hint: builder may use INT8 (+ a calibrator)
config->setFlag(BuilderFlag::kOBEY_PRECISION_CONSTRAINTS);  // force the per-layer overrides below
someLayer->setPrecision(DataType::kHALF);                   // per-layer override
someLayer->setOutputType(0, DataType::kHALF);               // per-layer override

Each hint above maps to a concrete graph edit — there is no longer a "tell the

builder to prefer X" knob, so state the dtype directly:

// After (11.x strongly typed) — precision lives in the graph; every line above is gone.
auto network = std::unique_ptr<INetworkDefinition>(
    builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));
    // kSTRONGLY_TYPED: required on 10.12–10.x, accepted-but-ignored on 11.x (0 works too).
auto config = std::unique_ptr<IBuilderConfig>(builder->createBuilderConfig());
// No precision setFlag, no setPrecision / setOutputType.

// kFP16 hint  ->  cast the tensor's dtype explicitly:
auto* cast = network->addCast(*someTensor, DataType::kHALF);
// feed cast->getOutput(0) into the consumer layer

// kINT8 hint  ->  insert an explicit Quantize/Dequantize pair. The scale is a
// build-time constant (per-tensor scalar here); ModelOpt derives it from calibration,
// or supply it by hand. NOTE: TensorRT does not copy Weights, so qScale must stay
// alive until the engine is built.
float qScale = 1.0f / 127.0f;
auto* scale = network->addConstant(Dims{0, {}},
                                   Weights{DataType::kFLOAT, &qScale, 1})->getOutput(0);
auto* q  = network->addQuantize(*someTensor, *scale, DataType::kINT8)->getOutput(0);
auto* dq = network->addDequantize(*q, *scale, DataType::kFLOAT)->getOutput(0);
// feed dq into the consumer; TensorRT fuses the Q/DQ into it so it runs in INT8.

// kOBEY_PRECISION_CONSTRAINTS + setPrecision/setOutputType  ->  nothing to port:
// the dtype is now fixed by the cast / Q-DQ above, so there is no constraint to obey.

setToType() is the only surviving per-layer type control, and is valid only

on ICastLayer, IDequantizeLayer, IDynamicQuantizeLayer, IFillLayer, and

IQuantizeLayer — not on arbitrary layers.

If the build pipeline parses ONNX via nvonnxparser::IParser, no parser change is

required — the parser honors the strong typing flag set on the network.

Step 3 — Rebuild and validate

• Rebuild the engine with the migrated flow.
• Run the captured baseline inputs through the new engine.
• Compare against the weakly-typed baseline:
• Outputs match within atol=5e-3, rtol=5e-3: migration is complete.
• Outputs diverge but stay within model accuracy tolerance: weak typing

was opportunistically using FP16/INT8 where the new graph keeps FP32 (or vice

versa). Acceptable if the downstream metric (top-1, BLEU, WER) is unchanged;

confirm with the original accuracy harness.

• Outputs diverge beyond tolerance: see Common Errors below.

If the source was ONNX, also run the AutoCast'd ONNX through ONNX Runtime and

compare against the FP32 ONNX before blaming TRT — graph-level conversion errors

should be caught there.

Migration Patterns Reference

| Pattern | Action |

|---------|--------|

| Network created with no flags (legacy implicit batch) | First migrate to explicit batch; only then add STRONGLY_TYPED |

| Multiple set_flag calls (FP16 + INT8 + TF32 + REFIT) | Keep REFIT, SPARSE_WEIGHTS, kTF32; drop the precision-hint flags (FP16/BF16/INT8/FP8) and the removed kOBEY/kPREFER_PRECISION_CONSTRAINTS |

| set_calibration_profile / IInt8Calibrator | Remove. PTQ calibration is replaced by Q/DQ in the ONNX. Use ModelOpt's PTQ for the model first. |

| Mixed FP16/FP32 per-layer overrides via layer.precision | Express the same intent with Cast ops in the ONNX, or in the Python network construction code |

| Plugin layers with explicit precision | Plugins must implement IPluginV3OneBuild::getOutputDataTypes; the network reads types from the plugin |

| Engine consumed by downstream framework expecting FP32 I/O | Set keep_io_types=True in AutoCast, or add explicit Cast to FP32 at the network outputs |

Common Errors

• "BuilderFlag::kFP16 is not compatible with a strongly typed network" —

a setFlag/set_flag call survived the migration. Re-run migrate.py or grep

for BuilderFlag\. / BuilderFlag:: in the migrated source.

• "--best is incompatible with --stronglyTyped" — drop --best from the

trtexec line; precision choices live in the graph now.

• "Network has no marked outputs" immediately after switching the flag —

some builders only call mark_output on the path conditioned by

platform_has_fast_fp16. Remove the conditional.

• AutoCast'd ONNX fails the parser with "If subgraph type mismatch" — AutoCast

can corrupt certain If node subgraphs. Add the affected If node names to

nodes_to_exclude so they stay FP32.

• Engine output is FP32 zeros after migration — keep_io_types=False was used

but the host code still feeds FP32 buffers. Either set keep_io_types=True or

cast the host input before binding.

• Accuracy regression of ~1-3% after migration — known case for

transformer-heavy networks: leftover Cast(to=FLOAT32) from AutoCast combined

with FP16 weights causes thrash. Work around by excluding the residual cast op

types in op_types_to_exclude.

Pitfalls

• Do not partially migrate. Half-migrated builds (some set_flag calls

removed, STRONGLY_TYPED not set) silently fall back to weak typing and the

warning is easy to miss. Check the engine build log for `"Strongly typed network

detected"` to confirm the flag took effect.

• Do not migrate without a baseline. Weak typing's per-layer precision choices

are non-deterministic across TRT versions; without the old outputs you have no

ground truth and cannot tell whether the new engine is correct.

• Do not run --stronglyTyped against a FP32 ONNX expecting FP16 speedup.

Strong typing builds exactly what the graph specifies — a FP32 ONNX produces a

FP32 engine. Run AutoCast first.

• Plugins must declare types. Old plugins that left output types as "follow

input" will fail to build under strong typing. Update them to the

IPluginV3OneBuild interface, the only plugin API that supports strong typing

in 11.x.

• Refit + strong typing: refittable strongly-typed engines work, but the refit

weights must match the precision declared in the network. A FP16 layer cannot be

refit with FP32 weights — convert host weights before calling setNamedWeights.

• Don't trust grep alone for completeness. migrate.py uses AST matching, so

it catches the precision flags regardless of how tensorrt is imported or

aliased via as — something string-level grep misses. It matches direct

attribute access (trt.BuilderFlag.FP16); dynamic forms such as

getattr(trt.BuilderFlag, 'FP16') are *not* rewritten and need manual review.

Run the AST helper as the source of truth.

References

• Migration guide: TensorRT 10.x → 11.x

— the authoritative description of the weak → strong typing change.

• Operator type rules: the TensorRT Operators Reference

— each operator's page lists its supported input/output types, i.e. how its output

dtype follows its inputs in a strongly-typed graph.

• API reference: the TensorRT C++/Python API docs for NetworkDefinitionCreationFlag,

BuilderFlag, ILayer::setToType, and INetworkDefinition::addCast / addQuantize /

addDequantize.

• TensorRT Model Optimizer: <https://nvidia.github.io/Model-Optimizer> — AutoCast and

quantization tooling for getting precision into the model.

• Sample code (TensorRT OSS, <https://github.com/NVIDIA/TensorRT>):

samples/trtexec/README.md Example 6

(strongly-typed plan), samples/python/strongly_type_autocast/sample.py

(AutoCast → strongly-typed INT8), and samples/python/network_api_pytorch_mnist/sample.py

(strongly-typed Python builder).