gpu-bench: Cross-Stack GPU Inference Benchmark (CUDA/TensorRT vs ROCm)

Most TensorRT benchmarks conflate two separate effects — graph compilation and lower precision — into one before/after number. I wanted to isolate them: a three-rung ladder (stock PyTorch → TensorRT FP32 → TensorRT FP16) that holds precision constant across the first two rungs and compilation constant across the last two, so each rung's gain traces back to one specific cause. This is Phase 1 on an RTX 3080; Phase 2 repeats the same harness on AMD's MI300X to see how much of it holds off NVIDIA.

Phase 1 — CUDA/TensorRT ladder (complete)

Hardware: RTX 3080 (10GB), WSL2 Ubuntu, PyTorch 2.13 + CUDA 12.6, TensorRT 11.1 Model: YOLO11n (pretrained COCO), 640x640 input

Three-rung ladder isolating compilation gains from precision gains:

Stack	Batch 1	Batch 4	Batch 8
PyTorch CUDA	8.6 ms / 117 fps	10.1 ms / 398 fps	13.0 ms / 616 fps
TensorRT FP32	4.7 ms / 220 fps	6.6 ms / 600 fps	10.8 ms / 740 fps
TensorRT FP16	4.4 ms / 230 fps	5.6 ms / 715 fps	8.0 ms / 1000 fps

Key findings:

• Compilation (TensorRT graph optimization, precision held at FP32) gains most at batch 1 (~45%), shrinking to ~19% at batch 8 — small batches are launch-overhead-bound, and fusion attacks exactly that fixed cost.

• FP16 gains ~0 at batch 1 (inside measured noise floor) but ~25% at batch 8 — precision only pays where bytes are the bottleneck (bandwidth-bound regime).

• The "FP16" engine is actually mixed-precision: 94.3% of nodes converted; detection-head coordinate/grid math (values up to 8400, exceeding the autocast safe range of ±512) retained in FP32.

• Measured run-to-run noise floor of ~±0.5 ms at batch 1 on shared desktop hardware; p95 tail dominated by desktop GPU contention (spikes scattered through runs, not startup-clustered).

• Reported memory (pytorch_mem_mb in results.csv) comes from torch.cuda.max_memory_allocated(), which only tracks PyTorch's own allocator — it's blind to TensorRT's internal allocations, so the tensorrt-fp32/fp16 rows undercount actual GPU footprint. Only comparable rung-to-rung within TensorRT, not against the PyTorch baseline.

Methodology

• 50 warmup runs discarded (lazy init, cuDNN autotuning, allocator warmup, clock ramp)
• 100 timed runs per config; torch.cuda.synchronize() inside the timing loop (GPU calls are async — without it you time the handoff, not the work)
• Median + p95 reported; raw pre-sort latencies logged (arrival order distinguishes warmup residue from ambient contention)
• Replicate runs per rung; deltas judged against measured noise floor
• Dynamic-shape TensorRT engine (batch 1-8) accepted a small specialization penalty to keep the config matrix consistent across rungs

Phase 2 — ROCm on MI300X (complete)

Hardware: AMD Instinct MI300X (192GB HBM3), Ubuntu, PyTorch + ROCm, MIGraphX 2.15 Model: YOLO11n (pretrained COCO), 640x640 input

The AMD-side ladder mirrors Phase 1, moving from PyTorch-ROCm baseline → MIGraphX FP32 → MIGraphX FP16. MIGraphX is the headline comparison here: it's AMD's first-party graph-level optimizer, the exact architectural equivalent to what TensorRT does on the CUDA side.

Stack	Batch 1	Batch 4	Batch 8
PyTorch ROCm (FP32)	6.0 ms / 165 fps	6.5 ms / 613 fps	7.0 ms / 1143 fps
MIGraphX (FP32)	2.1 ms / 465 fps	2.4 ms / 1680 fps	2.7 ms / 2909 fps
MIGraphX (FP16)	1.0 ms / 974 fps	1.0 ms / 3846 fps	1.1 ms / 7277 fps

Key findings:

• The Brute Force Baseline: Out of the box, unoptimized PyTorch FP32 on the MI300X (1,143 fps @ batch 8) naturally outmuscles the fully optimized TensorRT FP16 engine on the RTX 3080 (1,008 fps) purely through raw hardware scale.
• Static Shape Requirement: Unlike the dynamic-shape TensorRT engine in Phase 1, hitting peak MIGraphX numbers required compiling dedicated static binaries (.mxr) for each specific batch size. Bypassing dynamic shape-checking overhead is what allows the engine to pre-allocate exact memory strides and hit maximum hardware saturation.
• Zero-Latency Scaling: Going from batch 1 to batch 8 in MIGraphX FP16 only increased total execution time by a fraction of a millisecond (1.03ms → 1.10ms). The MI300X is so wide that it absorbs the 8x workload increase with practically zero latency penalty, scaling throughput almost perfectly linearly to an insane 7,277 fps.

Files

• bench.py — the harness (swap MODEL_NAME / RUN_LABEL per rung)
• bench_migraphx.py — the AMD-specific compiled engine harness
• results.csv — all runs, appended
• setup_rocm.sh — Phase 2 environment bootstrap for the MI300X box (ROCm sanity check, venv, PyTorch-ROCm wheel, ultralytics)