TensorRT-LLM — NVIDIA's Fastest LLM Inference

TensorRT-LLM (TRT-LLM) is NVIDIA’s open-source inference library for Large Language Models and Visual Gen models on NVIDIA GPUs. It delivers state-of-the-art optimizations — custom kernels for attention/GEMM/MoE, plus algorithmic runtime tricks like prefill-decode disaggregation and speculative decoding — behind an easy Python API. Apache 2.0, ~14.3k GitHub stars, fully open-sourced on GitHub since March 2024. Current line: release 1.3 (Python 3.10–3.12, CUDA 13.2, PyTorch 2.11).

What Problem Does It Solve?#

LLM inference is bound by memory bandwidth (weights + KV cache) and compute. Generic frameworks leave most of it on the table:

Architecture#

Built on PyTorch, not a C++ black box:

• High-level Python LLM API — define models, build engines, run generation in a few lines
• Custom kernels for attention, GEMMs, MoE
• Python + C++ runtimes orchestrate execution
• Modular — models written in native PyTorch, easy to extend
• Scales from single GPU to multi-GPU / multi-node (TP, PP, CP, EP)

Integrates with the wider ecosystem: NVIDIA Dynamo (datacenter-scale distributed serving) and the Triton Inference Server.

Key Optimizations#

Performance Numbers#

• Llama 4 — >40,000 tokens/s on B200 GPUs

• 1,000 tokens/s/user with Llama 4 Maverick on Blackwell

• H200 — ~12,000 tok/s on Llama 2 13B
• DeepSeek-R1: world-record inference on Blackwell (FP4)
• XQA kernel: 2.4x Llama-70B throughput within the same latency budget
• Falcon-180B INT4 AWQ on a single H200

Visual Generation#

Since April 2025 TRT-LLM also covers Visual Gen — diffusion models for image and video. NVIDIA has scaled video generation across an NVL72 rack (72 GPUs) with TRT-LLM.

Serving & Deployment#

trtllm-serve llama-3.1-8b-instruct-fp8 # OpenAI-compatible server

trtllm-bench latency --model <model-dir> # benchmark engine

• Triton Inference Server backend for production serving
• Works with NVIDIA Dynamo and NIM
• Day-0 support for new open-weights models (GPT-OSS, EXAONE 4.0, DeepSeek)

Getting Started#

from tensorrt_llm import LLM, SamplingParams

llm = LLM(model="nvidia/Llama-3.1-8B-Instruct-FP8")

params = SamplingParams(max_tokens=128, temperature=0.8)
outputs = llm.generate(["Tell me about TensorRT-LLM."], params)

for out in outputs:
    print(out.outputs[0].text)

pip install tensorrt-llm
# or pull the NGC container with prebuilt wheels

TRT-LLM vs vLLM#

2026 consensus: vLLM for general serving and portability; TRT-LLM when you want absolute peak performance on NVIDIA hardware and can pay the compile time.

Caveats#

• NVIDIA GPUs only
• Engine compile is slow (minutes), and engines are GPU-architecture-specific
• Telemetry is on by default — opt out via TRTLLM_NO_USAGE_STATS=1 or --no-telemetry
• 3-month deprecation policy for APIs

Conclusion#

TRT-LLM is NVIDIA’s path to the fastest LLM and Visual Gen inference. The numbers speak: >40k tokens/s on B200. The cost: CUDA lock-in plus engine compile time. Pick it when peak performance on NVIDIA hardware matters more than portability.

References#

• TensorRT-LLM GitHub ↗
• TensorRT-LLM Documentation ↗
• TensorRT-LLM Quick Start Guide ↗
• DeepSeek-V4 on NVIDIA Blackwell — Tech Blog ↗
• Triton Inference Server ↗
• NVIDIA Dynamo ↗