vLLM — High-Performance LLM Serving Engine • blog.dopana

vLLM is a high-performance open-source LLM inference & serving engine from UC Berkeley (SOSP 2023), Apache 2.0. Latest: v0.26.0 (Jul 2026). ~77.5k GitHub stars, 2.79M weekly PyPI downloads. #1 OSS project by contributors (GitHub Octoverse 2025).

PagedAttention — The Core Innovation#

PagedAttention manages KV cache like OS virtual memory: fixed-size blocks, dynamic alloc/free, copy-on-write sharing. Results:

Memory waste drops from 60-80% to <4%
24x higher throughput vs naive HF Transformers serving
Prefix sharing: system prompt computed once, reused across requests

Key Features#

Feature	Description
Continuous batching	New requests join running batches mid-stream
Chunked prefill	Splits long prompts into chunks, avoids blocking decode
Prefix caching	Up to 40% latency reduction for repeated system prompts
Tensor/pipeline/expert parallelism	Scale across 8+ GPUs
Speculative decoding	EAGLE 3.1, DFlash — up to 2x throughput
Disaggregated prefill/decode	Separate servers for independent scaling
KV cache quantization	FP8, NVFP4, INT8, INT4

Quantization Support#

Format	Implementation	Hardware
FP8 (W8A8)	Dynamic/static, per-token-group activation	Hopper+
NVFP4	NVIDIA native 4-bit float (SM100+)	Blackwell
MXFP4	OCP Microscaling 4-bit	Blackwell, Hopper
INT4 (AWQ/GPTQ)	Marlin kernels	Turing+
INT8	llm-compressor, static	Ampere+
KV cache	FP8, NVFP4, INT8, INT4	Hopper+

2026 consensus: BF16 → FP8 (Hopper) → NVFP4/MXFP4 (Blackwell).

Hardware#

NVIDIA CUDA: Volta → Blackwell (SM 7.0–12.0). Full feature coverage
AMD ROCm: MI300X, MI355X
Intel XPU: Arc Pro B-series
Google TPU: Plugin (PyTorch + JAX)
Apple Silicon: vllm-metal plugin (MLX-based), ~525 tok/s M4 Max
CPU: x86/ARM/PowerPC/S390X
Huawei Ascend, AWS Trainium/Inferentia, IBM Spyre: Plugins

200+ model architectures supported on Hugging Face.

Since 2025-2026, vLLM expanded to multi-modal:

Image: LLaVA, Qwen2.5-VL, Pixtral, Gemma3
Video: Qwen2.5-VL, Qwen3-VL
Audio: Whisper, Qwen2-Audio, Ultravox
Omni: Qwen3-Omni — interleaved audio+video

vLLM-Omni (Nov 2025) adds: Diffusion Transformers (DiT), TTS, non-autoregressive generation, heterogeneous outputs.

Serving Features#

OpenAI-compatible API: /v1/chat/completions, /v1/completions, /v1/embeddings — drop-in replacement
Anthropic Messages API
Structured outputs: xgrammar + guidance. guided_json, guided_regex, guided_grammar
Tool calling: Hermes, Mistral, Llama3, Qwen, xLAM, Granite…
Multi-LoRA: Fused kernels for dense + MoE layers
Realtime API: WebSocket streaming
vLLM Semantic Router: Multimodal routing, load balancing

Performance#

H100 single-GPU throughput (batch=32, FP16):

Engine	7B	13B	70B
vLLM	180 tok/s	120	50
SGLang	160	110	45
TGI	140	90	40
TensorRT-LLM	280	200	100
llama.cpp	8	6	2

Concurrent serving — Llama 3 70B (A100 80GB):

Concurrent	vLLM	Ollama
10	~30 tok/s	~15 tok/s
50	~18 tok/s	~4 tok/s
100	~12 tok/s	OOM
GPU mem utilization	~96%	60-70%
KV cache waste	<4%	40-60%

vLLM vs Alternatives#

Dimension	vLLM	TGI	TRT-LLM	SGLang	llama.cpp
PagedAttention	✅ Native	❌	Partial	RadixAttention	❌
Throughput	Baseline	~78%	2-3x peak	~90-95%	~5%
Cold start	~62s	Fast	~28min compile	~58s	Instant
Hardware	Multi-platform	CUDA/ROCm	CUDA only	CUDA/ROCm	All
OpenAI API	Full+Anthropic	Partial	Via Triton	Full	Partial
Quantization	FP8/FP4/INT4…	Same	FP4/FP8/INT4	Same	GGUF

2026 consensus: Default to vLLM for GPU serving. SGLang for heavy prefix sharing. TRT-LLM for static peak performance. TGI in maintenance mode.

Timeline 2025-2026#

Date	Milestone
Jan 2025	vLLM V1 architecture rewrite
Nov 2025	vLLM-Omni released — omni-modality
Jan 2026	Streaming Requests, Realtime WebSocket API
Mar 2026	P-EAGLE parallel speculative decoding
Apr 2026	FP8 KV-Cache, RTX PRO 6000 NVFP4 — 6.5x throughput
May 2026	EAGLE 3.1, Native RL APIs
Jul 2026	v0.26.0 (PyPI)

vLLM — High-Performance LLM Serving Engine