MXFP4/MXFP8 Quantization — Running LLMs Locally
MXFP4/MXFP8 is the new 4-bit quantization standard for LLMs. Enables 120B-param models on a single 80GB GPU via llama.cpp and vLLM.
From 2025-2026, a number format revolution changed local LLM inference. MXFP4 (Microscaling FP4) — the OCP-standardized 4-bit floating-point format — fits a 120B-parameter model on a single 80GB GPU with near-FP16 quality.
What is MXFP4?#
MXFP4 is a block floating-point 4-bit format standardized by OCP (v1.0, Sep 2023), backed by Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm.
Structure: Block of 32 elements shares one 8-bit power-of-2 exponent scale (E8M0). Each element is E2M1 (1 sign, 2 exponent, 1 mantissa). Values range [-6, 6]. Cost: 4.25 bits/element (32×4 + 8 bits / 32).
Format family: MXFP8 (E5M2/E4M3), MXFP6 (E3M2/E2M3), MXFP4 (E2M1), MXINT8.
NVFP4 (NVIDIA, Jun 2025): Block size 16 (vs 32), scale in FP8 E4M3 (not E8M0), plus per-tensor FP32 scale. Higher fidelity at 4.5 bits/element. Designed for Blackwell FP4 tensor cores.
HiF4 (2026): Another 4-bit BFP variant. Claims higher accuracy than NVFP4 on LLaMA, Qwen, Mistral, DeepSeek-V3.1.
Models Using MXFP4/NVFP4#
| Model | Format | Details |
|---|---|---|
| DeepSeek-V4 (1.6T MoE) | NVFP4 QAT | Expert weights + sparse-attention indexer in FP4 |
| GPT-OSS 120B/20B | MXFP4 QAT | MoE experts (>90% params) trained in MXFP4. 120B fits 1×80GB GPU |
| Llama 4 Scout/Maverick | NVFP4 | NVIDIA checkpoints available |
| Qwen 3, 3.6 | NVFP4 | Quantized via nvidia-modelopt |
| DeepSeek-R1 | NVFP4/MXFP4 | NVIDIA: NVFP4; AMD: MXFP4-ASQ |
By Apr 2026, OpenAI, Anthropic, Google, and most open-source-as-a-service providers run MXFP4 as default inference format.
llama.cpp with FP4#
llama.cpp supports FP4 since early 2026:
- NVFP4 (mainline): Type ID
GGML_TYPE_NVFP4 = 40. CUDA dp4a kernel, generic MMQ kernel, SYCL, Vulkan — all landed. Blackwell-native dispatch in build b8967 (Apr 2026). - MXFP4 (ik_llama.cpp fork by Iwan Kawrakow): Support since Nov 2025, includes GPT-OSS with fused-MoE optimizations.
Hardware reality:
| GPU | NVFP4 performance |
|---|---|
| Blackwell (RTX 5090, B200, GB200) | Full tensor-core acceleration |
| Ada Lovelace (RTX 4090) | Memory savings, small speedup |
| Ampere (RTX 3090, A100) | Memory savings, no compute win |
| AMD MI300X | MXFP4-aligned via ROCm |
vLLM with FP4#
vLLM supports both MXFP4 W4A4 and NVFP4 ModelOpt format on Blackwell GPUs:
- MXFP4 W4A4: Both weights and activations at
float4_e2m1fn_x2. Two methods: single-scale (online, baseline) and dual-scale (needs checkpoint prep). - NVFP4: Load pre-quantized models from HF with
--quantization modelopt. Supports dense and MoE. - KV cache FP8:
--kv-cache-dtype fp8— quantize KV cache at runtime.
Benchmark (Spheron, Apr 2026) — Llama 3.3 70B:
| GPU | Precision | Tok/s | VRAM (weights) | Cost/1M tok |
|---|---|---|---|---|
| B200 SXM6 | NVFP4 | ~12,841 | ~35 GB | ~$0.161 |
| B200 SXM6 | FP8 | ~6,972 | ~70 GB | ~$0.296 |
| H100 SXM | FP8 | ~3,066 | ~70 GB | ~$0.263 |
| A100 80G | AWQ INT4 | ~1,800 | ~35 GB | ~$0.253 |
Quantization Options#
llama.cpp (static, offline)#
llama-quantize applies quantization offline. Options:
- K-quants: Q2_K → Q6_K with S/M/_L controlling per-tensor mix
- I-quants: IQ1_S → IQ4_NL, IQ4_XS — non-linear, need calibration
- NVFP4/MXFP4: Latest static formats
- KV cache runtime:
--cache-type-k q4_0 --cache-type-v q4_0
vLLM (dynamic)#
- FP8 online dynamic:
--quantization fp8. Activations get dynamic per-tensor scales. No calibration needed. - FP8 static + dynamic activations: Weights static, activations dynamic.
- FP8 KV cache dynamic:
--kv-cache-dtype fp8. Per-head scales supported. - NVFP4: Static weights (via ModelOpt). Activations static or dynamic.
Quality vs Bit Width#
- MXFP4 baseline loses up to 10% on MMLU-Pro (DeepSeek-R1) vs FP16.
- NVFP4 closes most of this gap.
- New techniques (OAS, MBS from ICML 2026) bring MXFP4 within 1% of NVFP4 at 6.2% GEMM overhead.
- AMD evaluation (Oct 2025): MXFP4 accuracy scales with model size. DeepSeek-R1-0528 retains 96%+ with MXFP4. Smaller models (70B) degrade more — MXFP6 or mixed MXFP4-MXFP6 significantly closes the gap.
Conclusion#
MXFP4 and NVFP4 changed the game: a 120B-parameter model fits on a single 1M+ cluster. With llama.cpp and vLLM support mature, self-hosting powerful LLMs is no longer out of reach.