LiteRT — On-Device AI Inference (Formerly TF Lite)
Google's on-device inference runtime, formerly TensorFlow Lite. CompiledModel API, GPU/NPU acceleration, on-device LLMs via LiteRT-LM. .tflite format.
LiteRT (Lite Runtime) is Google’s open-source runtime for on-device ML and GenAI inference — previously known as TensorFlow Lite (renamed September 4, 2024). It’s the most widely deployed ML runtime in the world, shipping inference inside billions of Android devices. The .tflite file format is unchanged; only the name and the project home moved (github.com/google-ai-edge/LiteRT).
Why the Rename?#
TensorFlow Lite grew far beyond TensorFlow: it now converts models authored in PyTorch, JAX, and Keras, so naming it after a single framework was misleading. “LiteRT” is framework-neutral — same team, same code, same format.
Two Inference APIs#
| API | Status | Notes |
|---|---|---|
| Interpreter | Legacy, stable | The classic API, still works |
| CompiledModel | New (production-ready Jan 2026) | Streamlined GPU/NPU acceleration, significantly faster |
The Pipeline#
Trained model (TF/Keras/JAX/PyTorch)
│
▼
Converter ──────────────► .tflite file (FlatBuffers)
│
▼
LiteRT runtime (CPU / GPU / NPU / DSP / TPU delegates)Converters:
import tensorflow as tf
# From Keras / SavedModel
converter = tf.lite.TFLiteConverter.from_saved_model("model_dir")
# or: from_keras_model(model), from_jax(jax_fn)
# Quantize to INT8 (post-training quantization)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
open("model.tflite", "wb").write(tflite_model)PyTorch models convert via the LiteRT Torch Converter (uses torch.export). LLMs use the .litertlm packaging format.
Hardware Acceleration (Delegates)#
| Delegate | Hardware |
|---|---|
| GPU | OpenGL/OpenCL/Vulkan (Android), Metal (iOS), WebGPU (web) |
| Qualcomm QNN | Snapdragon NPU |
| MediaTek APU | MediaTek SoCs |
| Samsung Exynos | Exynos NPU (S.LSI) |
| Google Tensor | Pixel NPU |
| Edge TPU | Coral boards |
| Hexagon DSP | Older Snapdragons |
| Core ML | Apple Neural Engine (iOS/macOS) |
| XNNPACK | CPU fallback |
LiteRT ships ~1.4x faster GPU inference than legacy TF Lite plus new NPU acceleration.
Quantization#
- Post-training INT8 / FP16 — smallest and fastest models
- AI Edge Quantizer — newer toolkit aimed at LLMs
- 8-bit quantized models run directly on GPU/NPU delegates
On-Device LLMs#
LiteRT now runs LLMs on-device via LiteRT-LM — Gemma, Phi-4-mini, Qwen, Llama variants. Inference stays local: low latency, full privacy, works offline.
Web & Microcontrollers#
- LiteRT.js — high-performance browser inference via WebGPU and WASM
- LiteRT for microcontrollers (formerly TF Lite Micro) — runs on MCUs
Running Inference#
// Android — CompiledModel API (Kotlin)
val model = ai.edge.litert.loadCompiledModel(modelBuffer)
val output = model.run(inputTensor)# Python — Interpreter API
import ai_edge_litert as litert
interpreter = litert.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()
interpreter.invoke()LiteRT vs Competitors#
| Runtime | Vendor | Notes |
|---|---|---|
| LiteRT | Most mature, billions of devices, on-device LLMs | |
| ONNX Runtime Mobile | Microsoft | Cross-framework, strong CPU |
| Core ML | Apple | Apple Silicon/ANE, iOS-only |
| ExecuTorch | Meta | PyTorch-native, closing the gap fast |
Conclusion#
LiteRT is the reference standard for on-device inference — the most mature toolchain across Android, iOS, web, IoT, and microcontrollers, now with production-grade NPU acceleration and on-device LLMs. If your model must run on a phone without a network round-trip, LiteRT remains the safest choice in 2026.