TensorFlow Serving — Production Model Deployment • blog.dopana

TensorFlow Serving is Google’s flexible, high-performance serving system for machine learning models in production. It handles the inference side of ML: takes a trained model, manages its lifetime, and serves predictions to clients with versioned access — through a high-performance, reference-counted lookup table.

What It Solves#

Serving ML models in production is harder than it looks:

How to update a model without dropping requests or breaking clients?
How to run multiple models / versions on the same GPU?
How to batch requests to keep the GPU busy?
How to test a new model safely (canary / A/B)?

TensorFlow Serving answers all of these out of the box.

Key Features#

Feature	What it does
Versioned models	Serve multiple versions simultaneously
Zero-downtime updates	Deploy new versions without client changes
Canary & A/B testing	Route traffic to experimental versions
GPU batching	Groups requests into batches with latency controls
gRPC + REST	Both interfaces out of the box
Multi-servable	TF models, embeddings, vocabularies, feature transforms
Low overhead	Adds minimal latency to inference

Architecture#

ModelServer — the serving binary
SavedModel — the serialized model format it consumes
Version manager — loads/unloads versions, enables rollback
Batching scheduler — batches inference requests for GPU efficiency

REST API Endpoints#

GET  /v1/models/{model}                          # status
GET  /v1/models/{model}/versions/{v}/metadata    # metadata
POST /v1/models/{model}:predict                  # prediction
POST /v1/models/{model}:classify                 # classification
POST /v1/models/{model}:regress                  # regression

text

Requests are JSON; binary data (images) is Base64-encoded.

Serve a Model in 60 Seconds#

# Pull the image and clone demo models
docker pull tensorflow/serving
git clone https://github.com/tensorflow/serving

TESTDATA="$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata"

# Start the server
docker run -t --rm -p 8501:8501 \
    -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" \
    -e MODEL_NAME=half_plus_two \
    tensorflow/serving

bash

# Predict
curl -d '{"instances": [1.0, 2.0, 5.0]}' \
    -X POST http://localhost:8501/v1/models/half_plus_two:predict
# Returns => { "predictions": [2.5, 3.0, 4.5] }

bash

Exporting a SavedModel#

import tensorflow as tf
from tensorflow import keras

model = keras.Sequential([keras.layers.Dense(1, input_shape=(1,))])
model.compile(optimizer="sgd", loss="mse")

# Train...
model.fit(x_train, y_train, epochs=10)

# Export for TensorFlow Serving
model.export("models/regressor/1")   # version dir "1"

python

Versioning: each SavedModel lives in a numbered directory — dropping a new directory deploys a new version instantly.

GPU Batching#

The scheduler groups individual requests into batches for joint GPU execution:

--enable_batching=true
--batching_parameters_file=config.json

text

With configurable batch size, timeout, and padding. This dramatically raises throughput for small-request workloads.

TFX — The Production Pipeline#

TensorFlow Serving is the last stage of TFX (TensorFlow Extended), Google’s end-to-end production ML pipeline:

Stage	Component
Data validation	Data Validation (schema, anomalies)
Transformation	Transform (feature engineering)
Training	Trainer (TF + Keras)
Evaluation	Evaluator (metrics, fairness)
Serving	TensorFlow Serving / Model Server

Alternatives#

System	Best for
TF Serving	Google-scale TF models, versioned serving
Triton Inference Server	Multi-framework GPU serving (industry standard)
TorchServe	PyTorch-native serving
ONNX Runtime	Cross-framework CPU/GPU/mobile
vLLM	LLM serving with PagedAttention

Conclusion#

TensorFlow Serving remains the most mature production model server for TensorFlow models — Google-grade versioning, batching, and canary/A-B testing with minimal latency overhead. If your stack is TensorFlow (or Keras), it’s the default choice; for heterogeneous teams, Triton is the multi-framework alternative. Combined with TFX, it gives you a complete path from training to fleet-scale serving.

TensorFlow Serving — Production Model Deployment