Redpanda: Kafka-Compatible Streaming Data Platform

A database answers “what is the current state?”, while Redpanda/Kafka answers “what events have happened, and who needs to process them?”. Redpanda is a streaming data platform compatible with Apache Kafka — rebuilt from the ground up in C++ to be faster, lighter, and simpler to operate.

History: From Vectorized to Redpanda#

Redpanda’s story begins in 2019, when Alex Gallego founded the company Vectorized. His journey is quite remarkable:

• Studied cybersecurity at NYU, then worked at FactSet (where he learned C++), then became the first engineer at YieldMo — where he grew frustrated with Apache Storm’s limitations in real-time processing
• Founded Concord Systems — a high-performance real-time processing application written in C++, acquired by Akamai in 2016
• At Akamai, he designed a storage engine that improved performance over Kafka — the foundation of what would become Redpanda

Key milestones:

[!NOTE] The Redpanda team’s design philosophy is “60 seconds to WOW” — everything from spinning up a cluster to processing real events should take no more than 60 seconds, so developers never get blocked by operational work.

The Problem Kafka Left Behind#

Kafka (open-sourced by LinkedIn in 2011) is the industry standard for streaming — but its original architecture carries real costs:

• JVM: Kafka is written in Scala/Java running on the JVM — it eats RAM, has GC pauses, and needs heavy tuning
• ZooKeeper: you must operate a separate ZooKeeper cluster just to manage metadata
• Tail latency: p99.999 (the 99.999th percentile) can degrade to seconds under high load
• Complex operations: many components, many configs, hard to optimize for your hardware

Redpanda looked at those pain points and redesigned from scratch.

Architecture: A Ground-Up Redesign#

graph TB
    P[Producer / Kafka Client] --> B["Redpanda Broker<br/>(C++ · Seastar · no JVM)"]
    B --> S["Partition<br/>(append-only log)"]
    B --> R["Raft Quorum<br/>(replaces ZooKeeper)"]
    S --> C[Consumer / Kafka Client]
    B --> T["Redpanda Console<br/>(management UI)"]

• Written in C++ on Seastar — the high-performance framework from ScyllaDB, using a thread-per-core model (one dedicated thread per CPU core, no memory shared across cores)
• No JVM, no ZooKeeper — Redpanda manages its own metadata using the Raft protocol (faster leader elections, fewer resources)
• Single binary + the rpk CLI — install, configure, and operate with far less moving parts than Kafka
• Kafka API compatible — producers, consumers, and most of the Kafka ecosystem keep working; often you only change the broker endpoint

Performance: Why 10x Faster?#

What is tail latency? (ELI5)#

Imagine p99.999 means “99.999% of requests complete under threshold X”. With 100 requests that sounds great — but with 10 million requests, that’s still 1,000 requests in the slowest “tail”. If those 1,000 requests are million-dollar trades, is that acceptable? That’s exactly why tail latency decides the quality of a streaming platform.

Benchmark results#

Redpanda published a comparison of p99.999 end-to-end latency against Kafka, using acks=all (committing to all replicas before acknowledging):

Across 9 test scenarios, Redpanda was 196% to 10,847% faster than Kafka — in one scenario Kafka took 1.8 seconds while Redpanda took 16 milliseconds.

[!TIP] This is a benchmark published by Redpanda itself — read it with a critical eye and always benchmark on your own workload. “10x” is their claim, not an absolute truth.

Why is it fast?#

• Thread-per-core + shared-nothing: each core gets one pinned thread — no locks, no context switches, exactly Seastar’s model
• Own memory management instead of the page cache: Redpanda knows exactly how much data each request reads/writes, using its own DMA-aligned cache instead of relying on the OS page cache
• Leverages modern Linux: io_uring (async I/O), O_DIRECT, DPDK (more efficient packet processing)
• Automatic kernel tuning: rpk redpanda tune all and rpk iotune benchmark your hardware and generate an optimized config — no manual guessing
• Profile-Guided Optimization (PGO) since 26.1: up to 47% lower p999 latency and 15% better CPU utilization on the same hardware
• Write caching since 24.1: up to 90% lower latency when you can trade some durability guarantees

Real-World Use Cases#

• Event streaming: passing events between services, e.g. OrderCreated, PaymentCompleted, UserLoggedIn
• Microservices: the “backbone” for asynchronous communication, reducing direct coupling between services
• Real-time analytics: collecting clickstreams, logs, metrics, transactions… feeding near-instant analysis systems
• Data pipelines: the middle layer between databases/APIs and downstream systems like data warehouses, lakes, or search engines
• IoT: ingesting large volumes of device telemetry and distributing it to processing systems
• CDC (Change Data Capture): capturing changes from PostgreSQL/MySQL and streaming them to other systems
• AI/ML streaming: feeding real-time data into processing pipelines or inference

Thanks to its low latency, Redpanda is also used in algorithmic trading, real-time gaming, SIEM, and latency-sensitive IoT systems.

Example: An E-Commerce System#

graph LR
    Order["Order Service"] -->|publish| Topic["topic: orders"]
    Topic --> Payment["Payment Service"]
    Topic --> Inventory["Inventory Service"]
    Topic --> Analytics["Analytics"]
    Analytics --> Lake["Data Lake"]

Instead of the Order Service calling each service directly, it just publishes one event:

{
  "order_id": 12345,
  "user_id": 789,
  "total": 599000,
  "status": "created"
}

Redpanda stores and distributes this event to many consumers. Payment, Inventory, Analytics… each processes independently — if one service is slow, the others are not affected.

Quick Start#

# Run Redpanda with Docker
docker run -d --name redpanda -p 9092:9092 \
  docker.redpanda.com/redpandadata/redpanda:latest \
  redpanda start

# Create a topic and produce/consume with rpk
rpk topic create orders
rpk topic produce orders
rpk topic consume orders --group my-group

Existing Kafka code keeps working as-is — just point your broker endpoint at Redpanda.

Ecosystem#

• Redpanda Console: a UI for managing clusters, inspecting data pipelines, and debugging without the command line
• Schema Registry: schema management for events (Confluent-compatible)
• Tiered storage: automatically moves older data from NVMe to cheaper object storage
• WASM transforms: run real-time data transformations with WebAssembly right inside the broker
• Redpanda Cloud / BYOC: a managed version, or bring your own cloud
• Kafka Connect compatible: reuse your existing connectors

Redpanda vs Kafka: Which to Choose?#

If you operate Kafka and struggle with ZooKeeper, JVM tuning, or tail latency — Redpanda is a “drop-in” replacement worth considering. If you need the widest ecosystem and the maturity of an industry standard, Kafka remains very strong.

Conclusion#

Redpanda proves that API compatibility doesn’t mean inheriting an old architecture. By rewriting the Kafka API in C++ on Seastar — no JVM, no ZooKeeper, thread-per-core — Redpanda delivers dramatically lower latency and simpler operations, while your existing Kafka applications keep running unchanged.

References#

• Redpanda GitHub Repository ↗
• What makes Redpanda fast? — Redpanda Blog ↗
• Redpanda Documentation ↗
• Redpanda vs Kafka vs Confluent: An Honest Comparison — Confluent ↗
• Redpanda Business Breakdown & Founding Story — Contrary Research ↗
• Redpanda Raises $100M in Series C Funding — Redpanda ↗
• Redpanda Raises $100M Series D — Redpanda ↗
• Seastar — High Performance C++ Framework ↗
• Apache Kafka Documentation ↗