Scaling Real-Time Chat System Architecture with WebSockets

Building a chat app for 100 simultaneous users with WebSockets is straightforward: a single Node.js or Go server easily handles it. However, when the system scales to 100,000 or millions of concurrent users (CCU), a single server collapses due to TCP connection state exhaustion and I/O bottlenecks.

How can a distributed system accept a message from User A connected to Server 1 and deliver it instantaneously to User B connected to Server 2? This article breaks down the complete production architecture for scaling real-time chat.

1. Mental Model (ELI5): The Postal Service & Local Desks#

Imagine postal delivery inside a large international apartment complex:

Basic Chat App (Single Server): One front desk manager in a small hall. Everyone enters and talks directly. It is instant, but if 10,000 people enter simultaneously, the single manager is overwhelmed.
Distributed Chat App (Multi-Server Cluster): We open multiple service counters (WebSocket Gateways). When you drop off a letter at counter #1 for a friend at counter #2:
- Your counter drops the letter into a high-speed conveyor belt (Kafka for durable message logs).
- A central announcement system (Redis Pub/Sub) broadcasts: “New message for Room B!”.
- Counter #2 (where your friend is standing) picks up the alert and delivers the letter directly into their hands via their persistent connection (WebSocket).

2. Core Scaling Challenges with WebSockets#

Unlike standard HTTP (Stateless requests where connections close immediately):

Stateful Connections: Every WebSocket connection maintains an open TCP socket between the client and a specific server instance.
Resource Limitations: Each open socket consumes File Descriptors (FDs) and RAM (~10KB to 50KB per socket). A standard machine typically saturates at 50k–100k active connections before exhausting resources.
Cross-Node Routing: In a distributed cluster, Client A and Client B are frequently connected to different physical machines. Server 1 needs a reliable mechanism to route packets to Server 2.

3. High-Level Distributed Architecture#

Here is the production-grade distributed architecture for a high-concurrency chat system:

flowchart TD
    subgraph Clients["Client Layer"]
        UserA["Client A (Sender)"]
        UserB["Client B (Recipient)"]
    end

    subgraph Edge["Load Balancing & Gateway Layer"]
        LB["Layer 7 Load Balancer (Nginx / HAProxy / Envoy)"]
        WS1["WebSocket Gateway #1"]
        WS2["WebSocket Gateway #2"]
        WSS["WebSocket Gateway #N"]
    end

    subgraph RealTime["Real-Time Backplane Layer"]
        Redis[(Redis Cluster / PubSub / Dragonfly)]
        Presence[(Presence & Routing Cache)]
    end

    subgraph EventStream["Durable Storage & Processing Layer"]
        Kafka[(Apache Kafka Topic: chat-messages)]
        Worker["Chat Worker / Storage Consumer"]
        DB[(Primary DB: PostgreSQL / ScyllaDB)]
    end

    UserA -->|1. WSS Connection| LB
    UserB -->|1. WSS Connection| LB
    LB -->|Consistent Hash / Sticky| WS1
    LB -->|Consistent Hash / Sticky| WS2

    WS1 -->|2. Ingest message| Kafka
    Kafka -->|3. Consume & Persist| Worker
    Worker -->|Write history| DB

    WS1 -->|4. Publish event| Redis
    Redis -->|5. Broadcast / Route| WS2
    WS2 -->|6. Push to client| UserB

    WS1 -.->|Track session| Presence
    WS2 -.->|Track session| Presence

4. Key Architectural Building Blocks#

4.1. WebSocket Gateway Cluster#

Single Responsibility: Manage raw TCP/TLS handshakes, compress/decompress payloads (JSON or Protobuf), and forward messages to the internal event buses.
Strip all heavy business logic out of the gateway nodes to keep them lightweight and memory-efficient.

4.2. Message Backplane: Redis Pub/Sub vs Redis Streams#

Redis Pub/Sub: Ultra-low latency (sub-millisecond), broadcasts incoming events across all active gateway nodes to find the node hosting the recipient.
Limitation: Fire-and-forget delivery. If a gateway node restarts or experiences network blips, the broadcast event is dropped. This is why we pair it with Kafka.

4.3. Message Queue & Persistence: Apache Kafka#

Guarantees Zero Message Loss via immutable distributed logs.
Guarantees message ordering per channel/room by setting Partition Keys to room_id or conversation_id.
Background consumer workers process messages asynchronously to persist them into primary storage (PostgreSQL, MongoDB, or ScyllaDB) without blocking real-time transit.

4.4. Presence & Session State Management#

Stored in a fast in-memory key-value store (e.g., Redis Hash):

Key: user:session:<user_id>
Value: { "server_id": "ws-node-02", "status": "online", "last_heartbeat": 1772183200 }

text

5. Practical Implementation: Gateway + Redis Pub/Sub (Node.js & TypeScript)#

Here is a runnable example of a distributed WebSocket Gateway instance:

ws-gateway-node.ts

import { WebSocketServer, WebSocket } from 'ws';
import Redis from 'ioredis';
import { v4 as uuidv4 } from 'uuid';

const SERVER_ID = `node-${process.env.NODE_ID || uuidv4().slice(0, 8)}`;
const PORT = Number(process.env.PORT) || 8080;

const redisPub = new Redis(process.env.REDIS_URL || 'redis://localhost:6379');
const redisSub = new Redis(process.env.REDIS_URL || 'redis://localhost:6379');

// Map of local connections held on this specific gateway instance
const localConnections = new Map<string, WebSocket>();

const wss = new WebSocketServer({ port: PORT });

wss.on('connection', (ws: WebSocket, req) => {
  const userId = new URL(req.url || '', `http://${req.headers.host}`).searchParams.get('userId');
  if (!userId) {
    ws.close(4001, 'UserId required');
    return;
  }

  localConnections.set(userId, ws);
  console.log(`[${SERVER_ID}] User ${userId} connected.`);

  // Register session presence in Redis
  redisPub.set(`presence:${userId}`, SERVER_ID, 'EX', 60);

  ws.on('message', async (data: string) => {
    try {
      const payload = JSON.parse(data.toString());
      const { recipientId, content } = payload;

      const chatEvent = {
        senderId: userId,
        recipientId,
        content,
        timestamp: Date.now(),
      };

      // Publish to distributed message backplane
      await redisPub.publish('chat:messages', JSON.stringify(chatEvent));
    } catch (err) {
      console.error('Failed to process message:', err);
    }
  });

  ws.on('close', () => {
    localConnections.delete(userId);
    redisPub.del(`presence:${userId}`);
    console.log(`[${SERVER_ID}] User ${userId} disconnected.`);
  });
});

// Subscribe to distributed chat events from other gateway nodes
redisSub.subscribe('chat:messages');
redisSub.on('message', (_channel, messageStr) => {
  const message = JSON.parse(messageStr);
  const targetWs = localConnections.get(message.recipientId);

  // If recipient is connected to this instance, push message directly
  if (targetWs && targetWs.readyState === WebSocket.OPEN) {
    targetWs.send(JSON.stringify(message));
  }
});

console.log(`WebSocket Gateway [${SERVER_ID}] running on port ${PORT}`);

typescript

6. OS & Infrastructure Optimizations for Millions of Connections#

When scaling to hundreds of thousands of concurrent connections per node:

Increase Linux File Descriptor Limits:

# In /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576

bash

Optimize TCP Kernel Buffers (/etc/sysctl.conf):

fs.file-max = 2097152
net.ipv4.tcp_max_syn_backlog = 65536
net.core.somaxconn = 65536
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 65536 4194304

ini

Heartbeat Protocol (Ping/Pong): Establish a 30s client-server ping/pong cycle to detect dead/ghost TCP sockets (e.g., sudden mobile disconnects) and release memory immediately.

[!TIP] Do not fetch historical chat messages over WebSockets! Use standard REST / GraphQL endpoints with cursor-based pagination for history loading. Keep WebSocket channels solely focused on real-time event streaming.