blog.dopana

Back

Building a chat app for 100 simultaneous users with WebSockets is straightforward: a single Node.js or Go server easily handles it. However, when the system scales to 100,000 or millions of concurrent users (CCU), a single server collapses due to TCP connection state exhaustion and I/O bottlenecks.

How can a distributed system accept a message from User A connected to Server 1 and deliver it instantaneously to User B connected to Server 2? This article breaks down the complete production architecture for scaling real-time chat.

1. Mental Model (ELI5): The Postal Service & Local Desks#

Imagine postal delivery inside a large international apartment complex:

  1. Basic Chat App (Single Server): One front desk manager in a small hall. Everyone enters and talks directly. It is instant, but if 10,000 people enter simultaneously, the single manager is overwhelmed.
  2. Distributed Chat App (Multi-Server Cluster): We open multiple service counters (WebSocket Gateways). When you drop off a letter at counter #1 for a friend at counter #2:
    • Your counter drops the letter into a high-speed conveyor belt (Kafka for durable message logs).
    • A central announcement system (Redis Pub/Sub) broadcasts: “New message for Room B!”.
    • Counter #2 (where your friend is standing) picks up the alert and delivers the letter directly into their hands via their persistent connection (WebSocket).

2. Core Scaling Challenges with WebSockets#

Unlike standard HTTP (Stateless requests where connections close immediately):

  • Stateful Connections: Every WebSocket connection maintains an open TCP socket between the client and a specific server instance.
  • Resource Limitations: Each open socket consumes File Descriptors (FDs) and RAM (~10KB to 50KB per socket). A standard machine typically saturates at 50k–100k active connections before exhausting resources.
  • Cross-Node Routing: In a distributed cluster, Client A and Client B are frequently connected to different physical machines. Server 1 needs a reliable mechanism to route packets to Server 2.

3. High-Level Distributed Architecture#

Here is the production-grade distributed architecture for a high-concurrency chat system:

Load Balancing & Gateway LayerClient LayerDurable Storage & Processing LayerDurable Storage & Processing LayerReal-Time Backplane LayerReal-Time Backplane Layer1. WSS Connection1. WSS ConnectionConsistent Hash / StickyConsistent Hash / Sticky2. Ingest message3. Consume & PersistWrite history4. Publish event5. Broadcast / Route6. Push to clientTrack sessionTrack sessionClient A (Sender)Client B (Recipient)Layer 7 Load Balancer (Nginx /HAProxy / Envoy)WebSocket Gateway #1WebSocket Gateway #2WebSocket Gateway #NRedis Cluster / PubSub /DragonflyPresence & Routing CacheApache Kafka Topic:chat-messagesChat Worker / StorageConsumerPrimary DB: PostgreSQL /ScyllaDB

4. Key Architectural Building Blocks#

4.1. WebSocket Gateway Cluster#

  • Single Responsibility: Manage raw TCP/TLS handshakes, compress/decompress payloads (JSON or Protobuf), and forward messages to the internal event buses.
  • Strip all heavy business logic out of the gateway nodes to keep them lightweight and memory-efficient.

4.2. Message Backplane: Redis Pub/Sub vs Redis Streams#

  • Redis Pub/Sub: Ultra-low latency (sub-millisecond), broadcasts incoming events across all active gateway nodes to find the node hosting the recipient.
  • Limitation: Fire-and-forget delivery. If a gateway node restarts or experiences network blips, the broadcast event is dropped. This is why we pair it with Kafka.

4.3. Message Queue & Persistence: Apache Kafka#

  • Guarantees Zero Message Loss via immutable distributed logs.
  • Guarantees message ordering per channel/room by setting Partition Keys to room_id or conversation_id.
  • Background consumer workers process messages asynchronously to persist them into primary storage (PostgreSQL, MongoDB, or ScyllaDB) without blocking real-time transit.

4.4. Presence & Session State Management#

Stored in a fast in-memory key-value store (e.g., Redis Hash):

Key: user:session:<user_id>
Value: { "server_id": "ws-node-02", "status": "online", "last_heartbeat": 1772183200 }
text

5. Practical Implementation: Gateway + Redis Pub/Sub (Node.js & TypeScript)#

Here is a runnable example of a distributed WebSocket Gateway instance:

6. OS & Infrastructure Optimizations for Millions of Connections#

When scaling to hundreds of thousands of concurrent connections per node:

  1. Increase Linux File Descriptor Limits:
    # In /etc/security/limits.conf
    * soft nofile 1048576
    * hard nofile 1048576
    bash
  2. Optimize TCP Kernel Buffers (/etc/sysctl.conf):
    fs.file-max = 2097152
    net.ipv4.tcp_max_syn_backlog = 65536
    net.core.somaxconn = 65536
    net.ipv4.tcp_rmem = 4096 87380 4194304
    net.ipv4.tcp_wmem = 4096 65536 4194304
    ini
  3. Heartbeat Protocol (Ping/Pong): Establish a 30s client-server ping/pong cycle to detect dead/ghost TCP sockets (e.g., sudden mobile disconnects) and release memory immediately.

[!TIP] Do not fetch historical chat messages over WebSockets! Use standard REST / GraphQL endpoints with cursor-based pagination for history loading. Keep WebSocket channels solely focused on real-time event streaming.

7. References#