blog.dopana

Back

You open ChatGPT, type a question, and words appear one by one as if someone is typing live. Where does that feeling of โ€œspeedโ€ come from?

Most people guess โ€œWebSocket must be the fastestโ€. The surprising truth: most LLM APIs today (OpenAI, Anthropic, DeepSeek) stream tokens over SSE (Server-Sent Events) โ€” a far simpler technology than WebSocket.

So when does SSE win? When does WebSocket win? And if you want to scale to 10k, 100k, or 1 million connections, which one is the real bottleneck?

This post answers from the perspective of a Software Architect specializing in realtime systems and high-performance backends.

1. Explain Like Iโ€™m 10 (ELI5)#

Imagine two ways to communicate:

  • SSE is like a school loudspeaker: only the principal (server) speaks, the whole school (clients) listens. One-way, simple, everyone hears clearly. A student who wants to reply must raise a hand and walk to a separate room (a different HTTP request).
  • WebSocket is like a two-way video call: both sides talk continuously over the same line. More flexible, but you must set up the call (handshake), hold the line, and handle drops yourself (manual reconnect).

[!NOTE] SSE = one-way (server โ†’ client) over plain HTTP. WebSocket = two-way (client โ†” server) over one persistent TCP connection.

2. How they work#

SSE: a dangling HTTP connection#

The client sends a normal GET request, the server never answers fully โ€” it keeps the connection open and pours text down gradually:

loop[every token]GET /api/chat/stream (HTTP)HTTP 200 text/event-streamdata: {"token": "hel"} data: {"token": "lo"} data: [DONE] "Client (EventSource)"Server"Client (EventSource)"Server

Strengths:

  • Passes through every proxy, CDN, and load balancer because it is just HTTP.
  • Browsers auto-reconnect and send Last-Event-ID to resume.
  • Easy to debug: open the Network tab and watch text flow.

Weaknesses:

  • Server โ†’ client only. Client โ†’ server needs a separate HTTP endpoint.
  • Text only (UTF-8); binary needs base64.

WebSocket: handshake once, talk forever#

The client upgrades from HTTP to WebSocket exactly once, then both sides exchange binary frames:

loop[bidirectional]GET /ws (Upgrade: websocket)101 Switching Protocolsframe(token)frame(cancel / approve / position)frame(token)ClientServerClientServer

Strengths:

  • Native bidirectional messaging on the same connection.
  • Frame overhead of only 2โ€“6 bytes (vs ~600 bytes of HTTP headers per request).
  • Supports both text and binary.

Weaknesses:

  • You own reconnect, heartbeat, resume, and re-auth logic.
  • Some proxies/CDNs need special config for the upgrade handshake.
  • No HTTP/2 multiplexing benefit like SSE gets.

3. Quick comparison (TL;DR)#

CriterionSSEWebSocket
Latency๐ŸŸข Very low๐ŸŸข Very low, usually a bit lower
Server โ†’ client๐ŸŸข Excellent๐ŸŸข Excellent
Client โ†’ server๐ŸŸก Needs another HTTP call๐ŸŸข Native
Protocol overhead๐ŸŸข Low๐ŸŸข Very low (2โ€“6 byte frames)
Backend CPU๐ŸŸข Usually simpler๐ŸŸก Slightly higher
Memory๐ŸŸข Good๐ŸŸข Good
10k connections๐ŸŸข Can be fine๐ŸŸข Can be fine
100k+ connections๐ŸŸก Depends on stack/proxy๐ŸŸข Usually a better fit
LLM streaming๐ŸŸข Great fit๐ŸŸข
Bidirectional realtimeโŒ๐ŸŸข Great fit

[!TIP] Rule of thumb: server pushes down only โ†’ SSE. Both sides talk continuously โ†’ WebSocket.

4. Case 1: one-way streaming (LLM chat)#

The server sends:

token 1
token 2
token 3
token 4
...
text

With SSE:

Server โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€> Client
       stream data continuously
text

Very efficient. A Node.js server takes a few lines:

The browser client needs no library:

client-sse.js
const es = new EventSource('/api/chat/stream');
es.onmessage = (e) => {
  if (e.data === '[DONE]') return es.close();
  appendToChat(JSON.parse(e.data).token);
};
javascript

WebSocket can do it too:

WebSocket frame
WebSocket frame
WebSocket frame
...
text

Also very fast โ€” but you never use its bidirectional advantage. Like buying a flagship phone just to listen to the radio.

5. Case 2: tiny, frequent, bidirectional messages#

Think games, cursor positions, collaborative editing:

Client โ†’ position
Server โ†’ position
Client โ†’ position
Server โ†’ position
...
text

Say 100โ€“1000 messages/second/client.

WebSocket is usually the better fit here, because the connection is designed for continuous two-way traffic with low frame overhead โ€” no new HTTP request per upstream message.

Each HTTP request carries ~600 bytes of headers. A WebSocket frame adds 2โ€“6 bytes. Multiply by 100 msg/s ร— 10k clients and the bandwidth gap is very real.

client-ws.js
// WebSocket: one connection, send and receive freely
const ws = new WebSocket('wss://example.com/agent');
ws.onmessage = (e) => renderAgentEvent(JSON.parse(e.data));
// Cancel mid-stream, approve a tool call, send position โ€” same wire
ws.send(JSON.stringify({ type: 'cancel' }));
ws.send(JSON.stringify({ type: 'approve-tool-call', id: 'tool-123' }));
javascript

6. The important caveat: donโ€™t expect 10x#

Donโ€™t expect:

WebSocket is 10x faster than SSE.

In real applications, latency is dominated by:

LLM generation
    โ†“
database
    โ†“
business logic
    โ†“
network
    โ†“
browser rendering
text

not by SSE vs WebSocket themselves.

If the LLM takes 500ms to produce the first token, WebSocket beating SSE by a few ms is irrelevant. Localhost benchmarks show ~18ms average WebSocket latency at 1000 connections โ€” slightly better โ€” but for pure server โ†’ client broadcast to many passive clients, SSE is often leaner thanks to plain HTTP semantics.

[!WARNING] Optimizing the wrong layer is a classic disease: switching protocols without measuring LLM TTFT (time-to-first-token), DB queries, or render cost rarely improves UX.

7. For your AI project: what to choose?#

If the goal is AI chat / LLM streaming, I would choose:

POST /api/chat
       โ–ผ
     Server
       โ–ผ
SSE /api/chat/stream
       โ”œโ”€token
       โ”œโ”€token
       โ”œโ”€token
       โ”œโ”€token
       โ””โ”€done
text

Over WebSocket. Reason: simpler, passes through any infrastructure, auto-reconnects, fast enough.

But if you later build:

AI Agent
   โ†•
Browser
   โ†•
Tool execution
   โ†•
Terminal
   โ†•
Agent events
text

with many bidirectional realtime events (cancel generation, approve tool calls, mid-stream steering, multi-device sync), WebSocket starts making much more sense.

Decision checklist:

  • Stream tokens down only? โ†’ SSE.
  • Need cancel/steer/approve from the client mid-stream? โ†’ WebSocket (or SSE + extra HTTP POSTs, which gets messy as interactions grow).
  • Need binary (realtime audio/video)? โ†’ WebSocket (or WebRTC).
  • Behind old CDNs/proxies? โ†’ SSE is safer.

8. Scaling to 10k / 100k / 1M connections#

If your goal is 10k / 100k / 1M concurrent connections, the story changes a lot โ€” SSE vs WebSocket is only a small part. What matters more:

  • Connection architecture: event loop (Node.js, Go, Elixir), sticky sessions, sharding.
  • Proxy/load balancer: can Nginx, ALB, Cloudflare hold keep-alive + upgrade?
  • Kernel/socket limits: file descriptors, TCP tuning, TIME_WAIT.
  • Backpressure: where do you buffer when the server is faster than the client?
  • Resume/reconnect strategy when mass drops happen.

In short:

SSE = very good performance + simpler for one-way streaming. WebSocket = better performance for high-frequency bidirectional realtime.

References#