Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Real-Time Systems — Protocols, Architecture, and Production Reality

​

Real-World Stories: Why This Matters

​

1. Real-Time Communication Protocols

​

1.1 WebSocket — Full-Duplex Persistent Connection

Client → Server: GET /chat HTTP/1.1
                 Upgrade: websocket
                 Connection: Upgrade
                 Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==

Server → Client: HTTP/1.1 101 Switching Protocols
                 Upgrade: websocket
                 Connection: Upgrade
                 Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

[TCP connection is now a WebSocket — bidirectional frames flow freely]

• Full-duplex: Both sides send and receive simultaneously
• Persistent: Connection stays open until explicitly closed
• Low overhead: After handshake, frame headers are 2-14 bytes (vs HTTP headers of 200-800+ bytes per request)
• Protocol: ws:// (unencrypted) and wss:// (TLS-encrypted, always use this in production)
• Binary and text: Supports both UTF-8 text frames and binary frames

​

1.2 Server-Sent Events (SSE) — One-Way Server Push

Client → Server: GET /events HTTP/1.1
                 Accept: text/event-stream

Server → Client: HTTP/1.1 200 OK
                 Content-Type: text/event-stream

                 data: {"score": "3-1", "minute": 67}

                 event: goal
                 data: {"team": "home", "scorer": "Martinez"}

                 id: 1042
                 data: {"score": "4-1", "minute": 72}

• One-way: Server to client only (client sends data via regular HTTP requests)
• Auto-reconnect: The browser’s EventSource API automatically reconnects if the connection drops
• Event IDs: The id: field enables resumption — on reconnect, the browser sends Last-Event-ID so the server can replay missed events
• Text only: UTF-8 text (no binary), but you can Base64-encode or use JSON
• HTTP/2 multiplexing: SSE connections can share an HTTP/2 connection, eliminating the browser’s 6-connection-per-domain limit

​

1.3 Long Polling — The Reliable Fallback

Client → Server: GET /messages?since=1042    [held open...]
                                              [30 seconds pass, no new data]
Server → Client: 204 No Content

Client → Server: GET /messages?since=1042    [held open...]
                                              [new message arrives after 3s]
Server → Client: 200 OK
                 [{"id": 1043, "text": "hello"}]

Client → Server: GET /messages?since=1043    [cycle continues]

• Corporate environments where WebSocket connections are blocked by proxies or firewalls
• Serverless architectures (AWS Lambda) where persistent connections are not possible
• Low-frequency updates where the overhead of a persistent connection is not justified
• As a fallback when WebSocket connections fail (Socket.IO does this automatically)

​

1.4 WebRTC — Peer-to-Peer Real-Time Communication

1. Signaling: Peers exchange session descriptions (SDP — Session Description Protocol) through a signaling server (usually WebSocket). SDP describes codecs, network candidates, and encryption keys.
2. ICE Gathering: Each peer discovers its network addresses using STUN servers (Simple Traversal of UDP through NAT). STUN tells you your public IP and port.
3. Connectivity Check: Peers try to connect directly using discovered addresses. If direct connection fails (symmetric NAT, corporate firewall), traffic is relayed through a TURN server (Traversal Using Relays around NAT).
4. DTLS Handshake: Once connected, peers establish encrypted channels using DTLS (Datagram TLS).
5. Media/Data Flow: Audio and video flow over SRTP (Secure RTP). Arbitrary data flows over SCTP data channels.

Peer A                    Signaling Server                    Peer B
  |--- SDP Offer ----------->|                                  |
  |                           |--- SDP Offer ------------------>|
  |                           |<-- SDP Answer ------------------|
  |<-- SDP Answer ------------|                                  |
  |                                                              |
  |--- ICE Candidates ------->|--- ICE Candidates ------------->|
  |<-- ICE Candidates --------|<-- ICE Candidates --------------|
  |                                                              |
  |<=============== Direct P2P Connection (DTLS/SRTP) =========>|

​

TURN Server Cost: Self-Hosted vs Managed Services

• ICE candidate ordering: Configure ICE to always try direct (host) and STUN (server-reflexive) candidates before TURN (relay). This is the default, but verify your configuration is not inadvertently preferring relay candidates.
• TURN-only-when-needed: Use TURN as a fallback, never as default. Some misconfigured clients always use TURN, turning a 15% cost into a 100% cost.
• Bandwidth-adaptive streams: When on TURN, automatically downgrade video quality (720p to 480p or audio-only) to reduce relay bandwidth.
• Regional TURN placement: Place TURN servers in regions where your users are. A TURN server in US-East relaying between two users in Tokyo adds unnecessary latency and cross-region bandwidth cost.
• Connection quality monitoring: Track what percentage of your connections use TURN. If it is significantly above 20%, investigate whether your STUN configuration is correct or if a specific network environment (corporate VPN, specific ISP) is forcing unnecessary relay.

​

1.5 Protocol Comparison

​

1.6 Decision Guide: Which Protocol for Which Use Case

​

2. WebSocket Architecture at Scale

​

2.1 Connection Lifecycle

1. HTTP Handshake (Upgrade request)
      ↓
2. Connection Open (onopen fires)
      ↓
3. Message Exchange (frames: text, binary, ping, pong)
      ↓
4. Heartbeat (ping/pong every 30-60s to detect dead connections)
      ↓
5. Close (clean: close frame exchange; unclean: TCP reset/timeout)

​

2.2 Connection Limits and Capacity

​

2.3 Scaling Horizontally

                    ┌──────────────┐
                    │  Redis Pub/Sub│
                    │  or Kafka     │
                    └──┬───┬───┬───┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ▼            ▼            ▼
        ┌─────────┐  ┌─────────┐  ┌─────────┐
        │  WS      │  │  WS      │  │  WS      │
        │  Server 1│  │  Server 2│  │  Server 3│
        └─────────┘  └─────────┘  └─────────┘
          │  │  │      │  │  │      │  │  │
         Users A-D   Users E-H   Users I-L

​

2.4 Connection State Management

• In-memory on each server: Fast, but lost on restart. Other servers cannot look up who is connected where.
• Redis hash: HSET connections:user123 server ws-server-7 channels ["general","random"]. Fast lookups, shared across servers, but adds a Redis dependency to every connect/disconnect event.
• Distributed hash table (e.g., Hazelcast, etcd): For very large deployments where a single Redis instance becomes a bottleneck.

​

2.5 Load Balancing WebSockets

​

2.6 Reconnection Strategies

function reconnect(attempt) {
  // Base delay: 1s, 2s, 4s, 8s... capped at 30s
  const baseDelay = Math.min(1000 * Math.pow(2, attempt), 30000);

  // Add random jitter (0-100% of base delay) to prevent thundering herd
  const jitter = Math.random() * baseDelay;
  const delay = baseDelay + jitter;

  setTimeout(() => {
    const ws = new WebSocket('wss://example.com/ws');
    ws.onopen = () => {
      attempt = 0; // Reset on successful connection
      // Send last-received message ID to recover missed messages
      ws.send(JSON.stringify({ type: 'resume', lastMessageId: lastId }));
    };
    ws.onclose = () => reconnect(attempt + 1);
  }, delay);
}

​

2.7 Connection Recovery Patterns

// Client-side connection recovery with gap detection
class RealtimeClient {
  constructor(url) {
    this.url = url;
    this.lastSeqByChannel = {};  // Track last received sequence per channel
    this.pendingOutbound = [];    // Buffer outbound messages during disconnect
    this.isConnected = false;
  }

  onReconnect(ws) {
    this.isConnected = true;

    // Step 1: Request missed messages for all active channels
    for (const [channel, lastSeq] of Object.entries(this.lastSeqByChannel)) {
      ws.send(JSON.stringify({
        type: 'resume',
        channel,
        lastSeq
      }));
    }

    // Step 2: Flush buffered outbound messages
    while (this.pendingOutbound.length > 0) {
      const msg = this.pendingOutbound.shift();
      ws.send(JSON.stringify(msg));
    }
  }

  send(message) {
    if (this.isConnected) {
      this.ws.send(JSON.stringify(message));
    } else {
      // Buffer during disconnect -- cap at 100 messages to prevent memory issues
      if (this.pendingOutbound.length < 100) {
        this.pendingOutbound.push(message);
      }
    }
  }

  onMessage(msg) {
    // Track sequence for gap detection
    if (msg.seq && msg.channel) {
      const expectedSeq = (this.lastSeqByChannel[msg.channel] || 0) + 1;
      if (msg.seq > expectedSeq) {
        // Gap detected -- request missing messages
        this.requestGapFill(msg.channel, expectedSeq, msg.seq - 1);
      }
      this.lastSeqByChannel[msg.channel] = msg.seq;
    }
  }
}

• Outbound message queue: Buffer messages the user sends while disconnected (chat messages, edits, reactions). On reconnect, flush the queue with idempotency keys so the server can deduplicate. Cap the buffer size (100-500 messages) to prevent memory exhaustion on mobile devices.
• Optimistic UI with reconciliation: Show sent messages immediately in the UI with a “pending” indicator (gray checkmark, spinner). On reconnect, confirm delivery and update the indicator, or show a “failed to send” state if the server rejects after replay.
• Local state snapshot: Before transitioning to “disconnected” state, snapshot the current application state (open channels, scroll positions, draft messages). On reconnect, restore this state so the user returns to exactly where they were.

​

2.8 Graceful Shutdown During Deploys

1. Stop accepting new connections on the old server instance (remove from load balancer)
2. Send a “reconnect” control message to all connected clients: {"type": "reconnect", "reason": "server_maintenance"}
3. Wait for clients to disconnect (with a timeout, e.g., 30 seconds)
4. Force-close remaining connections after the timeout
5. New server instance is already accepting connections — clients reconnect to it

​

3. Building Chat Systems

​

3.1 Architecture Overview

┌─────────────────────────────────────────────────┐
│           Client (Browser / Mobile App)          │
│  - WebSocket connection to edge server           │
│  - Local message cache for offline access        │
│  - Optimistic UI (show message before server ack)│
└──────────────────────┬──────────────────────────┘
                       │ WebSocket (wss://)
┌──────────────────────▼──────────────────────────┐
│           Real-Time Layer                        │
│  - WebSocket servers (connection termination)    │
│  - Pub/sub backbone (Redis, Kafka, NATS)         │
│  - Presence service (who is online)              │
│  - Typing indicator service (fire-and-forget)    │
└──────────────────────┬──────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────┐
│           Persistence Layer                      │
│  - Message storage (Cassandra, ScyllaDB, DynamoDB│
│  - Message search (Elasticsearch)                │
│  - User/channel metadata (PostgreSQL)            │
│  - Media storage (S3 + CDN)                      │
└─────────────────────────────────────────────────┘

​

3.2 Message Ordering

| 41 bits: timestamp (ms since epoch) | 10 bits: worker ID | 13 bits: sequence |
|  ~69 years of milliseconds          | 1024 workers       | 8192 per ms/worker|

• Timestamp bits provide rough ordering without centralized coordination
• Worker ID ensures uniqueness across servers
• Sequence number handles multiple messages within the same millisecond on the same worker

​

3.3 Delivery Guarantees

1. Client sends message with a client-generated UUID (client_msg_id)
2. Server persists message, assigns Snowflake ID, returns ACK with the server ID
3. If client does not receive ACK within timeout, it resends with the same client_msg_id
4. Server checks client_msg_id for deduplication before persisting
5. Other clients receiving the message use the Snowflake ID for dedup on the display side

​

3.4 Read Receipts and Typing Indicators

​

3.5 Group Chat Fan-Out

​

3.6 Offline Messages and Push Notifications

1. Message arrives at the server for a conversation
2. Check connection registry — is the recipient connected?
3. If yes: deliver via WebSocket
4. If no: store in the “undelivered messages” queue for that user, and optionally trigger a push notification (APNs for iOS, FCM for Android)
5. On reconnect: client sends lastMessageId, server replays all messages after that ID

​

3.7 End-to-End Encryption (E2EE) Basics

• Each user generates a long-term identity key pair and ephemeral prekeys
• A “double ratchet” algorithm derives new encryption keys for every message, providing forward secrecy (compromising a key does not decrypt past messages) and future secrecy (compromising a key does not decrypt future messages)
• For group chats, the sender encrypts the message key once per group member using each member’s public key (Sender Keys optimization reduces this cost)

​

4. Live Feeds and Notifications

​

4.1 Presence Systems: Who Is Online?

1. Each WebSocket connection sends heartbeats to the presence service
2. If no heartbeat is received within the timeout (e.g., 30 seconds), the user transitions to “offline”
3. The presence service maintains a per-user state machine: online → idle (5 min no activity) → offline (connection lost)
4. Presence changes are published via pub/sub to all users who have subscribed to that user’s presence (typically friends, channel co-members)

• Do not broadcast globally. A user’s presence change only matters to their contacts, not all users.
• Batch presence updates. Instead of sending a separate message for each online/offline transition, batch them into periodic snapshots (e.g., every 5 seconds: “here are all the presence changes in your contact list since the last batch”).
• Stale presence is acceptable. Showing a user as “online” for 30 seconds after they disconnect is fine. Showing them as “offline” when they are actually typing is not. Bias toward showing “online” longer.

​

4.2 Notification Delivery: In-App, Push, Email

​

4.3 SSE for Dashboards and Monitoring

// Client
const events = new EventSource('/api/metrics/stream');

events.addEventListener('cpu', (e) => {
  updateCPUChart(JSON.parse(e.data));
});

events.addEventListener('memory', (e) => {
  updateMemoryChart(JSON.parse(e.data));
});

events.onerror = () => {
  // EventSource auto-reconnects with Last-Event-ID
  console.log('Connection lost, auto-reconnecting...');
};

# Server (Python/FastAPI)
async def metrics_stream(request: Request):
    async def event_generator():
        last_id = request.headers.get('Last-Event-ID', '0')
        while True:
            metrics = await get_latest_metrics(since=last_id)
            for metric in metrics:
                yield f"id: {metric.id}\nevent: {metric.type}\ndata: {json.dumps(metric.data)}\n\n"
                last_id = metric.id
            await asyncio.sleep(1)
    return StreamingResponse(event_generator(), media_type="text/event-stream")

​

5. Collaborative Editing

​

5.1 Operational Transformation (OT) — How Google Docs Works

Document state: "ABCD"

User 1: Insert 'X' at position 1 → "AXBCD"
User 2: Delete character at position 2 → "ABD"

These operations were made concurrently (neither user saw the other's edit).

If we apply User 1's insert first:
  "ABCD" → Insert X at 1 → "AXBCD" → Delete at position 2... but position 2 is now 'B' not 'C'!

OT transforms User 2's operation against User 1's:
  User 1 inserted before position 2, so User 2's position shifts: delete at position 3 instead.
  "AXBCD" → Delete at 3 → "AXBD" ✓

Conversely, if we apply User 2's delete first:
  "ABCD" → Delete at 2 → "ABD" → Insert X at 1 → "AXBD" ✓

Both orderings produce the same result. That is the OT guarantee.

​

5.2 CRDTs — Conflict-Free Replicated Data Types

• Yjs — The most widely used CRDT library for collaborative editing. Used by Notion, JupyterLab, and others.
• Automerge — Designed by Martin Kleppmann (author of “Designing Data-Intensive Applications”). More academically rigorous, with a focus on local-first software.

Character 'H' (id: A1) → 'e' (id: A2) → 'l' (id: A3) → 'l' (id: A4) → 'o' (id: A5)

User A inserts 'X' after A2: 'H' → 'e' → 'X' (id: A6) → 'l' → 'l' → 'o'
User B inserts 'Y' after A2: 'H' → 'e' → 'Y' (id: B1) → 'l' → 'l' → 'o'

Merge: Both insertions are after A2. A deterministic tiebreaker (e.g., compare IDs)
  decides the order: 'H' → 'e' → 'X' → 'Y' → 'l' → 'l' → 'o' (or Y before X)
  Both users converge to the same result without transformation.

​

5.3 OT vs CRDT: Trade-Offs

​

5.4 Cursor and Selection Synchronization

• Each client broadcasts cursor position and selection range via WebSocket (fire-and-forget, like typing indicators)
• Other clients render colored cursors and selection highlights with the user’s name
• Challenge: Cursor positions are expressed as document offsets, which change as other users insert or delete text. You must transform cursor positions against incoming operations — a simpler version of the OT problem

​

6. Real-Time at Scale — Production Concerns

​

6.1 Capacity Planning

• 1M concurrent connections at 40KB each = 40 GB RAM just for connection state
• That is 10-20 servers with 4-8 GB allocated for connections (leave headroom for message processing)
• Each server handles 50K-100K connections
• Message throughput: budget 1-5 ms CPU per message for routing, serialization, and pub/sub publish

​

6.2 Backpressure: When Clients Cannot Keep Up

• Per-connection send buffer limit: If the buffer exceeds a threshold (e.g., 1 MB), drop low-priority messages (presence updates, typing indicators) first, then older messages
• Client-side flow control: The client sends an acknowledgment for batches of messages. If acks slow down, the server throttles sending
• Channel downgrade: For users on slow connections, switch from pushing every message to pushing a “N new messages” notification and let the client pull on demand

​

6.3 Message Ordering Guarantees

​

6.4 Geographic Distribution

• WebSocket edge servers in each major region (US-East, US-West, EU, Asia)
• Users connect to the nearest edge via GeoDNS or Anycast
• Edge servers relay messages through a global pub/sub backbone (Kafka, NATS, or a custom mesh)
• Message persistence happens in a primary region, with edge servers providing low-latency delivery

​

6.5 Monitoring Real-Time Systems

​

6.6 Security

​

6.7 Real-Time Testing Strategies

​

Unit and Integration Testing

// Example: WebSocket integration test with k6
// k6 has native WebSocket support for load testing
import ws from 'k6/ws';
import { check } from 'k6';

export default function () {
  const url = 'wss://staging.example.com/ws?token=test-jwt';
  const res = ws.connect(url, {}, function (socket) {
    socket.on('open', () => {
      // Subscribe to a test channel
      socket.send(JSON.stringify({ type: 'subscribe', channel: 'load-test' }));
    });

    socket.on('message', (data) => {
      const msg = JSON.parse(data);
      check(msg, {
        'has sequence number': (m) => m.seq !== undefined,
        'sequence is increasing': (m) => m.seq > lastSeq,
        'delivery under 500ms': (m) => Date.now() - m.serverTimestamp < 500,
      });
    });

    // Keep connection alive for test duration
    socket.setTimeout(() => socket.close(), 60000);
  });

  check(res, { 'status is 101': (r) => r && r.status === 101 });
}

​

Load Testing

• Connection establishment rate: How many connections per second can the server accept? This tests the handshake path, authentication, and subscription setup.
• Message delivery latency (p50, p95, p99) at load: Latency under 100 connections is meaningless. Test at 50%, 80%, and 100% of expected peak concurrent connections.
• Message throughput ceiling: At what message rate does delivery latency start degrading? This is your system’s saturation point.
• Memory growth over time: Run a 4-hour soak test and plot memory usage. WebSocket servers that leak memory per-connection or per-message will show a slow, steady climb.
• Reconnection storm recovery: Simultaneously disconnect 10% of connections and measure how long until all clients are reconnected and caught up on missed messages.

# Example: Artillery WebSocket load test scenario
config:
  target: "wss://staging.example.com"
  phases:
    - duration: 60
      arrivalRate: 100     # 100 new connections/sec for 60 seconds = 6,000 connections
    - duration: 300
      arrivalRate: 0       # Hold connections for 5 minutes (soak phase)
  ws:
    subprotocol: "chat-v1"

scenarios:
  - engine: ws
    flow:
      - send: '{"type":"auth","token":"{{$randomString()}}"}'
      - think: 1
      - send: '{"type":"subscribe","channel":"load-test"}'
      - think: 5
      - loop:
          - send: '{"type":"message","text":"hello from Artillery"}'
          - think: 2
        count: 50

​

Chaos Testing for Connections

1. Kill the WebSocket server mid-stream. kill -9 the process while clients are connected. Verify all clients detect the disconnection within your heartbeat timeout (30-60 seconds), reconnect with jitter, and resume without message loss.
2. Introduce asymmetric packet loss. Use tc (Linux traffic control) or Toxiproxy to add 10% packet loss on the server-to-client path only. Verify clients detect degradation and the server’s backpressure mechanisms activate before send buffers overflow.
3. Partition the pub/sub backbone. Temporarily block traffic between WebSocket servers and Redis/Kafka. Verify that messages sent during the partition are delivered after recovery (if using durable pub/sub like Kafka) and that non-durable messages (typing indicators) are simply dropped without cascading failure.
4. Exhaust file descriptors. Set ulimit -n to a low value (e.g., 1024) on a test server and push connections past that limit. Verify the server rejects new connections gracefully with a proper error instead of crashing or hanging.
5. Simulate mobile network transitions. Rapidly cycle a test client through connected-disconnected-connected states (simulating WiFi to cellular handoff). Verify the reconnection and gap-fill path handles rapid state changes without duplicate subscriptions or orphaned server-side state.

• Toxiproxy (by Shopify): A TCP proxy that simulates network conditions — latency, bandwidth limits, connection resets, timeouts. Run it between your test clients and WebSocket servers.
• Chaos Monkey / Litmus (for Kubernetes): Randomly terminate WebSocket server pods to test connection migration and reconnection.
• tc (traffic control): Linux kernel tool for injecting latency, packet loss, and bandwidth limits at the network interface level.

​

Interview Questions

​

Real-World Architecture Examples

​

Cross-Chapter Connections

​

Recommended Resources

​

Quick Reference Cheatsheet

​

Interview Deep-Dive Questions

​

Q1: You inherit a WebSocket service that works fine at 10K connections but falls over at 50K. Walk me through your investigation.

• File descriptors first. This is the most frequent cause and the easiest to verify. ulimit -n on most Linux distros defaults to 1024. At 50K connections, you need at least 50K file descriptors plus headroom for log files, database connections, and the pub/sub client. I would check cat /proc/<pid>/limits for the running process and compare Max open files against the actual count from ls /proc/<pid>/fd | wc -l. If the soft limit is anywhere near 50K, that is the issue. Fix: set ulimit -n 200000 in the service unit file, and also raise fs.file-max at the kernel level if it is low.
• Memory next. Each WebSocket connection consumes 20-50KB between TCP socket buffers, TLS session state, and application-level data. At 50K connections and 40KB each, that is 2GB just for connection state. If the server has 4GB total and your application plus the JVM or Node.js runtime already uses 1.5GB, you are pushing into swap or hitting OOM. I would check memory consumption patterns with pmap or the runtime’s heap profiler, looking specifically for per-connection allocations that are larger than expected — maybe someone stored the full user object instead of just the user ID on each connection.
• Ephemeral port exhaustion. If the WebSocket server itself makes outbound connections — to a Redis pub/sub, a database, or an upstream service — each outbound connection consumes an ephemeral port. The default range is only ~28K ports. If the server is creating and tearing down outbound connections frequently without reusing them (connection pool misconfiguration), you can hit this limit. Check with ss -s and look at the TIME_WAIT count.
• CPU saturation from message fan-out. If each incoming message triggers a broadcast to all 50K connections, the serialization and write syscall overhead per message scales linearly. At 50K connections with 100 messages per second, you are doing 5 million serialization-and-send operations per second. Profile the CPU: is it spending time in JSON serialization, in TLS encryption, or in kernel syscalls for socket writes?
• The load balancer or proxy in front. Many reverse proxies have their own connection limits that are lower than the backend server’s capacity. Nginx’s worker_connections default is 1024. An ALB has a default of 100 concurrent connections per target in some configurations. The server might be fine; the proxy might be the bottleneck.

​

Follow-up: You discover the cause is memory — each connection holds 200KB instead of the expected 40KB. How do you find the leak?

• Heap snapshot diff. In Node.js, I would take a heap snapshot at 10K connections and another at 20K connections, then compare them in Chrome DevTools. I am looking for objects whose count grew proportionally with the connection count. Often the culprit is a retained reference — an event listener that was added on connection open but never removed on close, or a message history buffer that was supposed to be capped but has no size limit.
• Common culprits I have seen in production. (1) Each connection subscribes to a pub/sub channel, but the subscription callback closure captures a reference to the entire request context including large headers and the HTTP body from the upgrade request. (2) A message broadcast function that creates a per-connection copy of the message payload instead of serializing once and sending the same buffer to all connections. (3) TLS session caching gone wrong — the server is caching TLS session tickets per-connection instead of per-client.
• The fix pattern. Once I identify the oversized object, I either eliminate it (why is the full HTTP upgrade request retained after the connection is established?), cap it (ring buffer instead of unbounded array), or share it (serialize the message once, send the same Buffer to all connections instead of serializing per-connection).

​

Follow-up: The system is in production and crashing right now. You cannot afford a heap snapshot. What do you do?

• Immediate mitigation: Reduce the connection limit per server by pulling some instances out of the load balancer rotation, spreading the load. If I have auto-scaling, scale out horizontally to reduce per-instance connection count below the threshold where the problem manifests.
• Non-invasive data collection: Check /proc/<pid>/status for VmRSS (resident set size) to confirm memory is the issue. Use jemalloc’s stats endpoint if the runtime supports it. For Node.js, process.memoryUsage() exposed via a health endpoint gives heap used vs heap total without pausing the process.
• If I suspect a specific code path: Add a lightweight gauge metric that tracks the size of the data structure I suspect (connection map size, subscription count, buffer sizes) and emit it to the monitoring system. This costs almost nothing and gives me the trend data I need.
• Last resort: Enable core dumps, let the next OOM kill generate a dump, and analyze offline with gdb or lldb while fresh instances handle the traffic.

​

Going Deeper: How would you design the connection state management from scratch to prevent this class of problem?

• Define a ConnectionContext struct or class with a fixed set of fields: user ID, server ID, subscribed channels (set with max cardinality), last heartbeat timestamp, and a capped send buffer (ring buffer, max 1MB). No extensible maps or “metadata” bags. If you need to add a field, it is a code change with a review.
• Implement a per-connection memory accounting system. When a connection is created, increment a gauge. When it is destroyed, decrement. Alert if the per-connection memory (total RSS / active connections) drifts above the budget. This catches regressions before they become outages.
• Separate the hot path (message routing) from the cold path (user profile lookup, channel metadata). The connection context should hold only what is needed to route a message. Everything else is looked up on demand from a shared cache.

​

Q2: Explain the trade-offs between Redis Pub/Sub and Kafka as the backbone for cross-server WebSocket message routing.

• Redis Pub/Sub is fire-and-forget. When a message is published to a channel, Redis delivers it to all currently subscribed clients. If a WebSocket server is temporarily disconnected from Redis — during a network blip, a Redis failover, or during the server’s own restart — every message published during that window is gone. There is no replay, no offset, no “give me what I missed.” This makes Redis Pub/Sub ideal for ephemeral events: typing indicators, presence updates, cursor positions in a collaborative editor. These are events where the latest state supersedes any missed intermediate states. Redis Pub/Sub is also extremely fast — sub-millisecond publish-to-subscribe latency within the same datacenter — and operationally simple. A single Redis instance can handle hundreds of thousands of publishes per second.
• Kafka provides durable, ordered, replayable message streams. Each message is written to a partition log and retained for a configurable duration (hours, days, or indefinitely). Consumers track their offset — the position in the log they have read up to. If a WebSocket server restarts, it resumes from its last committed offset and replays everything it missed. This makes Kafka necessary for durable messages: chat messages, notifications, bid updates in an auction — anything where losing a message means data loss visible to the user. The trade-off is higher latency (typically 5-20ms end-to-end vs sub-1ms for Redis), more operational complexity (Zookeeper or KRaft, partition management, consumer group coordination), and higher resource consumption.
• The hybrid approach, which is what I have seen work best in production: Use Redis Pub/Sub as the primary real-time delivery path for all messages (including durable ones). Simultaneously, write durable messages to Kafka. If a WebSocket server detects it missed messages (via sequence number gap detection on the client), it fetches the missing messages from Kafka’s log. This gives you Redis’s speed for the happy path and Kafka’s durability for recovery. The critical insight: you are not choosing one or the other. You are using each for what it is good at.

​

Follow-up: Redis Pub/Sub has no backpressure mechanism. What happens when a subscriber cannot keep up, and how do you handle it?

• Monitor the Redis subscriber output buffer size. Alert if any subscriber’s buffer exceeds 50% of the configured limit. INFO CLIENTS in Redis shows per-client buffer usage. This gives you early warning.
• Set aggressive output buffer limits with a hard cap. Configure client-output-buffer-limit pubsub 256mb 64mb 60 — meaning: hard limit 256MB (disconnect immediately), soft limit 64MB sustained for 60 seconds (disconnect if above 64MB for over a minute). These prevent a single slow subscriber from exhausting Redis’s memory.
• Implement subscriber health checks. Each WebSocket server should monitor the age of the last message it received from Redis. If no message arrives for a threshold period (and messages are expected), proactively reconnect the subscription. Do not wait for Redis to kill it.
• Decouple subscription from processing. Instead of processing messages synchronously in the Redis subscription callback, push them to a local in-process queue (bounded, with drop-oldest policy for non-durable messages) and process from the queue. This isolates the subscription speed from the processing speed.

​

Follow-up: You chose the hybrid Redis + Kafka approach. How do you ensure a message is not displayed twice on the client — once from Redis real-time delivery and once from Kafka replay?

​

Q3: A product manager asks you to add “who is typing” to a chat product that has 200K concurrent users. How do you design it, and what do you explicitly not do?

• Client emits a typing_start event when the user begins typing in a conversation. This is sent over the existing WebSocket connection as a lightweight message: {"type": "typing", "conversation_id": "abc", "user_id": "123"}. The client throttles emissions to at most one every 3 seconds to prevent flooding.
• Server broadcasts the event to other participants in the conversation via the existing pub/sub backbone (Redis Pub/Sub). No Kafka, no persistence, no acknowledgment. Fire-and-forget.
• Receiving clients display the indicator for 4-5 seconds, then auto-expire it. If another typing_start arrives from the same user, reset the timer. No typing_stop event is strictly necessary — the auto-expiry handles it. I would send a typing_stop when the user submits the message (so the indicator disappears immediately rather than lingering for 4 seconds), but if it is lost, no harm done.

• No persistence. Typing state is never written to a database. It is ephemeral by nature. If a user disconnects and reconnects, they do not need to know who was typing 10 seconds ago.
• No delivery guarantees. No ACKs, no retries, no sequence numbers on typing events. If Redis drops a typing event during a blip, nobody notices.
• No fan-out to offline users. If a conversation participant is offline, do not queue the typing indicator for them. Do not send push notifications. This sounds obvious, but I have seen systems where typing events flowed through the same delivery pipeline as messages, including the “queue for offline delivery” path, which wasted storage and push notification budget.
• No per-user tracking on the server. The server does not maintain a data structure of “who is currently typing in which conversation.” It simply relays the event. The client is responsible for tracking the state and auto-expiring stale indicators. This keeps the server stateless with respect to typing.

​

Follow-up: The product manager now wants “Alice is typing…” to show the user’s name. How does this change your design if you have 50 participants in a channel?

​

Follow-up: How do you prevent a malicious client from sending 1,000 typing events per second to flood the system?

​

Q4: Compare OT and CRDTs for collaborative editing. When would you choose each, and what is the hardest production problem you would face with each?

• Server-mediated architecture where all clients connect to a central server (the Google Docs model). OT is simpler when you already have a server ordering operations.
• Rich document formats with many operation types (text, tables, images, formatting). OT’s N-squared transformation matrix is painful but at least each transformation function can be hand-tuned. CRDT metadata overhead grows with document complexity.
• Undo/redo is a hard requirement. OT has a well-understood inverse operation model. CRDT undo requires tracking causal history, which is significantly more complex.

• Local-first or offline-first applications where users may edit for hours without connectivity and then sync. CRDTs were designed for exactly this. OT requires server round-trips for conflict resolution.
• Peer-to-peer architectures without a central server. Think a collaborative tool that syncs directly between devices.
• When I want to use an existing library (Yjs) rather than building conflict resolution from scratch. Yjs has been battle-tested and the ergonomics are excellent.

​

Follow-up: You are building a Notion-like tool. The document is not just text — it is a tree of blocks (paragraphs, headings, lists, embeds). How does this affect your choice?

​

Follow-up: How do you test that your collaborative editing system actually converges correctly?

• Fuzz testing with random operations. Generate thousands of random document operations (insert, delete, format, move) across N simulated clients. Apply them in every possible order permutation (or a large random sample of orderings). Verify that all clients converge to the same document state. This is the most effective test — it finds edge cases that hand-written tests never cover. The Yjs and Automerge test suites are heavily based on this approach.
• Specific regression tests for known-hard cases. Concurrent insert at the same position. Delete of a range that overlaps with an insert. Formatting change that spans a boundary where another user is typing. Block move that conflicts with a block delete. Each of these should be an explicit test case.
• Long-running convergence soak test. Simulate a document with 10 clients editing continuously for 24 hours. Periodically snapshot all clients’ document states and verify they are identical. This catches slow divergence bugs — the kind where documents drift apart by one character every few thousand operations.

​

Q5: You are designing a system where 100,000 users are watching a live sports event and need score updates within 1 second. Walk through your architecture.

• Data ingestion: A small service consumes score data from the sports data provider’s API (or webhook). This service validates, normalizes, and publishes score events to an internal message bus (Redis Pub/Sub is sufficient here — the event rate is low, maybe one event every few seconds per game, and the data is ephemeral).
• Edge SSE servers: A fleet of SSE servers, each handling 10-20K concurrent SSE connections. Each server subscribes to the Redis Pub/Sub channel for score events. When a score event arrives, the server writes it to all active SSE connections. At 100K users, that is 5-10 edge servers.
• Why SSE and not WebSocket: Auto-reconnection with Last-Event-ID is built into the browser’s EventSource API. If a user’s connection drops (mobile network switch, laptop wakes from sleep), the browser reconnects automatically and the server replays missed events from a small in-memory buffer keyed by event ID. With WebSocket, I would have to build all of this reconnection and replay logic myself. For a read-only stream, that is unnecessary complexity.
• CDN edge caching (the advanced optimization). If the sports data provider sends updates every 5 seconds, I can put a CDN (Cloudflare, Fastly) in front of the SSE endpoints with a 1-2 second cache TTL. Every CDN edge location holds the SSE connection to my origin and fans out to thousands of local users. This turns my 100K origin connections into maybe 50 CDN-to-origin connections, with the CDN handling the fan-out at the edge. Fastly and Cloudflare both support SSE streaming through their CDN. This is the architecture that many live-score websites use.
• Fallback for environments where SSE is blocked. Long polling with a 5-second timeout as the fallback. The response includes the latest score state plus a cursor. The next request includes the cursor. Simple, works everywhere.

​

Follow-up: The product team now wants to show a real-time activity feed — “John just predicted Team A will win” — alongside the scores. 10,000 users are submitting predictions per minute. How does this change your architecture?

​

Follow-up: How do you handle the thundering herd when a goal is scored and 100K clients all simultaneously receive the event?

• Push enough data in the SSE event that the client does not need to fetch more. Include the scorer name, new score, and minute in the event payload. The client can update the UI immediately without any HTTP request.
• Stagger client-initiated requests with client-side jitter. If the client does need to fetch additional data (like a highlight clip URL), add a random delay of 0-5 seconds before making the request. setTimeout(() => fetch('/highlights/latest'), Math.random() * 5000).
• CDN caching for supplementary data. The highlight clip URL, updated standings, and commentary are the same for all 100K users. Serve them from CDN with a short cache TTL. The first request hits origin; the next 99,999 hit the CDN edge.

​

Q6: Your WebSocket-based chat application works perfectly in development but users report “messages arrive 5-10 seconds late” in production. How do you diagnose this?

• Step 1: Instrument the message path. Add timestamps at each hop: (1) when the sender’s client sends the message, (2) when the sender’s WebSocket server receives it, (3) when it is published to the pub/sub backbone, (4) when the recipient’s WebSocket server receives it from pub/sub, (5) when it is written to the recipient’s WebSocket, and (6) when the recipient’s client receives it. The gap between these timestamps tells me exactly where the latency lives. If the gap is between (3) and (4), the pub/sub layer is the bottleneck. If it is between (5) and (6), it is a network or proxy issue.
• Step 2: Check the load balancer idle timeout. This is the most common surprise. AWS ALB defaults to a 60-second idle timeout, and some HTTP proxies (Cloudflare, nginx) have shorter timeouts. If the WebSocket connection goes idle for longer than the proxy timeout, the proxy silently kills it. The client does not know the connection is dead until it tries to send or until the next heartbeat fails. Messages sent to that connection on the server side are written to a dead socket and silently dropped. The client eventually reconnects and fetches missed messages — hence the 5-10 second delay (heartbeat detection time + reconnect time + gap-fill time). The fix: ensure the heartbeat interval is shorter than the smallest idle timeout in the proxy chain.
• Step 3: Check Nagle’s algorithm and TCP buffering. Nagle’s algorithm (TCP_NODELAY not set) causes the TCP stack to buffer small messages and send them in batches. A small WebSocket frame (like a chat message) might be held by the TCP stack for up to 200ms waiting for more data. In some cases, interaction with delayed ACKs on the receiving side can push this to seconds. The fix: set TCP_NODELAY on the WebSocket server’s TCP sockets. Most WebSocket libraries do this by default, but some do not.
• Step 4: Check pub/sub consumer lag. If using Kafka, check consumer lag metrics. If the WebSocket servers’ Kafka consumer group has significant lag (thousands of unconsumed messages), every message is delayed by the time it takes to process the backlog. This can happen if the consumer’s processing throughput dropped (GC pauses, resource contention) or if the message rate spiked (viral event in a large channel).
• Step 5: Is it all users or specific users? If specific users report latency but others are fine, the issue is likely geographic (user far from the server region, adding 200ms+ RTT) or network-specific (user’s corporate proxy buffering WebSocket traffic, mobile carrier doing transparent proxying that interferes with WebSocket frames). Ask the affected users to run a WebSocket latency test from their network and compare with unaffected users.

​

Follow-up: You discover the latency is in the pub/sub layer — Redis Pub/Sub has 3-5 seconds of latency. What could cause that?

• Redis is under memory pressure and swapping to disk. Check INFO memory for used_memory vs maxmemory, and check the OS for swap usage. If Redis is swapping, every operation that touches swapped-out pages stalls.
• A slow subscriber is blocking the event loop. In Redis’s single-threaded model, if one subscriber is slow to consume messages, Redis buffers messages for that subscriber, and in extreme cases the buffer management itself can slow down the event loop. Check CLIENT LIST for clients with large obl (output buffer length) values.
• Network saturation between the WebSocket servers and Redis. If the Redis instance is in a different availability zone or region, the inter-AZ network might be congested. Check network throughput and packet loss between the WebSocket servers and Redis.
• Too many subscriptions. Each WebSocket server subscribes to a Redis channel for every conversation its connected users are in. At 50K connections with an average of 20 conversations each, that is 1 million subscription patterns. Redis evaluates every publish against all subscriber patterns. If you are using pattern-based subscriptions (PSUBSCRIBE), the matching cost is higher than exact channel matches. Switch to exact SUBSCRIBE per channel.

​

Going Deeper: If you had to redesign the pub/sub layer from scratch for sub-50ms guaranteed delivery latency, what would you do differently?

​

Q7: Explain backpressure in a WebSocket system. What happens when you ignore it, and how do you implement it?

• Per-connection send buffer cap. Set a maximum send buffer size per connection (1-5MB depending on the application). Monitor the buffer size. When it approaches the limit, start shedding load for that connection.
• Priority-based message shedding. When a connection’s buffer is above 50% capacity, drop low-priority messages first: typing indicators, presence updates, cursor positions. These are ephemeral and the latest state supersedes any dropped intermediate states. Between 50-80%, start dropping older messages in the buffer (the client will gap-fill on reconnect). Above 80%, send a “you are too slow, please reconnect” control message and close the connection.
• Adaptive message quality. For connections with growing buffers, switch from push-every-message to push-notification-only mode: instead of sending the full message payload, send a lightweight notification (“3 new messages in #general”) and let the client pull the full messages via HTTP when its connection improves. This is what Slack’s hybrid push-pull model does.
• Server-side monitoring. Track a distribution of per-connection buffer sizes across the fleet. Alert on the 99th percentile buffer size, not the average. The average will look healthy because 99% of connections are fast. The p99 is where the time bombs are.

​

Follow-up: How does TCP flow control interact with WebSocket backpressure? Isn’t TCP already handling this?

​

Follow-up: How would you implement graceful degradation for a user on a slow connection rather than just disconnecting them?

• Tier 1 (full fidelity): Buffer is below 30% capacity. Client receives everything: messages, typing indicators, presence updates, read receipts, cursor positions.
• Tier 2 (reduced fidelity): Buffer is between 30-60%. Stop sending typing indicators, presence updates, and cursor positions for this connection. These are reconstructed from the current state when the connection recovers. This reduces message volume by roughly 40-60% in a typical chat application.
• Tier 3 (essential only): Buffer is between 60-85%. Only send actual chat messages and direct mentions. Aggregate channel activity into periodic summaries (“12 new messages in #general”) instead of individual messages. Switch the client to pull-on-demand mode where opening a channel triggers an HTTP fetch rather than relying on the push stream.
• Tier 4 (disconnection): Buffer exceeds 85%. Send a control message indicating the connection quality is too poor, close the connection cleanly, and let the client reconnect when conditions improve.

​

Q8: How would you implement end-to-end encryption for group messages in a chat application with 100 participants?

• Each group member generates a “sender key” — a symmetric key used to encrypt their messages to the group. The sender key is itself distributed to each group member via individual 1:1 encrypted channels (using the double ratchet).
• When User A sends a message to the group, they encrypt it once with their sender key. All 100 recipients can decrypt it using User A’s sender key that they received earlier.
• The result: The sender encrypts once (not 100 times). The server stores one ciphertext (not 100). Fan-out is handled by the server distributing the same ciphertext to all recipients. Massive bandwidth improvement.
• The trade-off: When a member leaves the group, every remaining member must generate a new sender key and distribute it to all other members. Otherwise, the departed member could still decrypt future messages if they have the old sender keys. This “group ratchet” on member removal is O(N) key distributions for N remaining members. For a group of 100, removing one member triggers 99 key distributions. This is manageable but gets expensive for groups with frequent membership changes.

• Server-side search is impossible (the server cannot read messages). You must build client-side search indexes, which means every client indexes all their messages locally. For mobile clients with limited storage, this is a significant constraint.
• Server-side content moderation, spam filtering, and link previews cannot operate on encrypted content. Moderation must be reporter-based (users flag content) or client-side (the client scans after decryption, which raises privacy concerns).
• Message history for new group members requires the group to re-encrypt historical messages for the new member’s key, or accept that new members cannot see messages from before they joined.

​

Follow-up: A user loses their phone and gets a new one. How do they recover their message history if messages are end-to-end encrypted?

​

Q9: You need to deploy a WebSocket service across 3 regions (US, EU, Asia) to serve a global user base. What are the hardest problems you will face?

​

Follow-up: A conversation has 3 participants: one in Tokyo, one in London, one in New York. Where do you “home” the conversation, and what latency do they each experience?

​

Follow-up: How do you handle the case where the home region goes down mid-conversation?

• Pre-designate a failover region for each conversation (e.g., if home is US-East, failover is EU-West). Store this mapping in a globally replicated metadata store (DynamoDB Global Tables, CockroachDB, or Consul).
• Failure detection: Each region monitors the health of other regions via heartbeats. If US-East misses heartbeats from EU-West’s WebSocket fleet, EU-West may be down. After a configurable threshold (10-15 seconds to avoid false positives from transient network issues), declare the region unhealthy.
• Failover: All conversations homed in the failed region are automatically re-homed to their designated failover region. The failover region begins accepting and sequencing messages. The last known sequence number from the failed region is used as the starting point (retrieved from the globally replicated metadata store or from Kafka’s durable log).
• Recovery: When the failed region comes back, it does not automatically reclaim conversations. There is a manual or operator-triggered process to re-home conversations back, avoiding split-brain scenarios where both regions think they own the conversation.

​

Q10: What is the difference between total ordering, causal ordering, and per-sender ordering in a chat system? When is each sufficient?

​

Follow-up: Two users in the same conversation both send a message at nearly the same time. With per-conversation sequence numbers, one gets sequence 42 and the other gets 43. But User A sees their message first and User B sees their message first. Is this a problem?

​

Follow-up: How do vector clocks work for causal ordering, and what is their practical limitation?

​

Q11: Walk me through how you would implement graceful shutdown for a WebSocket server during a rolling deployment without losing messages.

• Step 1: Remove from load balancer. Signal the load balancer to stop routing new connections to the instance being drained. On AWS ALB, this is deregistration. On Kubernetes, this is failing the readiness probe while keeping the liveness probe healthy (so the pod is not killed but stops receiving new traffic).
• Step 2: Send a reconnect advisory. Send a control message to all connected clients: {"type": "system", "action": "reconnect", "reason": "server_maintenance", "delay_ms": 5000}. The delay_ms field tells the client to wait a random duration between 0 and 5000ms before reconnecting. This is the jitter that prevents thundering herd.
• Step 3: Wait for voluntary disconnections. Start a drain timer (e.g., 30 seconds). Well-behaved clients will receive the advisory, buffer any pending outbound messages, close the connection, and reconnect to a different server (since this one is no longer in the load balancer pool). During this window, the server continues processing messages normally for still-connected clients.
• Step 4: Force-close remaining connections. After 30 seconds, force-close any connections that did not disconnect voluntarily. These are likely stale connections (client crashed, network partition) or clients that do not implement the reconnect advisory. Send a WebSocket close frame with status code 1001 (“Going Away”) before TCP-closing.
• Step 5: Flush state to durable storage. Before the process exits, ensure the connection registry is updated (all connections for this server are removed from Redis), and any buffered messages that were not yet published to the pub/sub backbone are flushed.
• Step 6: New instance is already accepting connections. The new server version was started and added to the load balancer before Step 1 began. Clients reconnecting from the draining instance land on the new version. On reconnect, each client sends its lastMessageId and the new server replays missed messages from the message store.

​

Follow-up: How does this work in Kubernetes with a rolling deployment strategy?

• Define a preStop hook that sends the reconnect advisory and sleeps for the drain duration. The preStop hook runs before the SIGTERM is sent to the process.
• Set terminationGracePeriodSeconds to be longer than the drain duration (e.g., 45 seconds for a 30-second drain). This gives the pod time to complete the drain before Kubernetes force-kills it.
• The readiness probe must fail immediately when the drain starts (so Kubernetes removes the pod from the Service endpoint and the load balancer stops sending new connections). The liveness probe must continue passing during the drain (so Kubernetes does not restart the pod prematurely).
• Use a PodDisruptionBudget to ensure Kubernetes does not drain too many WebSocket pods simultaneously. If you have 10 pods, a PDB of maxUnavailable: 1 ensures only one pod drains at a time, preventing a scenario where 50% of your capacity is simultaneously draining.

​

Follow-up: What happens if the reconnect advisory message itself is lost because the client’s connection is already degraded?

​

Q12: Your WebSocket server uses JSON for message serialization. A staff engineer proposes switching to Protocol Buffers. Evaluate this proposal.

• Serialization speed. Protobuf serialization is typically 5-10x faster than JSON serialization and 2-5x faster than JSON parsing. For a WebSocket server processing 50K messages per second, if each message takes 0.1ms to serialize as JSON but 0.01ms as Protobuf, that is the difference between 5 seconds and 0.5 seconds of CPU time per second — a meaningful reduction that frees CPU for message routing.
• Message size. Protobuf messages are typically 30-50% smaller than their JSON equivalents. Field names are replaced by numeric tags, types are encoded efficiently (varints for integers), and there is no quoting or delimiters. For a chat message, JSON might be 200 bytes; Protobuf might be 120 bytes. At 50K messages per second to 50K connections, that is a bandwidth saving of 200GB/hour. At cloud bandwidth prices, that adds up.
• Schema enforcement. Protobuf requires a schema definition (.proto file). This acts as a contract between client and server. Malformed messages fail at deserialization rather than causing runtime errors when a field is missing or the wrong type. This catches integration bugs earlier.

• Debugging and observability. JSON is human-readable. When I tail a WebSocket connection’s traffic with wscat or inspect it in browser devtools, I can read the messages. Protobuf is binary — I need a decoder and the .proto file to make sense of it. This makes debugging production issues slower. Every engineer on the team needs the proto file and a decoder tool, and log inspection becomes a multi-step process.
• Client complexity. Browser clients need a Protobuf library (protobuf.js or protobuf-ts). This adds bundle size (protobuf.js is ~40KB minified+gzipped). Native JSON parsing in browsers is implemented in C++ inside the JS engine and is highly optimized — the gap between JSON and Protobuf in browser JavaScript is much smaller than in server-side Java or Go.
• Schema evolution. Adding a field to a Protobuf message is backward-compatible (old clients ignore new fields). Removing or renaming a field is not. You must maintain field numbers forever. With JSON, clients simply ignore fields they do not recognize, and schema evolution is more forgiving (though less safe).
• Migration cost. Every client and server must be updated simultaneously, or you need a compatibility layer that accepts both JSON and Protobuf during the transition. For a mobile app with a 2-week release cycle and users who do not update, this transition period could last months.

​

Follow-up: What about MessagePack or CBOR as alternatives that are binary but self-describing?

​

Going Deeper: At what scale does serialization format actually matter? Give me the numbers.

​

Advanced Interview Scenarios

​

Q13: Your product manager says “we need real-time updates on the dashboard.” You have 48 hours. The obvious answer is WebSockets. Why might that be the wrong call, and what do you actually build?

​

Follow-up: The PM comes back and says the dashboard now needs to serve 500,000 concurrent viewers during a product launch event. Does your answer change?

​

Follow-up: A colleague argues that polling “wastes resources” compared to WebSocket. How do you respond with data?

​

Q14: You have a Slack-like application with one channel that has 500,000 members. A user posts a message. Walk me through exactly what happens and where it breaks.

​

Follow-up: The 500K-member channel receives 200 messages per minute during a company all-hands. How do you prevent this from degrading performance for the rest of the platform?

​

Follow-up: How do you detect that a channel has become “hot” before it causes problems?

​

Q15: A mobile user is on a subway, cycling between connectivity and dead zones every 30-90 seconds. Describe exactly what happens to their WebSocket connection and chat experience, and what you build to make it seamless.

​

Follow-up: The user has been offline for 4 hours (long flight). They open the app. What is different about this reconnection compared to the 90-second subway gap?

​

Q16: Your WebSocket service is under a DDoS attack — an attacker is opening 100,000 connections per second. The connections authenticate successfully (the attacker has stolen valid API keys). How do you defend?

​

Follow-up: The attacker changes tactics — instead of opening many connections, they open 100 connections and send 10,000 messages per second per connection to large channels, amplifying fan-out. How do you defend?

​

Q17: You are debugging a production incident: two users in the same collaborative editing session are looking at different document content. The OT/CRDT system has silently diverged. How do you find and fix the root cause?

​

Follow-up: How do you build automated divergence detection into a collaborative editing system?

​

Q18: Your team runs 50 WebSocket servers behind an ALB. On Tuesday at 2 AM, a Redis cluster failover happens (primary dies, replica promotes). No messages are lost in Redis — but 30% of your users report they stopped receiving messages for 2 minutes. What happened?

​

Follow-up: How would you design the pub/sub layer to be resilient to broker failures without the complexity of Kafka?

​

Q19: A staff engineer on your team proposes using Cloudflare Durable Objects for the real-time collaboration layer instead of traditional WebSocket servers with Redis. Evaluate this architecture.

​

Follow-up: The Durable Object for a popular document gets 500 concurrent WebSocket connections. Is this a problem?

​

Q20: Your company has a microservice that sends real-time price updates to trading dashboards. Latency requirement: p99 under 50ms from price change to screen update. Current p99 is 200ms. Where do you look?

​

Follow-up: The data provider sends price updates over a REST API that you poll every second. How does this change your latency analysis?

​

Q21: You are tasked with building a “live cursors” feature — showing where every user’s cursor is on a shared whiteboard, like Figma. There are 200 concurrent users on the same board. What are the non-obvious engineering challenges?

​

Follow-up: Figma uses binary WebSocket frames for cursor data. A junior engineer on your team says “just use JSON, it is easier to debug.” Who is right and at what scale does it matter?

​

Q22: Your chat system’s WebSocket servers are stateless — all state is in Redis and Postgres. A senior engineer argues this is wrong and you should keep user session state in-memory on the WebSocket server. Who is right?

​

Follow-up: How do you handle the case where a user is connected on 5 devices simultaneously, and session state changes need to propagate to all of them?