Skip to main content

​
Distributed Networking Fundamentals

Networks are the backbone of distributed systems. Understanding how networks behave, fail, and recover is essential for building reliable systems. If you think of a distributed system as a team of people working together, the network is the phone system connecting them. And like a phone system, it can have static, dropped calls, crossed wires, and dead zones — except in distributed systems, you have to design your software to keep working despite all of these problems simultaneously.
Module Duration: 10-14 hours
Key Topics: TCP/UDP, RPC, Message Delivery Semantics, Failure Detection, Service Mesh
Interview Focus: Network partitions, idempotency, exactly-once delivery

​
The Network Reality

The First Fallacy: The network is reliable. It’s not, and your system design must account for this.

​
TCP vs UDP for Distributed Systems

​
TCP Guarantees and Limitations

​
UDP Use Cases

​
Advanced: Kernel-Bypass Networking (RDMA & DPDK)

For most applications, the standard Linux TCP/IP stack is sufficient. However, at Staff/Principal level, when building ultra-high-performance systems (like HFT platforms or distributed databases like FaRM or eRPC), the OS kernel itself becomes the bottleneck.

​
The Problem: The “Kernel Tax”

Traditional networking suffers from three major overheads:
  1. Context Switching: Moving between user space and kernel space for every send() or recv().
  2. Data Copying: Packets are copied from the NIC to kernel buffers, then to user-space buffers (CPU-intensive).
  3. Interrupt Handling: The CPU is interrupted for every incoming packet, thrashing caches.

​
1. RDMA (Remote Direct Memory Access)

RDMA allows one computer to read or write directly into the memory of another computer without involving either system’s OS or CPU.
Key RDMA Verbs:
  • SEND/RECEIVE: Two-sided (both CPUs involved briefly).
  • READ/WRITE: One-sided (the remote CPU is never even notified). This is the fastest but hardest to program.

​
2. DPDK (Data Plane Development Kit)

DPDK moves the entire networking stack into user space.
  • Poll-Mode Drivers: Instead of waiting for interrupts, the CPU constantly “polls” the NIC for new packets.
  • Hugepages: Minimizes TLB misses by using 1GB memory pages.
  • Zero-Copy: Applications read directly from the NIC’s ring buffer.

​
Comparison: When to Bypass the Kernel?

Staff Tip: Only reach for RDMA/DPDK if your p99 latency requirements are sub-100 microseconds. For most “at-scale” systems, optimizing your serialization (Protobuf) and batching is more impactful than kernel bypass.

​
Hardware Offloading (SmartNICs & DPUs)

As we scale beyond 100Gbps networking, even Kernel-Bypass (DPDK) starts to consume significant CPU cycles just for processing packets. The modern solution is to move the entire Networking Data Plane into dedicated hardware.

​
1. The DPU (Data Processing Unit)

A DPU (or SmartNIC) is a “computer in front of the computer.” It contains its own ARM cores, memory, and specialized hardware accelerators.
  • AWS Nitro: Perhaps the most famous example. AWS moved VPC networking, EBS storage encryption, and management logic off the main Xeon CPU onto dedicated Nitro cards.
  • Offloaded Tasks:
    • Encryption (TLS/IPSec): Zero CPU cost for the application.
    • Storage Virtualization: NVMe-over-Fabrics offload.
    • Network Policy: Firewalls and security groups enforced in hardware.

​
2. Infrastructure-as-Code in Hardware

By using DPUs, cloud providers can offer “Bare Metal” instances that still have full VPC networking and EBS support, because the infrastructure logic lives on the DPU, not in the host OS. Staff Tip: When designing for “Cloud Native” scale, the DPU is the boundary that allows you to separate the Tenant workload from the Provider infrastructure.

​
Message Delivery Semantics

Critical Interview Topic: Understanding at-most-once, at-least-once, and exactly-once semantics is fundamental for distributed systems design.

​
At-Most-Once Delivery

​
At-Least-Once Delivery

​
Exactly-Once Delivery

​
Global Traffic Management: Anycast vs. GSLB

When your system is distributed across the planet, the first problem is: How do I get the user to the nearest healthy datacenter?

​
1. DNS-based GSLB (Global Server Load Balancing)

The traditional way. The DNS server detects the user’s IP (or their resolver’s IP) and returns the IP of the closest DC.
  • Pros: Simple to implement.
  • Cons: DNS Caching. Even with low TTLs, some resolvers or browsers cache results for minutes. If a DC fails, users may still be routed to it until the cache expires.

​
2. IP Anycast (The Modern Way)

Anycast allows multiple physical servers (in different parts of the world) to share the same IP address.
Benefits of Anycast:
  • Lowest Latency: Packets naturally flow to the closest network edge.
  • Instant Failover: If the London Edge goes down, BGP stops advertising that IP from there. The internet automatically routes the next packet to the next closest DC (e.g., Paris).
  • DDoS Mitigation: Anycast naturally spreads the load of a flood attack across many global points of presence (PoPs).

​
Comparison Matrix

Staff Tip: For a global, high-availability service, use Anycast for your entry point (Edge/LB) and use DNS as a secondary layer for steering traffic between specific backends.

​
Remote Procedure Calls (RPC)

​
RPC Fundamentals

Google’s high-performance RPC framework. Used internally at Google for virtually all inter-service communication (billions of RPCs per second), and increasingly the standard for microservice communication in the broader industry.
Pros:
  • Binary protocol (efficient)
  • Strong typing with Protocol Buffers
  • Streaming support
  • Generated client/server code
Cons:
  • Binary = harder to debug
  • Browser support requires grpc-web
  • Steeper learning curve

​
RPC Failure Modes

Interview Question: “What can go wrong with an RPC call?”

​
Failure Detection

​
Heartbeat-Based Detection

​
Phi Accrual Failure Detector

​
Gossip Protocols

​
How Gossip Works

​
SWIM Protocol

​
Service Mesh and Sidecars

​
Modern Networking Patterns

​
Service Mesh Comparison

​
Interview Practice

Question: Design an RPC client that handles transient failures gracefully.Key Points:
  • Retry with exponential backoff and jitter
  • Idempotency keys for non-idempotent operations
  • Circuit breaker to fail fast when service is down
  • Deadline propagation across service calls
  • Connection pooling for efficiency
Question: What is the Two Generals Problem and how does it relate to distributed systems?Answer: The Two Generals Problem demonstrates the impossibility of achieving consensus over an unreliable channel.Scenario:
  • Two armies (A and B) must attack simultaneously to win
  • They communicate via messengers through enemy territory
  • Messengers can be captured (messages lost)
The Problem:
  • A sends “Attack at dawn” to B
  • A doesn’t know if B received it
  • B sends “Acknowledged” to A
  • B doesn’t know if A received the ack
  • A sends “Acknowledged your ack” to B
  • This continues infinitely!
Implications for Distributed Systems:
  • No protocol can guarantee agreement with unreliable messaging
  • This is why we need:
    • Timeouts and retries
    • Idempotency
    • Eventual consistency (accept uncertainty)
    • Consensus protocols (for synchronous systems)
Question: Design a system that sends notifications to millions of users with at-least-once delivery.Architecture:
Key Design Decisions:
  1. Message Queue: Kafka for durability and replay
  2. Partitioning: By user_id for ordering per user
  3. Deduplication: Store sent notification IDs in Redis
  4. Retry Strategy: Exponential backoff per channel
  5. Dead Letter Queue: For failed notifications
  6. Rate Limiting: Per user and per channel

​
Key Takeaways

Networks Fail

Design for packet loss, delays, partitions, and duplicates. Never assume reliability.

Idempotency is Essential

Make operations safe to retry. Use idempotency keys for non-idempotent operations.

Choose Semantics Wisely

Understand at-most-once, at-least-once, and exactly-once. Pick based on your use case.

Detect Failures Carefully

False positives are dangerous. Use adaptive failure detection like Phi Accrual.

​
Next Steps

Time & Clocks

Understand logical clocks, vector clocks, and TrueTime

Consistency Models

Deep dive into linearizability, serializability, and eventual consistency

​
Interview Deep-Dive

Strong Answer:
  • At-most-once means the sender fires and forgets. No retries. If the message is lost, it is gone. This is suitable for metrics or telemetry where losing a few data points is acceptable. UDP-based protocols often provide this.
  • At-least-once means the sender retries until it gets an acknowledgment. The message will definitely arrive, but it may arrive more than once. This is the default for most message queues (Kafka, RabbitMQ). The receiver must be prepared to handle duplicates.
  • Exactly-once is the holy grail. It means every message is processed exactly one time. True exactly-once across an unreliable network is technically impossible (Two Generals Problem). What systems actually provide is “effectively exactly-once” through a combination of at-least-once delivery plus idempotent processing on the receiver side.
  • For a payment system, I would choose at-least-once delivery with idempotent processing. The reason: losing a payment (at-most-once) is unacceptable, and exactly-once is not achievable in the general case. So I ensure every payment request carries an idempotency key. If the same key arrives twice, the second request returns the cached result of the first without re-executing the payment. This gives the caller the illusion of exactly-once while the underlying transport provides at-least-once.
Follow-up: How would you implement an idempotency layer for this payment system at scale?I would maintain an idempotency store — a fast key-value store like Redis or DynamoDB — that maps idempotency keys to their response. Before processing any payment, the service checks this store. If the key exists, it returns the stored response. If not, it processes the payment, stores the result under the key, and returns it. The critical subtlety is atomicity: the “check, process, store” sequence must be atomic with respect to concurrent retries of the same key. I would use a distributed lock or a compare-and-swap operation on the idempotency key to prevent two workers from processing the same payment simultaneously. The idempotency key has a TTL (e.g., 24 hours) to prevent unbounded storage growth. For additional safety, I would include a unique request ID in the payment’s database record and use a database-level unique constraint as a second line of defense.
Strong Answer:
  • First, I determine the scope: is this affecting all calls to Service B, or only calls from Service A? If Service B’s latency is elevated for all callers, the problem is likely on the server side (GC pauses, resource exhaustion, slow dependency). If only Service A is affected, the problem is likely network-related or specific to the A-B path.
  • I check distributed traces (Jaeger/Zipkin) to see where the latency is. Is the time spent in network transit, in Service B’s queue, or in Service B’s processing? If the trace shows the request arriving at B quickly but B responding slowly, it is a server-side issue. If the request takes a long time to arrive, it is a network issue.
  • For network issues, I check for packet loss and retransmission rates between A and B (TCP retransmit metrics, ss -ti output). A common culprit is a congested switch or a misconfigured firewall doing deep packet inspection. I also check if A and B are in different availability zones, which adds cross-AZ latency and increases the chance of intermediate network issues.
  • For server-side issues on B, I look at B’s CPU, memory, GC pause logs, thread pool utilization, and connection pool exhaustion. A full connection pool means B is waiting for a downstream dependency, not that B itself is slow.
  • I also check the timeout configuration: is A’s timeout reasonable given B’s p99 latency? If A has a 100ms timeout but B’s p99 is 95ms, you will see frequent timeouts just from normal tail latency.
Follow-up: The investigation reveals that Service B has occasional 2-second GC pauses. How does this affect the broader system, and what would you do about it?A 2-second GC pause in Service B has cascading effects. During the pause, B cannot process requests or send heartbeats. If B is behind a load balancer, the LB may keep sending requests to B, which pile up in B’s TCP receive buffer. When B resumes, it processes a burst of stale requests, some of which have already been timed out by their callers, wasting resources. Meanwhile, callers that timed out may have retried to another instance, creating duplicate processing. If B is a Raft leader, a 2-second pause can trigger a leader election (typical election timeout is 1-2 seconds). Mitigation strategies: first, tune the GC — switch to a low-pause collector like ZGC or Shenandoah, reduce heap size, or reduce allocation rate. Second, add health-check-based load balancing that removes B from the pool during pauses. Third, implement request deadlines: when B resumes, it should check if each queued request’s deadline has passed and discard expired ones rather than processing them.
Strong Answer:
  • A simple timeout detector uses a fixed threshold: if no heartbeat arrives within T milliseconds, the node is declared dead. The problem is choosing T. Too short and you get false positives (declaring a healthy but slow node as dead). Too long and genuine failures take too long to detect. In a system with variable network latency, no single T value is optimal.
  • The Phi Accrual detector (used by Cassandra and Akka) replaces the binary dead/alive decision with a continuous suspicion level. It maintains a sliding window of recent heartbeat inter-arrival times, computes their mean and standard deviation, and then calculates the probability that the node is still alive given how long it has been since the last heartbeat. The phi value is the negative log of this probability.
  • A phi of 1 means roughly a 10% chance the node is dead. A phi of 8 means effectively certain it is dead. The threshold is configurable: Cassandra defaults to phi=8.
  • The key advantage is adaptiveness. If the network becomes jittery and heartbeat intervals increase, the detector automatically adjusts its expectations. A fixed timeout would start producing false positives; the Phi detector would simply widen its confidence interval.
Follow-up: What are the failure modes of the Phi Accrual detector itself?The Phi detector has two main failure modes. First, the cold-start problem: when a node first joins the cluster, there is no heartbeat history to build a distribution from. During this window, the detector either uses a default conservative timeout or declares the node “unknown” until enough samples are collected. Second, the detector assumes heartbeat intervals follow a roughly normal distribution. If the actual distribution is bimodal (e.g., fast when not under GC, very slow during GC), the normal approximation underestimates the probability of long gaps, leading to premature suspicion during GC pauses. Cassandra mitigates this by using a fairly high phi threshold (8) and a large sample window (1000 intervals). A more sophisticated approach would use a non-parametric distribution or explicitly model the bimodal behavior.
Strong Answer:
  • gRPC uses HTTP/2, binary serialization (Protocol Buffers), and provides strong typing with code generation. REST uses HTTP/1.1 (typically), text-based JSON, and relies on conventions (OpenAPI) for typing.
  • I would choose gRPC for internal service-to-service communication in a microservices backend. The reasons: binary serialization reduces payload size by 3-10x compared to JSON, HTTP/2 multiplexing eliminates head-of-line blocking and reduces connection overhead, and the generated client/server stubs eliminate an entire class of integration bugs (mismatched field names, wrong types). gRPC also natively supports streaming, which is critical for real-time data flows.
  • I would choose REST/JSON for external-facing APIs (public APIs, mobile clients, browser clients). JSON is human-readable, universally supported, and easy to debug with curl. gRPC in the browser requires grpc-web, which adds complexity. REST also has better caching semantics (HTTP caching headers, CDN support).
  • The hybrid approach most companies use: gRPC internally between microservices, REST/JSON externally at the API gateway. The gateway translates between the two.
Follow-up: What are the operational challenges of running gRPC at scale that you would not face with REST?The biggest operational challenge is debugging. gRPC traffic is binary, so you cannot simply tcpdump and read the payload like you can with JSON. You need gRPC-aware tooling (grpcurl, Envoy’s gRPC access logging, or middleware that logs deserialized payloads). Second, load balancing is more complex. HTTP/2 connections are long-lived and multiplexed, so traditional L4 (TCP) load balancers will pin all traffic from one client to one server. You need L7 load balancing that understands HTTP/2 frames and distributes individual RPCs across backends. Third, schema evolution requires discipline. If you delete a field in a proto definition without proper deprecation, you can silently break consumers. REST/JSON is more forgiving because unknown fields are typically ignored. These are solvable problems, but they require investment in tooling and process that REST does not demand.