Documentation Index
Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
Use this file to discover all available pages before exploring further.
Distributed Networking Fundamentals
Networks are the backbone of distributed systems. Understanding how networks behave, fail, and recover is essential for building reliable systems. If you think of a distributed system as a team of people working together, the network is the phone system connecting them. And like a phone system, it can have static, dropped calls, crossed wires, and dead zones — except in distributed systems, you have to design your software to keep working despite all of these problems simultaneously.Module Duration: 10-14 hours
Key Topics: TCP/UDP, RPC, Message Delivery Semantics, Failure Detection, Service Mesh
Interview Focus: Network partitions, idempotency, exactly-once delivery
Key Topics: TCP/UDP, RPC, Message Delivery Semantics, Failure Detection, Service Mesh
Interview Focus: Network partitions, idempotency, exactly-once delivery
The Network Reality
TCP vs UDP for Distributed Systems
TCP Guarantees and Limitations
UDP Use Cases
Advanced: Kernel-Bypass Networking (RDMA & DPDK)
For most applications, the standard Linux TCP/IP stack is sufficient. However, at Staff/Principal level, when building ultra-high-performance systems (like HFT platforms or distributed databases like FaRM or eRPC), the OS kernel itself becomes the bottleneck.The Problem: The “Kernel Tax”
Traditional networking suffers from three major overheads:- Context Switching: Moving between user space and kernel space for every
send()orrecv(). - Data Copying: Packets are copied from the NIC to kernel buffers, then to user-space buffers (CPU-intensive).
- Interrupt Handling: The CPU is interrupted for every incoming packet, thrashing caches.
1. RDMA (Remote Direct Memory Access)
RDMA allows one computer to read or write directly into the memory of another computer without involving either system’s OS or CPU.- SEND/RECEIVE: Two-sided (both CPUs involved briefly).
- READ/WRITE: One-sided (the remote CPU is never even notified). This is the fastest but hardest to program.
2. DPDK (Data Plane Development Kit)
DPDK moves the entire networking stack into user space.- Poll-Mode Drivers: Instead of waiting for interrupts, the CPU constantly “polls” the NIC for new packets.
- Hugepages: Minimizes TLB misses by using 1GB memory pages.
- Zero-Copy: Applications read directly from the NIC’s ring buffer.
Comparison: When to Bypass the Kernel?
| Feature | Standard TCP/IP | DPDK | RDMA (RoCE/Infiniband) |
|---|---|---|---|
| Complexity | Low (Standard Sockets) | High (Special Libraries) | Very High (Hardware-dependent) |
| Latency | 50μs - 1ms | 10μs - 50μs | < 5μs |
| CPU Usage | High (Kernel overhead) | Medium (Polling) | Low (Offloaded to NIC) |
| Hardware | Anything | Standard NICs | RDMA-capable NICs (RNICs) |
Hardware Offloading (SmartNICs & DPUs)
As we scale beyond 100Gbps networking, even Kernel-Bypass (DPDK) starts to consume significant CPU cycles just for processing packets. The modern solution is to move the entire Networking Data Plane into dedicated hardware.1. The DPU (Data Processing Unit)
A DPU (or SmartNIC) is a “computer in front of the computer.” It contains its own ARM cores, memory, and specialized hardware accelerators.- AWS Nitro: Perhaps the most famous example. AWS moved VPC networking, EBS storage encryption, and management logic off the main Xeon CPU onto dedicated Nitro cards.
- Offloaded Tasks:
- Encryption (TLS/IPSec): Zero CPU cost for the application.
- Storage Virtualization: NVMe-over-Fabrics offload.
- Network Policy: Firewalls and security groups enforced in hardware.
2. Infrastructure-as-Code in Hardware
By using DPUs, cloud providers can offer “Bare Metal” instances that still have full VPC networking and EBS support, because the infrastructure logic lives on the DPU, not in the host OS.| Level | Networking Logic | CPU Cost | Latency |
|---|---|---|---|
| Standard | Kernel Stack | High | 100μs+ |
| Advanced | User-space (DPDK) | Medium | 10μs+ |
| Principal | Hardware (DPU/Nitro) | Zero | < 2μs |
Message Delivery Semantics
At-Most-Once Delivery
At-Least-Once Delivery
Exactly-Once Delivery
Global Traffic Management: Anycast vs. GSLB
When your system is distributed across the planet, the first problem is: How do I get the user to the nearest healthy datacenter?1. DNS-based GSLB (Global Server Load Balancing)
The traditional way. The DNS server detects the user’s IP (or their resolver’s IP) and returns the IP of the closest DC.- Pros: Simple to implement.
- Cons: DNS Caching. Even with low TTLs, some resolvers or browsers cache results for minutes. If a DC fails, users may still be routed to it until the cache expires.
2. IP Anycast (The Modern Way)
Anycast allows multiple physical servers (in different parts of the world) to share the same IP address.- Lowest Latency: Packets naturally flow to the closest network edge.
- Instant Failover: If the London Edge goes down, BGP stops advertising that IP from there. The internet automatically routes the next packet to the next closest DC (e.g., Paris).
- DDoS Mitigation: Anycast naturally spreads the load of a flood attack across many global points of presence (PoPs).
Comparison Matrix
| Feature | Geo-DNS | Anycast |
|---|---|---|
| Control | High (choose exact IP per user) | Lower (BGP decides path) |
| Failover Speed | Slow (Minutes, DNS TTL) | Fast (Seconds, BGP convergence) |
| Complexity | Low | High (requires BGP/Autonomous System) |
| Used By | Most mid-sized apps | Cloudflare, Google, AWS (Global Accelerator) |
Remote Procedure Calls (RPC)
RPC Fundamentals
Popular RPC Frameworks
- gRPC
- REST/JSON
- Thrift
Google’s high-performance RPC framework. Used internally at Google for virtually all inter-service communication (billions of RPCs per second), and increasingly the standard for microservice communication in the broader industry.Pros:
- Binary protocol (efficient)
- Strong typing with Protocol Buffers
- Streaming support
- Generated client/server code
- Binary = harder to debug
- Browser support requires grpc-web
- Steeper learning curve
RPC Failure Modes
Failure Detection
Heartbeat-Based Detection
Phi Accrual Failure Detector
Gossip Protocols
How Gossip Works
SWIM Protocol
Service Mesh and Sidecars
Modern Networking Patterns
Service Mesh Comparison
| Feature | Istio | Linkerd | Consul Connect |
|---|---|---|---|
| Proxy | Envoy | Linkerd-proxy (Rust) | Envoy |
| Complexity | High | Low | Medium |
| Performance | Good | Best | Good |
| mTLS | Yes | Yes | Yes |
| Traffic Management | Advanced | Basic | Medium |
| Observability | Comprehensive | Good | Good |
| Best For | Large enterprises | Simplicity focused | HashiCorp stack |
Interview Practice
Q1: Design a robust RPC client
Q1: Design a robust RPC client
Question: Design an RPC client that handles transient failures gracefully.Key Points:
- Retry with exponential backoff and jitter
- Idempotency keys for non-idempotent operations
- Circuit breaker to fail fast when service is down
- Deadline propagation across service calls
- Connection pooling for efficiency
Q2: Explain the Two Generals Problem
Q2: Explain the Two Generals Problem
Question: What is the Two Generals Problem and how does it relate to distributed systems?Answer:
The Two Generals Problem demonstrates the impossibility of achieving consensus over an unreliable channel.Scenario:
- Two armies (A and B) must attack simultaneously to win
- They communicate via messengers through enemy territory
- Messengers can be captured (messages lost)
- A sends “Attack at dawn” to B
- A doesn’t know if B received it
- B sends “Acknowledged” to A
- B doesn’t know if A received the ack
- A sends “Acknowledged your ack” to B
- This continues infinitely!
- No protocol can guarantee agreement with unreliable messaging
- This is why we need:
- Timeouts and retries
- Idempotency
- Eventual consistency (accept uncertainty)
- Consensus protocols (for synchronous systems)
Q3: Design a distributed notification system
Q3: Design a distributed notification system
Question: Design a system that sends notifications to millions of users with at-least-once delivery.Architecture:Key Design Decisions:
- Message Queue: Kafka for durability and replay
- Partitioning: By user_id for ordering per user
- Deduplication: Store sent notification IDs in Redis
- Retry Strategy: Exponential backoff per channel
- Dead Letter Queue: For failed notifications
- Rate Limiting: Per user and per channel
Key Takeaways
Networks Fail
Design for packet loss, delays, partitions, and duplicates. Never assume reliability.
Idempotency is Essential
Make operations safe to retry. Use idempotency keys for non-idempotent operations.
Choose Semantics Wisely
Understand at-most-once, at-least-once, and exactly-once. Pick based on your use case.
Detect Failures Carefully
False positives are dangerous. Use adaptive failure detection like Phi Accrual.
Next Steps
Time & Clocks
Understand logical clocks, vector clocks, and TrueTime
Consistency Models
Deep dive into linearizability, serializability, and eventual consistency
Interview Deep-Dive
Explain the difference between at-most-once, at-least-once, and exactly-once delivery semantics. Which one would you choose for a payment processing system and why?
Explain the difference between at-most-once, at-least-once, and exactly-once delivery semantics. Which one would you choose for a payment processing system and why?
Strong Answer:
- At-most-once means the sender fires and forgets. No retries. If the message is lost, it is gone. This is suitable for metrics or telemetry where losing a few data points is acceptable. UDP-based protocols often provide this.
- At-least-once means the sender retries until it gets an acknowledgment. The message will definitely arrive, but it may arrive more than once. This is the default for most message queues (Kafka, RabbitMQ). The receiver must be prepared to handle duplicates.
- Exactly-once is the holy grail. It means every message is processed exactly one time. True exactly-once across an unreliable network is technically impossible (Two Generals Problem). What systems actually provide is “effectively exactly-once” through a combination of at-least-once delivery plus idempotent processing on the receiver side.
- For a payment system, I would choose at-least-once delivery with idempotent processing. The reason: losing a payment (at-most-once) is unacceptable, and exactly-once is not achievable in the general case. So I ensure every payment request carries an idempotency key. If the same key arrives twice, the second request returns the cached result of the first without re-executing the payment. This gives the caller the illusion of exactly-once while the underlying transport provides at-least-once.
You are debugging a production incident where an RPC call between Service A and Service B is intermittently timing out. Walk me through your debugging approach.
You are debugging a production incident where an RPC call between Service A and Service B is intermittently timing out. Walk me through your debugging approach.
Strong Answer:
- First, I determine the scope: is this affecting all calls to Service B, or only calls from Service A? If Service B’s latency is elevated for all callers, the problem is likely on the server side (GC pauses, resource exhaustion, slow dependency). If only Service A is affected, the problem is likely network-related or specific to the A-B path.
- I check distributed traces (Jaeger/Zipkin) to see where the latency is. Is the time spent in network transit, in Service B’s queue, or in Service B’s processing? If the trace shows the request arriving at B quickly but B responding slowly, it is a server-side issue. If the request takes a long time to arrive, it is a network issue.
- For network issues, I check for packet loss and retransmission rates between A and B (TCP retransmit metrics,
ss -tioutput). A common culprit is a congested switch or a misconfigured firewall doing deep packet inspection. I also check if A and B are in different availability zones, which adds cross-AZ latency and increases the chance of intermediate network issues. - For server-side issues on B, I look at B’s CPU, memory, GC pause logs, thread pool utilization, and connection pool exhaustion. A full connection pool means B is waiting for a downstream dependency, not that B itself is slow.
- I also check the timeout configuration: is A’s timeout reasonable given B’s p99 latency? If A has a 100ms timeout but B’s p99 is 95ms, you will see frequent timeouts just from normal tail latency.
What is the Phi Accrual failure detector and why is it better than a simple timeout-based detector?
What is the Phi Accrual failure detector and why is it better than a simple timeout-based detector?
Strong Answer:
- A simple timeout detector uses a fixed threshold: if no heartbeat arrives within T milliseconds, the node is declared dead. The problem is choosing T. Too short and you get false positives (declaring a healthy but slow node as dead). Too long and genuine failures take too long to detect. In a system with variable network latency, no single T value is optimal.
- The Phi Accrual detector (used by Cassandra and Akka) replaces the binary dead/alive decision with a continuous suspicion level. It maintains a sliding window of recent heartbeat inter-arrival times, computes their mean and standard deviation, and then calculates the probability that the node is still alive given how long it has been since the last heartbeat. The phi value is the negative log of this probability.
- A phi of 1 means roughly a 10% chance the node is dead. A phi of 8 means effectively certain it is dead. The threshold is configurable: Cassandra defaults to phi=8.
- The key advantage is adaptiveness. If the network becomes jittery and heartbeat intervals increase, the detector automatically adjusts its expectations. A fixed timeout would start producing false positives; the Phi detector would simply widen its confidence interval.
Compare gRPC and REST for inter-service communication in a microservices architecture. When would you pick one over the other?
Compare gRPC and REST for inter-service communication in a microservices architecture. When would you pick one over the other?
Strong Answer:
- gRPC uses HTTP/2, binary serialization (Protocol Buffers), and provides strong typing with code generation. REST uses HTTP/1.1 (typically), text-based JSON, and relies on conventions (OpenAPI) for typing.
- I would choose gRPC for internal service-to-service communication in a microservices backend. The reasons: binary serialization reduces payload size by 3-10x compared to JSON, HTTP/2 multiplexing eliminates head-of-line blocking and reduces connection overhead, and the generated client/server stubs eliminate an entire class of integration bugs (mismatched field names, wrong types). gRPC also natively supports streaming, which is critical for real-time data flows.
- I would choose REST/JSON for external-facing APIs (public APIs, mobile clients, browser clients). JSON is human-readable, universally supported, and easy to debug with curl. gRPC in the browser requires grpc-web, which adds complexity. REST also has better caching semantics (HTTP caching headers, CDN support).
- The hybrid approach most companies use: gRPC internally between microservices, REST/JSON externally at the API gateway. The gateway translates between the two.
tcpdump and read the payload like you can with JSON. You need gRPC-aware tooling (grpcurl, Envoy’s gRPC access logging, or middleware that logs deserialized payloads). Second, load balancing is more complex. HTTP/2 connections are long-lived and multiplexed, so traditional L4 (TCP) load balancers will pin all traffic from one client to one server. You need L7 load balancing that understands HTTP/2 frames and distributes individual RPCs across backends. Third, schema evolution requires discipline. If you delete a field in a proto definition without proper deprecation, you can silently break consumers. REST/JSON is more forgiving because unknown fields are typically ignored. These are solvable problems, but they require investment in tooling and process that REST does not demand.