Module 13: Load Balancing & Proxies As applications scale, single servers can’t handle the load. Load balancers distribute traffic across multiple servers, while proxies act as intermediaries for various purposes. This module covers both in depth.
Estimated Time : 3-4 hoursDifficulty : IntermediatePrerequisites : Modules 4-6 (Network fundamentals)
13.1 Why Load Balancing? The Single Server Problem 100,000 users │ ▼ ┌──────────────┐ │ Server │ ← Overwhelmed! │ (1 CPU, │ - High latency │ 8GB RAM) │ - Requests timeout └──────────────┘ - Single point of failure
The Solution: Distribute Load 100,000 users │ ▼ ┌──────────────┐ │ Load │ │ Balancer │ └──────┬───────┘ ┌────────────┼────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Server 1 │ │ Server 2 │ │ Server 3 │ │ ~33,333 │ │ ~33,333 │ │ ~33,333 │ │ users │ │ users │ │ users │ └──────────┘ └──────────┘ └──────────┘
13.2 Load Balancing Algorithms 1. Round Robin Requests are distributed sequentially.
Request 1 → Server A Request 2 → Server B Request 3 → Server C Request 4 → Server A ← Cycle repeats Request 5 → Server B ...
Pros : Simple, fair distribution
Cons : Doesn’t consider server load or capacity2. Weighted Round Robin Servers with higher capacity get more requests.
Server A (weight 3): Gets 3 out of 6 requests Server B (weight 2): Gets 2 out of 6 requests Server C (weight 1): Gets 1 out of 6 requests Sequence: A, A, A, B, B, C, A, A, A, B, B, C...
3. Least Connections Send to server with fewest active connections.
Server A: 150 connections Server B: 50 connections ← Next request goes here Server C: 200 connections
Best for : Long-lived connections (WebSockets, database connections)4. Least Response Time Send to server with fastest response time + fewest connections.
Server A: 50ms avg, 100 connections Server B: 30ms avg, 120 connections ← Faster, preferred Server C: 80ms avg, 80 connections
5. IP Hash Same client IP always goes to same server.
hash(client_ip) % num_servers = target_server IP 192.168.1.10 → hash → always Server B IP 192.168.1.11 → hash → always Server A
Use Case : Session persistence without sticky sessions6. Least Bandwidth Route to server currently serving least traffic (Mbps).
Algorithm Comparison Algorithm Best For Drawback Round Robin Simple, equal servers Ignores load Weighted RR Mixed capacity Static weights Least Connections Long connections Overhead tracking Least Response Time Performance-critical Needs health checks IP Hash Session stickiness Uneven distribution
13.3 Layer 4 vs Layer 7 Load Balancing Layer 4 (Transport Layer) Operates on TCP/UDP level. Sees IP addresses and ports only.
Client ──TCP──→ Load Balancer ──TCP──→ Server │ Sees: Source IP, Dest IP Source Port, Dest Port Can't see: HTTP headers, URLs, cookies
Pros : Very fast (no packet inspection), simple
Cons : Limited routing options, no content-based decisionsLayer 7 (Application Layer) Operates on HTTP/HTTPS level. Can inspect full request.
Client ──HTTP──→ Load Balancer ──HTTP──→ Server │ Sees: Everything! - URL path (/api/users) - Headers (Host, Cookie, Auth) - HTTP method (GET, POST) - Query parameters - Request body
Pros : Smart routing, SSL termination, caching, compression
Cons : More processing overhead, higher latencyWhen to Use Which Scenario Layer Simple TCP load distribution L4 Route by URL path L7 SSL termination L7 Gaming servers, databases L4 Microservices routing L7 A/B testing L7
13.4 Proxy Types Forward Proxy Sits between clients and the internet. Clients know about it.
┌─────────────────────────────────────────────────────┐ │ Corporate Network │ │ │ │ ┌────────┐ │ │ │Client A│──┐ │ │ └────────┘ │ ┌──────────┐ │ │ ├────►│ Forward │ │ │ ┌────────┐ │ │ Proxy │────► Internet │ │ │Client B│──┘ └──────────┘ │ │ └────────┘ │ └─────────────────────────────────────────────────────┘
Use Cases:
Content filtering (block certain sites)
Caching (reduce bandwidth)
Anonymity (hide client IPs)
Access control (authentication)
Reverse Proxy Sits between internet and servers . Clients don’t know about it.
┌─────────────────────────────┐ │ Data Center │ │ │ Internet ────► ┌───────────────┐ ┌────────────┐ │ │ Reverse │──►│ Server A │ │ │ Proxy │ └────────────┘ │ │ │ ┌────────────┐ │ │ (nginx, etc) │──►│ Server B │ │ └───────────────┘ └────────────┘ │ │ │ └─────────────────────────────┘
Use Cases:
Load balancing
SSL termination
Caching static content
DDoS protection
Compression
URL rewriting
13.5 SSL/TLS Termination The Problem Encryption is CPU-intensive. Every server doing SSL:
Client ──HTTPS──► LB ──HTTPS──► Server 1 (decrypts) ──HTTPS──► Server 2 (decrypts) ──HTTPS──► Server 3 (decrypts) Each server needs: - SSL certificates - CPU for encryption/decryption
The Solution: SSL Termination Load balancer handles all SSL:
Client ──HTTPS──► Load Balancer ──HTTP──► Server 1 (encrypted) (decrypts) (plain) Server 2 Server 3
Benefits:
Servers freed from SSL overhead
Single place to manage certificates
Easier SSL certificate renewal
SSL Passthrough vs Termination vs Re-encryption PASSTHROUGH (L4): Client ──HTTPS──► LB ──HTTPS──► Server (just forwards, doesn't decrypt) TERMINATION: Client ──HTTPS──► LB ──HTTP──► Server (decrypts, internal is plain) RE-ENCRYPTION: Client ──HTTPS──► LB ──HTTPS──► Server (decrypts, re-encrypts for backend)
13.6 Session Persistence (Sticky Sessions) Some applications need the same user to always reach the same server.
Why Needed? Request 1: Login to Server A (session created in memory) Request 2: Goes to Server B (no session! User logged out!)
Sticky Session Methods 1. Cookie-based: Load balancer sets cookie: SERVERID=server-a All requests with this cookie → Server A
2. IP Hash: Same IP always routes to same server (breaks with shared IPs, NAT)
3. Application Cookie: App sets session cookie LB reads it and routes accordingly
Better Alternative: Shared Session Store ┌───────────────────────────────────────────────────┐ │ Load Balancer │ │ (no sticky sessions) │ └───────────────────────┬───────────────────────────┘ ┌─────────────┼─────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ │Server A │ │Server B │ │Server C │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ │ └─────────────┼─────────────┘ ▼ ┌─────────────┐ │ Redis │ │ (sessions) │ └─────────────┘
All servers share sessions → any server can handle any request.
13.7 Health Checks Load balancers must know which servers are healthy.
Passive Health Checks Monitor real traffic for failures:
Server A: 5 consecutive 500 errors → mark unhealthy Server A: 10 successful responses → mark healthy again
Active Health Checks Periodically probe servers:
Every 5 seconds: GET /health → 200 OK? Server is healthy GET /health → timeout/500? Server is unhealthy
Health Check Configuration # Nginx example upstream backend { server 10.0.1.1:8080; server 10.0.1.2:8080; server 10.0.1.3:8080; # Health check parameters health_check interval=5s fails=3 passes=2; }
Health Check Endpoints Your application should expose health endpoints:
/health → Simple "I'm alive" /health/ready → "I'm ready to serve traffic" /health/live → "I'm running (not deadlocked)"
// /health response { "status" : "healthy" , "checks" : { "database" : "connected" , "cache" : "connected" , "disk" : "ok" }, "version" : "1.2.3" }
13.8 Common Load Balancer Products Cloud Load Balancers Provider L4 L7 Global AWS NLB ALB Global Accelerator GCP Network LB HTTP(S) LB Cloud CDN Azure Load Balancer Application Gateway Front Door
Software Load Balancers Product Type Use Case Nginx L7 Web apps, reverse proxy HAProxy L4/L7 High-performance, TCP/HTTP Traefik L7 Kubernetes, microservices Envoy L7 Service mesh, cloud-native
Hardware Load Balancers Vendor Products F5 BIG-IP Citrix ADC/NetScaler A10 Thunder
13.9 Load Balancing Patterns Blue-Green Deployment Active (Blue): 100% traffic → v1.0 Standby (Green): 0% traffic → v2.0 (testing) Switch: Blue: 0% traffic Green: 100% traffic → v2.0 (now live)
Canary Deployment Main: 95% traffic → v1.0 Canary: 5% traffic → v2.0 (testing with real users) Gradually increase canary: Main: 90% → 50% → 10% → 0% Canary: 10% → 50% → 90% → 100%
A/B Testing Route based on user attributes: User in test group A → Server with feature X User in test group B → Server without feature X Analyze which performs better
13.10 Reverse Proxy Use Cases URL Rewriting External: Internal: /api/users ──────► /v2/users-service/list /api/products ──────► /product-catalog/all
Path-Based Routing # Nginx config location /api/ { proxy_pass http://api-servers; } location /static/ { proxy_pass http://cdn-servers; } location / { proxy_pass http://web-servers; }
Host: api.example.com → API servers Host: www.example.com → Web servers Host: admin.example.com → Admin servers
Caching ┌────────┐ ┌───────────────┐ ┌────────┐ │ Client │ ──► │ Reverse Proxy │ ──► │ Server │ └────────┘ │ (cached) │ └────────┘ └───────────────┘ First request: Miss, fetch from server, cache Second request: Hit, serve from cache (no server hit)
13.11 Real-World Architecture Typical Web Application ┌─────────────────────────────────────────────────────────────┐ │ │ │ User → CDN → WAF → Load Balancer → Web Servers │ │ │ │ │ ├──────────► API Servers │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────────┐ │ │ │ │ Database │ │ │ │ │ (Redis) │ │ │ │ └──────────────┘ │ │ │ │ └─────────────────────────────────────────────────────────────┘
Microservices Architecture ┌────────────────┐ │ API Gateway │ │ (L7 routing) │ └───────┬────────┘ ┌────────────────────┼────────────────────┐ │ │ │ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ User Service│ │Order Service│ │Product Svc │ │ LB │ │ LB │ │ LB │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ ┌──┴──┐ ┌──┴──┐ ┌──┴──┐ │ │ │ │ │ │ │ │ │ │ │ │ (instances) (instances) (instances)
13.12 Key Takeaways
L4 vs L7 L4 is fast but simple. L7 is smart but more overhead.
Algorithm Matters Choose based on your traffic pattern and server capabilities.
Health Checks Always configure proper health checks to avoid routing to dead servers.
SSL Termination Offload SSL to load balancers to simplify certificate management.
Next Module
Module 14: Network Troubleshooting Master the tools and techniques for diagnosing network issues.
