​

Chapter 4: Global Distribution - Load Balancing and DNS

​

1. The Architecture of Planet-Scale Traffic

​

Maglev: The Distributed L4 Load Balancer

• No Shared State: Maglev instances do not communicate with each other to synchronize connection state. Instead, they use consistent hashing (Maglev Hashing) to ensure that packets from the same connection always end up at the same backend server, even if a Maglev node fails or a new one is added.

​

Deep Dive: The Maglev Hashing Algorithm

1. Lookup Table: Maglev maintains a large lookup table (typically 65,537 entries).
2. Preference Lists: Each backend generates a random-but-deterministic permutation of the lookup table indices.
3. Filling the Table: Backends take turns filling the lookup table based on their preference lists.
4. Result: When a backend is added or removed, only a small fraction of the lookup table entries change ($\frac{1}{N}$). This maintains connection persistence even during massive scale-out events.

• Connection Tracking: It maintains a local connection table but relies on its hashing algorithm as a fallback to ensure reliability.
• Performance: It handles traffic at line rate, distributing packets across the internal Google network with incredible efficiency.

​

GFE (Google Front End): The L7 Powerhouse

• Anycast IP: Every GFE announces the same global anycast IP via BGP. This ensures the user’s request hits the physically closest Google edge PoP.
• Protocol Optimization: GFEs support the latest protocols, including HTTP/3 (QUIC) and TLS 1.3, reducing the number of round trips (RTT) needed to establish a connection.
• SSL Termination: By terminating SSL at the edge, GFEs offload heavy cryptographic work from your backend VMs and reduce the latency of the TLS handshake.

​

2. Load Balancer Taxonomy

​

External Load Balancers (Internet-Facing)

​

Internal Load Balancers (VPC-Internal)

​

3. The Lifecycle of a Request: Internal vs. External

​

3.1 External Load Balancing Flow (The “Edge-to-Origin”)

1. Edge Entry: Request hits the nearest Google Edge PoP via Anycast.
2. GFE Termination: The Google Front End (GFE) terminates TLS. This is where Cloud Armor and Cloud CDN logic is applied.
3. URL Map Evaluation: The GFE checks the URL Map to see where the request should go.
4. B4 Backbone: The request is encapsulated and sent over Google’s private fiber backbone (B4) to the destination region.
5. Andromeda Delivery: Inside the region, the Andromeda SDN delivers the packet to the specific VM or Pod.

​

3.2 Internal Load Balancing Flow (The “Service-to-Service”)

1. Client Request: A VM in your VPC sends a request to the ILB’s internal IP.
2. Andromeda Redirection:
   • For L4 ILB: The Andromeda SDN redirects the packet directly to a backend instance using Maglev-style consistent hashing.
   • For L7 ILB: Traffic is routed to a pool of Envoy proxies managed by Google. These proxies handle the L7 logic (URL mapping, retries, weighted splits).
3. Local Backend: The traffic is delivered to a backend in the same region.

Principal Note: External LBs offer Global Anycast, meaning one IP for the whole world. Internal LBs are Regional, meaning you need an ILB in each region where you have clients, even if the backends are the same.

​

3.3 GKE Gateway API: The Modern Ingress

• GatewayClass: Managed by the Cloud Provider. Defines the type of LB (e.g., gke-l7-global-external-managed).
• Gateway: Managed by the Platform/Network team. Defines the IP and ports.
• HTTPRoute: Managed by the Application team. Defines the routing rules (paths, headers, weights).

​

3.4 Multi-Cluster Ingress (MCI)

• Global VIP: One IP for all clusters.
• Failover: If the cluster in us-east1 fails, traffic is automatically routed to europe-west1.
• Latency: Users are always routed to the cluster closest to them.

​

4. Cloud CDN: The Global Accelerator

​

4.1 How it Works: Cache Keys and Invalidation

• Optimization: You can include or exclude query strings, protocols, or specific headers from the cache key to improve the hit ratio.
• Invalidation: Propagates globally in minutes. You can invalidate by path (/images/*) or by header.

​

4.2 Signed URLs for Private Content

1. The application generates a URL with a cryptographic signature and an expiration time.
2. Cloud CDN verifies the signature at the edge.
3. Access is granted without the request ever reaching your backend, provided the signature is valid.

​

5. Security with Identity-Aware Proxy (IAP)

​

5.1 Beyond VPNs

• Context-Aware: IAP checks the user’s identity (IAM) and context (IP, device health).
• Centralized: No more managing VPN certificates or client software.
• Header Injection: IAP injects the X-Goog-IAP-JWT-Assertion header, allowing your application to trust the user’s identity without performing its own auth.

​

6. Cloud Armor: The Distributed Firewall

​

4.1 How it Integrates with GFE

​

4.2 Advanced Security Features

• WAF Rules (OWASP Top 10): Pre-configured rules for SQLi, XSS, LFI, and RFI.
• Threat Intelligence: Block traffic from known malicious actors, Tor exit nodes, or specific search engines using evaluateThreatIntelligence.
• Adaptive Protection: Uses ML to establish a baseline of “normal” traffic and alerts on anomalies. It can even auto-generate a suggested rule during a DDoS attack.
• Bot Management: Challenges clients with reCAPTCHA Enterprise if they exhibit bot-like behavior.

​

4.3 Bot Management Policy Example

# Create a security policy
gcloud compute security-policies create protect-my-app

# Add a rule to challenge suspicious traffic with reCAPTCHA
gcloud compute security-policies rules create 1000 \
    --security-policy=protect-my-app \
    --expression="evaluatePreconfiguredExpr('cve-canary')" \
    --action=redirect \
    --redirect-type=google-recaptcha

​

5. Cloud DNS: The Global Name Server

​

Architectural Highlights

• 100% Availability SLA: Provided by the massive redundancy of Google’s anycast infrastructure.
• DNSSEC: Protects your domains from spoofing and cache poisoning attacks.
• Split-Horizon DNS: Use the same domain name (e.g., app.internal) but resolve it to different IPs depending on whether the query comes from the internet or inside your VPC.

​

DNS Policies

1. Weighted Round Robin: Distribute traffic across multiple IPs based on weights.
2. Geolocation Routing: Return different IP addresses based on the user’s physical location.
3. Failover Policy: Automatically switch to a backup IP if the primary health check fails (integrates with Cloud Monitoring).

​

6. Advanced Load Balancing: TCP/UDP Proxy and Internal Cross-Region

​

6.1 External TCP/UDP Proxy Load Balancing

• Global Anycast: One IP for the world.
• TLS Offloading: Can terminate SSL/TLS at the edge (Proxy SSL) even for non-HTTP traffic.
• Packet Path: Traffic is terminated at the edge GFE and proxied over Google’s backbone to the backend.

​

6.2 Internal Load Balancing (Regional)

• No Proxy vMAC: Unlike traditional ILBs, GCP ILBs don’t use a proxy VM. Andromeda intercepts packets at the source VM’s vNIC and delivers them directly to the destination VM’s vNIC.
• Latency: This results in virtually zero latency overhead compared to direct VM-to-VM communication.

​

7. Health Checks and High Availability

• Types: HTTP, HTTPS, TCP, SSL, and HTTP/2.
• Parameters:
  • Check Interval: How often to check (e.g., every 5 seconds).
  • Timeout: How long to wait for a response (e.g., 5 seconds).
  • Healthy Threshold: Number of consecutive successes to mark as healthy.
  • Unhealthy Threshold: Number of consecutive failures to mark as unhealthy.
• Auto-Healing: When integrated with a Managed Instance Group (MIG), a failing health check can trigger the deletion and recreation of the VM.

​

8. Implementation: The “Perfect” Global Architecture

1. Cloud DNS with a CNAME pointing to the Global HTTP(S) LB Anycast IP.
2. Cloud Armor policy attached to the LB to block SQLi and Geo-block restricted countries.
3. Cloud CDN enabled on the Backend Service for static assets (/static/*).
4. Managed Instance Groups (MIGs) in at least three regions (us-east1, europe-west1, asia-east1).
5. Internal TCP/UDP LB in each region to handle traffic between the web tier and the database tier.

​

GFE Header Modification Rules

• X-Forwarded-For: Contains the client’s IP address and the chain of proxies it passed through.
• X-Cloud-Trace-Context: A unique ID used for end-to-end tracing across services. Essential for SREs using Cloud Trace.
• X-Forwarded-Proto: Indicates the protocol used by the client (usually https).
• X-GFE-Backend-Request-Cost: (Optional) Informs the backend of the processing cost incurred at the edge.

​

8. Troubleshooting the Traffic Path

1. Check Health Checks: 90% of “LB not working” issues are due to failing health checks. Ensure your firewall allows the GCP health check ranges (35.191.0.0/16 and 130.211.0.0/22).
2. Verify Firewall Rules: Your backend VMs must have an ingress rule allowing traffic from the LB proxies on the service port.
3. Inspect URL Map: Use the console’s “Traffic Distribution” view to ensure your path rules are actually routing to the correct backend services.
4. Cloud Trace: Use Cloud Trace to see the end-to-end latency of a request, identifying if the bottleneck is the LB, the network, or the backend application.

​

Interview Preparation