Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Part XVIII — Networking

​

Chapter 25: Networking for Engineers

​

25.1 DNS

Analogy: DNS is like the phone book of the internet — you look up a name and get a number (IP address). Just like you do not memorize every phone number, your browser does not memorize every IP address. It looks up the name in a series of “phone books” (DNS servers), each one more authoritative than the last, until it finds the number it needs. And just like phone books get outdated, DNS records have a TTL that controls how long the “listing” is trusted before you need to look it up again.

​

Recursive vs Iterative Resolution

​

TTL Implications for Deployments

​

DNS Rollback Lag — The Hidden Deployment Risk

• Pre-lower TTL before any DNS-dependent deploy. If your normal TTL is 3600s, drop it to 60s at least 2x the old TTL before the change.
• Prefer load-balancer-level traffic shifting over DNS-level. An ALB target group weight change or Istio traffic split takes effect in seconds, not minutes. DNS is a blunt instrument for traffic management.
• For regional failover: Use health-check-based DNS failover with already-lowered TTLs, not manual DNS changes. Route 53 health checks can automatically remove unhealthy endpoints without human intervention, but the failover time is still TTL-bounded.
• Monitor “DNS coherence” after a change: query multiple public resolvers (8.8.8.8, 1.1.1.1, 208.67.222.222) and your authoritative nameserver simultaneously. When all return the new record, propagation is effectively complete.

​

CDN Behavior and Edge Caching

​

TLS Certificate Rotation

​

Regional Failover — When an Entire Region Goes Down

• Cache is cold. Redis/Memcached in the standby region has not been serving traffic. Every request is a cache miss hitting the database. At 50K requests/second, this can overwhelm the database in seconds.
• Connection pools are empty. Database connection pools need to be established. The initial burst of connection creation adds latency.
• JIT compilation is cold. JVM-based services have not warmed their JIT compilers. Initial request processing is significantly slower.
• Autoscaling has not kicked in. If the standby region runs at minimal capacity, autoscaling takes 2-5 minutes to provision additional instances.

​

Brownouts — The Failure Mode Nobody Plans For

• Percentile-based alerting, not average-based. Alert on p99 latency, not mean latency. A brownout that makes 10% of requests 10x slower barely moves the mean but destroys the p99.
• Error budget burn rate. Instead of alerting on absolute error rate, alert on the rate at which you are consuming your error budget. A 5% error rate that is normally 0.1% is a 50x increase — a clear brownout signal even though 5% sounds low.
• Per-instance and per-AZ segmentation. Aggregate metrics hide brownouts. Break down every metric by instance, AZ, and region. A dashboard that shows per-instance error rates as a heatmap makes brownouts visually obvious.

​

Traffic Draining for Maintenance and Deploys

​

DNS Troubleshooting Commands Engineers Actually Use

​

25.2 HTTP, HTTPS, TLS

​

HTTP/2 vs HTTP/3 (QUIC): Concrete Comparison

​

25.3 Load Balancers

​

L4 vs L7 Load Balancer Comparison

​

25.4 Reverse Proxies and API Gateways

• Gateway: TLS termination, authentication (token validation), rate limiting, request logging, correlation ID injection, CORS
• Application: Authorization (business rules), input validation, business logic, database access
• Anti-pattern: The God Gateway — business logic in the gateway couples all services and creates a deployment bottleneck

​

25.5 Service Discovery

​

Client-Side vs Server-Side Discovery

​

25.6 WebSockets and Real-Time Communication

​

WebSockets vs SSE vs Long-Polling

​

Part XIX — Deployment and Release Engineering

​

Chapter 26: Deployment Strategies

​

26.1 Rolling Deployment

Choose this when: You need zero-downtime deploys with minimal infrastructure cost and your changes are backward-compatible. Avoid this when: You need instant rollback (rolling back means re-deploying, which takes minutes) or your change cannot coexist with the previous version (breaking schema changes, incompatible API contracts).

​

26.2 Blue-Green

Analogy: Blue-green deployment is like having two identical stages in a theater — you rehearse on one while the audience watches the other, then swap. The audience (your users) never sees the stagehands moving props around. If the new show has a problem mid-performance, you instantly swing the spotlight back to the original stage where the old show is still ready to go. The cost? You need two full stages (double the infrastructure), and both stages need to work with the same backstage systems (your database).

Choose this when: You need instant rollback (sub-second traffic switch), you’re doing a major release with high risk, or compliance requires pre-production validation in an identical environment. Avoid this when: You’re cost-constrained (requires double the infrastructure), you have complex stateful workloads (both environments sharing a database is the hardest part), or you deploy many times per day (the overhead of maintaining two full environments adds friction).

1. Before cutover: Run expand-only migrations (add columns, add tables). Both Blue and Green can work with the expanded schema.
2. Deploy Green: Green writes to both old and new columns. Green reads from new columns with fallback to old.
3. Cutover: Switch traffic from Blue to Green.
4. After cutover (days later): Run contract migrations (remove old columns, drop old tables) once Blue is no longer needed.

​

26.3 Canary

Choose this when: You have mature observability (metrics, dashboards, automated analysis), you’re deploying changes with unknown risk profiles, or you serve high traffic where even a brief full-rollout failure is unacceptable. Avoid this when: You lack the monitoring infrastructure to compare canary vs baseline metrics (you’ll be flying blind), your traffic volume is too low for statistical significance (canary analysis needs enough requests to detect differences), or the change is all-or-nothing (e.g., a database migration that all instances must run simultaneously).

​

Deployment Strategy Selection Guide

​

Blast-Radius Control

Phase 1: Deploy to "canary region" (lowest traffic, non-revenue-critical)
  - Example: ap-southeast-2 (Australia) if primary traffic is US/EU
  - Bake for 30-60 minutes. Monitor all metrics.
  - Rollback trigger: any metric regression vs pre-deploy baseline

Phase 2: Deploy to secondary region
  - Example: eu-west-1 (Ireland)
  - Bake for 30 minutes. Compare metrics against canary region.
  - Rollback trigger: divergence between regions or metric regression

Phase 3: Deploy to primary region
  - Example: us-east-1 (Virginia) -- your highest-traffic region
  - This is the last region to deploy. If something went wrong,
    you caught it in Phase 1 or 2 with minimal user impact.
  - Bake for 60 minutes. Full monitoring.

Phase 4: Deploy to remaining regions (if any)

​

Rollback Timing — How Fast Is “Fast Enough”?

​

Cache Invalidation After Deploy

DEPLOY PIPELINE — Cache Invalidation Steps
1. Deploy new application code (rolling/canary/blue-green)
2. Verify health checks pass on new instances
3. Invalidate application-level cache (Redis/Memcached):
   - If using versioned cache keys: no action needed (new keys auto-miss)
   - If NOT using versioned keys: flush affected key prefixes
     redis-cli EVAL "for _,k in pairs(redis.call('keys','config:*')) do redis.call('del',k) end" 0
4. Invalidate CDN cache for changed paths:
   - CloudFront: aws cloudfront create-invalidation --paths "/api/*" "/index.html"
   - Fastly: fastly purge --service-id $SVC_ID --key "deploy-$SHA"
   - Cloudflare: curl -X POST "api.cloudflare.com/.../purge_cache" -d '`{"files":["..."]}`'
5. Verify CDN is serving new content:
   - curl -sI "https://cdn.example.com/app.js" | grep -i "x-cache\|age\|etag"
   - Compare ETag/Last-Modified against expected values for the new build
6. Monitor cache hit rate:
   - Expect a temporary dip in cache hit rate (new keys are cold)
   - If hit rate does not recover within 10 minutes, investigate

​

26.4 Feature Flags

​

Feature Flag Best Practices

​

26.5 Graceful Shutdown and Connection Draining

1. Instance receives SIGTERM.
2. Stop accepting new requests (mark as not ready — fail health checks or deregister from service discovery).
3. Load balancer stops routing new traffic (health check fails or deregistration propagates — this takes a few seconds).
4. Drain in-flight requests — wait for all currently processing requests to complete (drain period).
5. Close resources — close database connections, flush logs and metrics, finish writing to message queues.
6. Timeout guard — if draining has not completed within the grace period, log the remaining requests and exit.
7. Process exits cleanly (exit code 0).
8. If the process has not exited, the orchestrator sends SIGKILL (non-catchable, immediate termination).

​

Concrete Graceful Shutdown Sequence

 Time 0s    SIGTERM received
    |        -> Set "shutting down" flag
    |        -> Return 503 on /health (stop new traffic)
    |        -> preStop hook delay (5-10s for LB deregistration)
    |
 Time 10s   LB has deregistered this instance
    |        -> No new requests arriving
    |        -> In-flight requests continue processing
    |
 Time 10-25s  Drain period
    |        -> Waiting for in-flight requests to complete
    |        -> Closing idle DB connections
    |        -> Flushing log buffers and metrics
    |
 Time 25s   Drain complete (or timeout reached)
    |        -> Log any requests that could not complete
    |        -> Exit process with code 0
    |
 Time 30s   terminationGracePeriodSeconds reached
    |        -> Kubernetes sends SIGKILL if process still alive

​

26.6 Zero-Downtime Database Migrations

​

The Expand-Contract Pattern (a.k.a. Parallel Change)

-- Phase 1: Expand
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;
-- Safe: old code ignores the new column, new code can start using it

-- Phase 2: Backfill historical data
UPDATE users SET email_verified = TRUE
WHERE id IN (SELECT user_id FROM email_verifications)
-- Run in batches to avoid long-running transactions

-- Phase 3: Contract (days/weeks later)
ALTER TABLE users DROP COLUMN legacy_email_status;
-- Only after ALL application code has been updated to use the new column

​

Online DDL Tools

​

Backfill Strategies

# Pseudocode: Batch backfill with rate limiting
last_id = 0
batch_size = 5000
while True:
    rows = db.execute(
        "UPDATE users SET email_verified = (email IS NOT NULL) "
        "WHERE id > %s AND id <= %s AND email_verified IS NULL",
        [last_id, last_id + batch_size]
    )
    if rows == 0:
        break
    last_id += batch_size
    time.sleep(0.1)  # Let replication catch up
    log(f"Backfilled up to id={last_id}")

​

Migration Safety Rules

​

26.7 CI/CD Pipeline Design

​

CI/CD Pipeline Best Practices

​

CI/CD Pipeline Maturity Model

​

26.8 Deployment Failure Scenarios

​

26.9 Deployment Readiness Checklist

​

Pre-Deploy

• Database migrations tested against production-sized data? Run migrations against a copy of production data. A migration that takes 2 seconds on staging can lock a table for 20 minutes in production.
• Database migrations backward-compatible? Both the old and new application code must work with the migrated schema. No column drops, no renames, no NOT NULL additions without defaults.
• Feature flags configured? New functionality is behind a flag, defaulting to off. The flag can be toggled without a deploy. Both flag-on and flag-off paths are tested.
• Monitoring dashboards ready? The team deploying has a dashboard showing: error rates, latency percentiles (p50, p95, p99), business metrics (transactions/min, sign-ups/min), and infrastructure metrics (CPU, memory, connection pool utilization). The dashboard is open before the deploy starts.
• Alerts configured? Automated alerts exist for: error rate spike, latency degradation, business metric drop, and resource exhaustion. Alerts should fire within 2 minutes of an issue.
• Rollback plan tested? The rollback procedure has been executed in staging within the last 30 days. An untested rollback is not a rollback — it’s a hope. Document the rollback steps and the expected time to complete.
• On-call engineer aware? The on-call engineer knows a deploy is happening, when it’s happening, and what changed. They have the rollback runbook open. Never surprise your on-call.
• Deploy window appropriate? Not during peak traffic. Not on Friday afternoon. Not before a holiday. Not during another team’s deploy. Check the deploy calendar.
• Dependency changes verified? If the deploy includes updated dependencies (library versions, API versions), those dependencies have been tested in staging with production-like traffic. Check for breaking changes in dependency changelogs.
• Configuration changes applied? Environment variables, secrets, and configuration files needed by the new version are already deployed to production. The new code will not start up and fail because a config value is missing.

​

During Deploy

• Watching dashboards in real-time? Someone is actively watching the monitoring dashboard during the entire rollout, not “deploying and going to lunch.”
• Canary metrics compared against baseline? If using canary deployment, automated or manual comparison of canary vs baseline metrics is happening at each rollout stage.
• Rollback trigger defined? The team has agreed on specific, objective criteria for rolling back: “If error rate exceeds X% for Y minutes, we roll back. No debate.”

​

Post-Deploy

• Health checks passing? All instances report healthy. No restarts, no OOMKills, no crash loops.
• Error rates stable? Error rate has returned to pre-deploy baseline within 15 minutes. Any new error types are investigated even if the rate is low.
• Latency stable? p50, p95, and p99 latency are within acceptable range of pre-deploy baseline.
• Business metrics normal? Transactions, sign-ups, conversions, or whatever your core business metric is — it hasn’t dropped.
• Log review done? Scan logs for new warnings or errors that weren’t present before the deploy. Even if metrics look fine, new log patterns can indicate latent issues.
• Feature flags activated (if applicable)? If the deploy was a code-only ship with features behind flags, the flag rollout plan is scheduled and documented.
• Deploy recorded? The deploy is logged with: who deployed, what changed (commit SHA or version), when, and any anomalies observed. This is essential for postmortem correlation.

​

26.10 GitOps — Declarative Infrastructure and Pull-Based Deployments

Analogy: Traditional deployment is like a chef calling out orders to the kitchen (“fire two steaks, drop fries now”). GitOps is like a restaurant where the chef writes the menu on a whiteboard, and the kitchen staff continuously check the whiteboard and prepare whatever is written there. You never tell the kitchen what to do directly — you change the whiteboard, and the kitchen converges to match it. The whiteboard is your Git repository. The kitchen is your cluster. The magic is that the whiteboard is versioned, auditable, and you can see exactly when someone changed “grilled salmon” to “pan-seared salmon.”

1. Declarative configuration. The entire desired state (Kubernetes manifests, Helm charts, Kustomize overlays, Terraform files) is stored in Git. Not scripts that produce state — the state itself.
2. Git as the single source of truth. The Git repository is the canonical definition of what should be running. If it is not in Git, it should not be in production.
3. Automated reconciliation. An agent running inside the cluster continuously compares the actual state against the desired state in Git. If they diverge (someone runs kubectl edit manually, a pod crashes, drift occurs), the agent automatically corrects it.
4. Pull-based deployment. Unlike traditional CI/CD where a pipeline pushes changes to the cluster (requiring cluster credentials in the CI system), GitOps agents pull changes from Git. The cluster reaches out to Git, not the other way around. This is a significant security improvement — your CI pipeline never needs direct access to production.

​

Push-Based vs Pull-Based Deployment

​

ArgoCD vs Flux

​

A GitOps Workflow in Practice

Developer                    Git Repository                ArgoCD/Flux                Cluster
   |                              |                            |                        |
   |-- 1. Push code change ------>|                            |                        |
   |                              |                            |                        |
   |   (CI builds image,          |                            |                        |
   |    tags as myapp:abc123)     |                            |                        |
   |                              |                            |                        |
   |-- 2. Update image tag ------>|  (deployment.yaml:         |                        |
   |      in manifests repo       |   image: myapp:abc123)     |                        |
   |                              |                            |                        |
   |                              |-- 3. Agent detects ------->|                        |
   |                              |      Git diff              |                        |
   |                              |                            |-- 4. kubectl apply --->|
   |                              |                            |      (reconcile)       |
   |                              |                            |                        |
   |                              |                            |<-- 5. Status report ---|
   |                              |                            |      (healthy/degraded)|
   |                              |                            |                        |
   |                              |   6. If drift detected:    |                        |
   |                              |      agent auto-corrects ->|-- re-apply manifests ->|

​

26.11 Deployment Observability — What to Watch and When

​

The Three Phases of Deployment Observability

1. Error rate delta. The difference between current error rate and baseline. Display as a graph with the deploy start marked as a vertical line. Any upward slope after the deploy line is a signal. Threshold: if error rate exceeds baseline + 0.5% for more than 2 minutes, investigate. If it exceeds baseline + 2%, roll back immediately.
2. Latency delta (p99). Same treatment as error rate. Latency often degrades before errors appear because the system is under stress but not yet failing. A p99 increase of >50% from baseline is a strong rollback signal. Watch for latency spikes at the exact moment new instances start receiving traffic (cold JVM, empty caches, connection pool warm-up).
3. Instance health. Number of healthy instances, restarts, OOMKills, crash loops. During a rolling deploy, you expect some instances to cycle. What you do not expect: instances restarting repeatedly (crash loop), instances being killed for exceeding memory limits (OOMKill), or instances failing readiness probes.
4. Saturation signals. CPU utilization, memory usage, connection pool utilization, thread pool saturation, queue depth. These are leading indicators — they degrade before errors start. A new version that uses 40% more CPU per request will not error immediately but will saturate and error under load.
5. Business metrics. Orders per minute, sign-ups per minute, messages sent per minute — whatever your product’s heartbeat is. Technical metrics can look green while business metrics tank (e.g., a redirect loop that returns 200 OK but prevents users from completing checkout). Business metrics are the ultimate source of truth.

​

The Deployment Dashboard

​

Curated Resources

​

Networking Deep Dives

• Cloudflare Learning Center — Arguably the best free resource for understanding DNS, CDN, DDoS protection, SSL/TLS, and networking fundamentals. Each topic gets a clear, illustrated explanation with real-world context. Start with “What is DNS?” and “What is a CDN?” then explore DDoS attack types and mitigation strategies.
• Julia Evans’ Networking Zines — Julia Evans creates visual, hand-drawn explanations of networking concepts that make complex topics click instantly. Her zines on DNS, HTTP, TCP, and networking tools (dig, curl, tcpdump) are some of the best learning materials in existence. The visual format encodes information differently than text and helps concepts stick. Highly recommended for both beginners and experienced engineers who want to solidify mental models.
• QUIC Protocol and HTTP/3 — Cloudflare’s Explanation — Cloudflare’s writeup on HTTP/3 and the QUIC protocol is the clearest explanation of why HTTP/3 moved from TCP to UDP, how QUIC eliminates head-of-line blocking, and what 0-RTT connection resumption means in practice. For the formal specification, see RFC 9000 (QUIC Transport Protocol) and RFC 9114 (HTTP/3).
• AWS Well-Architected Framework — Networking Pillar — AWS’s opinionated guide to networking architecture in the cloud. Covers VPC design, subnet strategies, load balancing, DNS, CDN (CloudFront), and hybrid connectivity. Even if you do not use AWS, the architectural patterns (public/private subnet separation, NAT gateways, transit gateways) apply universally.

​

Deployment and Release Engineering

• Netflix Tech Blog — Deployment and Delivery — Netflix has published extensively on their deployment infrastructure, including Spinnaker (their open-source continuous delivery platform), Kayenta (automated canary analysis), and their philosophy on progressive delivery. Key posts to read: “Automated Canary Analysis at Netflix with Kayenta” and “Full Cycle Developers at Netflix.” These are not theoretical — they describe systems handling 250M+ subscribers.
• Google SRE Book — Chapter on Release Engineering — Free online. Google’s chapter on release engineering describes how they manage deployments across a codebase with billions of lines of code and tens of thousands of engineers. Covers hermetic builds, release branches, cherry-picks, and the philosophy that release engineering is a distinct engineering discipline, not a side task for developers.
• Charity Majors’ Blog on Progressive Delivery — Charity Majors (co-founder of Honeycomb, former infrastructure engineer at Facebook and Parse) writes some of the most incisive content on observability, deployment, and engineering culture. Her posts on testing in production, progressive delivery, and the relationship between deploy frequency and reliability are essential reading. She challenges conventional wisdom with data and experience.
• LaunchDarkly Blog — Feature Flag Best Practices — LaunchDarkly is the leading feature flag platform, and their blog is a comprehensive resource on feature flag lifecycle management, progressive delivery patterns, experimentation, and the organizational practices that make feature flags sustainable rather than technical debt. Particularly valuable: their guides on flag cleanup, testing strategies for flagged code, and the distinction between release flags, experiment flags, and operational flags.

​

GitOps and Declarative Infrastructure

• ArgoCD Documentation and Getting Started Guide — The official ArgoCD docs are unusually well-written for a CNCF project. Start with the “Getting Started” tutorial to set up a working GitOps pipeline in under 30 minutes, then read the “Best Practices” guide for production patterns including multi-cluster management, RBAC configuration, and the App of Apps pattern for managing dozens of applications declaratively.
• Flux Documentation — Flux takes a more Kubernetes-native, composable approach to GitOps compared to ArgoCD. The docs cover the controller architecture (source-controller, kustomize-controller, helm-controller) and how to compose them for complex deployment workflows. The “GitOps Toolkit” section explains the building blocks that make Flux extensible.
• GitOps and Kubernetes by Billy Yuen, Alexander Matyushentsev, Todd Ekenstam, Jesse Suen — The definitive book on GitOps patterns. Covers both ArgoCD and Flux with production examples, secret management strategies, multi-tenancy, and the organizational changes needed to adopt GitOps. Particularly valuable: the chapters on handling secrets and managing configuration drift.

​

Database Migrations

• gh-ost: GitHub’s Online Schema Migration Tool — GitHub’s tool for online schema migrations in MySQL. The README alone is a masterclass in understanding MySQL locking behavior, binary log replication, and why traditional ALTER TABLE is dangerous on large production tables. Even if you don’t use MySQL, the design document explains migration safety principles that apply to any database.
• Strong Migrations (Rails) — A Ruby gem that detects dangerous migrations and suggests safe alternatives. Even if you don’t use Rails, the README is one of the best references for which database operations are safe and which are not, organized by database engine (PostgreSQL, MySQL, MariaDB). Bookmark the README as a migration safety checklist.

​

Cross-Chapter Connections

​

Deployment as Risk Management → Reliability Principles

​

Pre-Deploy Quality Gates → Testing, Logging & Versioning

​

Post-Deploy Monitoring → Caching & Observability

​

Networking Under the Hood → OS Fundamentals

​

WebSocket Deployment at Scale → Real-Time Systems

​

Gateway Versioning and Deployment → API Gateways & Service Mesh

​

Container and Serverless Deployment → Cloud Service Patterns

​

Interview Deep-Dive Questions

​

1. Walk me through what happens when a user types a URL into a browser and presses Enter. Go as deep as you can.

• Browser cache check. The browser first checks its own DNS cache for a cached A/AAAA record. If there is a valid entry (TTL has not expired), it skips DNS entirely. Then it checks the HTTP cache — if there is a cached response for this URL with a valid Cache-Control header (max-age not exceeded), the browser may render the page without any network request at all (a “cache hit” that returns a 200 from disk cache). If the cache entry is stale, the browser sends a conditional request with If-None-Match (ETag) or If-Modified-Since, and the server can respond with 304 Not Modified to save bandwidth.
• DNS resolution. If no cached DNS entry exists, the browser asks the OS stub resolver, which checks its own cache (on Linux, this may be systemd-resolved or nscd). If that misses, the query goes to the configured recursive resolver (ISP resolver, or something like 8.8.8.8 or 1.1.1.1). The recursive resolver performs iterative lookups: root nameserver returns a referral to the TLD nameserver (e.g., .com), the TLD nameserver returns a referral to the authoritative nameserver for the domain, and the authoritative nameserver returns the actual IP address. Each layer caches the result according to the TTL. For a domain behind Cloudflare, the authoritative nameserver returns an Anycast IP, so the user is routed to the nearest edge PoP by BGP.
• TCP handshake. The browser initiates a TCP connection to the resolved IP on port 443 (HTTPS). This is the three-way handshake: SYN, SYN-ACK, ACK. On modern kernels, TCP Fast Open (TFO) can send data in the SYN packet for repeat connections, saving one round-trip. If there is a load balancer in front of the server, the TCP handshake terminates at the load balancer (for L7/ALB) or passes through (for L4/NLB).
• TLS handshake. Over the established TCP connection, the browser and server negotiate TLS. With TLS 1.3, this is a single round-trip: the client sends a ClientHello with supported cipher suites and a key share; the server responds with its certificate, chosen cipher suite, and its key share. The browser verifies the certificate chain against its trust store, checks for revocation (OCSP stapling avoids a separate network call here), and both sides derive the session keys. For repeat connections, TLS 1.3 supports 0-RTT resumption — the browser can send encrypted application data in the first flight, though this has replay attack risks and is typically limited to idempotent GET requests.
• HTTP request. The browser sends an HTTP/2 (or HTTP/3 over QUIC) request. With HTTP/2, the request is multiplexed over the single TCP connection as a binary frame. The request includes the method (GET), the path, the Host header, cookies, Accept headers, and any Authorization tokens. If this is a CORS preflight (cross-origin POST with custom headers), the browser sends an OPTIONS request first.
• Server processing. The request hits the reverse proxy or API gateway (NGINX, Envoy, or a cloud ALB), which terminates TLS, applies rate limiting, routes based on the path, and forwards to the appropriate backend. The backend application processes the request — authentication, authorization, business logic, database queries — and returns an HTTP response with status code, headers, and body.
• Rendering. The browser receives the HTML response and begins parsing. It constructs the DOM, encounters CSS and JS references, and makes additional requests (multiplexed over the same HTTP/2 connection). CSS blocks rendering; JS blocks parsing (unless async or defer). The browser builds the CSSOM, combines it with the DOM into a render tree, performs layout, paint, and compositing. First Contentful Paint (FCP) happens when the first DOM content renders. Largest Contentful Paint (LCP) is when the largest content element renders — this is a Core Web Vital that Google uses for search ranking.

​

2. Explain the difference between Layer 4 and Layer 7 load balancers. When have you chosen one over the other, and why?

• Layer 4 operates at the transport layer — it sees TCP/UDP packets and makes routing decisions based on source/destination IP and port. It does not inspect the contents of the packets. Think of it as a traffic cop directing cars based on license plate numbers without knowing what is inside the car. Because it does not parse the application protocol, it is extremely fast and handles very high connection rates with minimal CPU overhead. AWS NLB, HAProxy in TCP mode, and MetalLB are L4 load balancers.
• Layer 7 operates at the application layer — it terminates the connection, parses the full HTTP request (URL path, headers, cookies, body), and makes routing decisions based on that content. Think of it as a concierge who reads your request, understands what you are asking for, and directs you to the right department. This is slower because it has to parse every request, but it is enormously more flexible. AWS ALB, NGINX, Envoy, and Traefik are L7 load balancers.
• When I choose L4: Database connection pooling is the clearest case. PgBouncer or a MySQL proxy needs raw TCP connections passed through — you cannot have an L7 balancer trying to parse PostgreSQL wire protocol as HTTP. I have also used L4 for gRPC services where we wanted the client-side load balancing to handle stream-level distribution and just needed the NLB to distribute initial TCP connections across backend pods. Another case: extremely high-throughput services (100K+ requests/second) where the L7 parsing overhead was measurable in our latency budget — we moved to NLB and handled path routing at the application layer instead.
• When I choose L7 (which is most of the time): Any HTTP/HTTPS service where I need path-based routing (/api/* to backend services, /static/* to a CDN origin), SSL/TLS termination (offloading crypto from the backends), header-based routing (A/B testing, canary by header), or connection multiplexing (one client connection fanned out to multiple backend connections). The latency overhead of L7 (typically 1-3ms) is insignificant for most applications, and the operational benefits — seeing full request metrics, injecting headers, doing request-level rate limiting — are enormous.
• The key trade-off to articulate: L4 is faster and simpler but blind. L7 is slower but smart. Default to L7 for HTTP workloads because the visibility and routing intelligence pay for themselves. Use L4 only when you genuinely need raw TCP/UDP passthrough or the L7 overhead is measurably impacting your latency budget.

​

3. You are migrating a high-traffic service from one cloud region to another. The service receives 50K requests/second. How do you execute this migration with zero downtime?

• Phase 0 — Prepare the destination. Before touching any traffic, stand up the full stack in the new region: compute (same instance types/container specs), database (read replica promoted to primary, or a new primary with data synced), caches (pre-warmed if possible, or accept a cold-cache period), dependent services, monitoring, and alerting. Run the full test suite against the new region’s stack. This phase can take days or weeks — do not rush it.
• Phase 1 — Lower DNS TTLs. Weeks before migration, lower DNS TTLs from whatever they are (often 3600 seconds) down to 60 seconds. You need to do this at least 2x the old TTL in advance so that all caches worldwide flush and pick up the low-TTL records. This ensures that when you change DNS later, the old records expire quickly.
• Phase 2 — Enable dual-write for data. Set up cross-region database replication (e.g., PostgreSQL logical replication, MySQL binlog-based replication, or DynamoDB Global Tables). All writes go to the old region’s primary and replicate to the new region asynchronously. Monitor replication lag — it should be under 1 second. For caches (Redis, Memcached), you have two options: dual-write from the application or accept that the new region’s cache will be cold and warm up under traffic.
• Phase 3 — Shift traffic gradually using DNS weighting. Use Route 53 weighted routing (or equivalent) to send 5% of traffic to the new region. Monitor closely: error rates, latency, data consistency (are reads in the new region seeing all the writes?). If replication lag causes stale reads, you may need to route writes specifically to the old region while reads can go to either. Increase to 25%, then 50%, then 75%, monitoring at each stage. This is effectively a canary deployment at the infrastructure level.
• Phase 4 — Promote the new region to primary. Once 100% of traffic is in the new region and metrics are stable, promote the new region’s database to primary (if it was a replica). Reverse the replication direction so the old region becomes the replica (this is your rollback safety net). Update DNS to remove the old region entirely. Keep the old region running for at least a week as a fallback.
• Phase 5 — Decommission the old region. After the bake period with no issues, stop replication, tear down old region infrastructure, and raise DNS TTLs back to normal.
• The critical gotchas to mention: Data consistency during the transition — if a user writes to the old region and then reads from the new region before replication catches up, they see stale data. For strong consistency requirements, you may need to route all writes through the old region until the cutover is complete. Also, long-lived connections (WebSockets, gRPC streams) will not follow DNS changes — you need connection draining in the old region that gracefully closes existing connections and lets clients reconnect (to the new region per updated DNS). And third-party webhooks or partner integrations that have hardcoded IPs or do not respect DNS TTLs will keep hitting the old region — you need to identify these early and coordinate separately.

​

4. Your team is debating whether to use blue-green or canary deployment for a critical payment service. What factors drive your recommendation?

• The way I think about this is to start with the constraints of a payment service specifically. Payments have three properties that dominate the deployment decision: (1) correctness is more important than availability (a wrong charge is worse than a brief outage), (2) the blast radius of a bug is measured in money, not just errors, (3) regulatory requirements (PCI-DSS, SOX) often mandate specific audit trails and approval workflows for production changes.
• My recommendation: use both, in layers. This is not a dodge — it is how mature payment systems actually work. Blue-green gives you the instant rollback guarantee that a payment service needs. Canary gives you the real-traffic validation that smoke tests cannot provide. Here is how I would layer them:
  • Layer 1: Blue-green for the infrastructure cutover. Deploy the new version to the Green environment. Run a comprehensive synthetic test suite against Green that exercises every payment path (card charges, refunds, partial captures, 3DS flows, webhook processing). This catches configuration errors, missing environment variables, and basic logic bugs before any real money is involved. The Blue environment stays fully operational as an instant rollback target.
  • Layer 2: Canary for real-traffic validation. After Green passes smoke tests, route 1% of real traffic to Green using weighted routing at the load balancer (or Istio traffic splitting if you are on a service mesh). Monitor not just error rates and latency, but business metrics: charge success rate, average transaction amount, refund processing time, webhook delivery rate, reconciliation accuracy. Compare canary metrics against the baseline Blue environment using statistical analysis. Promote gradually: 1% for 30 minutes, then 5% for 30 minutes, then 25%, then 50%, then 100%.
  • Layer 3: Feature flags for logic changes. Any change to the payment processing logic itself (new payment method, changed authorization flow, updated fraud scoring) is behind a feature flag. The deploy ships dormant code. The flag is enabled after the canary phase passes, as a separate step. This means even if the canary looks good, the new logic is not live until explicitly activated.
• Why not canary alone? Canary does not give you sub-second rollback. If the canary at 50% traffic starts producing incorrect charges, rolling back means re-routing traffic, which takes seconds to minutes depending on your infrastructure. With blue-green, the rollback is a load balancer switch — effectively instant. For a payment system, those seconds matter.
• Why not blue-green alone? Blue-green validates against smoke tests, not real traffic. Smoke tests cannot exercise every payment method, every card issuer’s behavior, every edge case in currency conversion, or the interaction between your code and the payment processor’s real API (which may behave differently from your sandbox). Canary catches issues that only appear under real-world traffic patterns.
• The database complication: Both approaches share the hardest problem — both Blue and Green environments hit the same database. Payment schema changes must use expand-and-contract migrations deployed days before the code change. For a payment service, I would add an additional constraint: no schema migration and code deploy in the same week. The blast radius of getting that wrong in a payment system is too high.

​

5. Explain the expand-and-contract migration pattern. Why is it necessary, and where have you seen teams get it wrong?

• The core idea is deceptively simple: never make a breaking schema change in a single step. Instead, split it into three separately deployed phases. Phase 1 (expand) adds the new structure alongside the old — add columns, add tables, add indexes. Both old and new application code work with the expanded schema. Phase 2 (migrate) deploys application code that dual-writes to both old and new structures and backfills historical data. Phase 3 (contract) removes the old structure once you are confident everything reads from the new one.
• Why it is necessary: During a rolling deployment, both the old and new versions of your application run simultaneously against the same database. If you drop a column in the same deploy that removes the code using it, the old instances (still running during rollout) will crash trying to read that column. If you need to roll back, the old code cannot run against the new schema. Expand-and-contract ensures that at every point in the process, any version of the application code (current, new, or rolled-back) works with the current schema.
• Where I have seen teams get it wrong:
  • Squeezing all three phases into one deploy. “It’s a small change, just add the column and update the code.” This works fine until you need to roll back, and the rolled-back code does not know about the new column or, worse, the column was dropped as part of the “contract” and the old code crashes. The discipline is to always use separate deploys, even for “small” changes.
  • Forgetting the backfill. The team adds a new column (expand) and updates the code to write to it (migrate), but never backfills the 50 million existing rows. Months later, someone queries the new column assuming it is populated and gets nulls for everything created before the migration. Then they backfill in a rush, lock the table for 20 minutes during peak traffic, and cause an outage.
  • Backfilling without rate limiting. A naive UPDATE users SET new_col = computed_value WHERE new_col IS NULL on a 100M row table generates enormous write volume, overwhelms replication, and causes read replicas to fall behind by minutes. The correct approach is batched updates with sleeps between batches, monitoring replication lag throughout.
  • Never doing the contract phase. The team adds the new column, starts using it, but never removes the old column. Over years, the schema accumulates dozens of deprecated columns that confuse new engineers, bloat row sizes, and make the ORM layer a minefield. I have seen tables with five email-related columns where nobody knew which one was authoritative. Set a cleanup date when you create the expand migration and enforce it.
  • Adding NOT NULL without a default. In PostgreSQL before version 11, ALTER TABLE ADD COLUMN ... NOT NULL DEFAULT 'foo' rewrites the entire table, locking it for the duration. Even in PG 11+ where the default is stored in the catalog, adding NOT NULL on an existing column without a default requires checking every row, which takes a full table scan with a lock. The safe pattern: add the column as nullable, backfill, then add the NOT NULL constraint using ALTER TABLE ... ADD CONSTRAINT ... NOT VALID followed by VALIDATE CONSTRAINT (which does not hold an exclusive lock).

​

6. What is the difference between deployment and release? Why does this distinction matter?

• Deployment is putting new code on servers. Release is making that code visible to users. In traditional workflows, these happen at the same time — you deploy a new binary and users immediately see the new behavior. Modern release engineering separates them completely. You can deploy code that is not released (hidden behind a feature flag that is off). You can release code that was deployed weeks ago (turning on a flag). And critically, you can un-release without un-deploying (turning the flag off instantly, no rollback needed).
• Why this distinction is powerful: It decouples two different kinds of risk. Deploy risk is “will the new binary start, connect to its dependencies, and handle requests without crashing?” Release risk is “will the new feature behave correctly for users, perform well at scale, and deliver the expected business outcome?” By separating them, you can address each independently. A deploy that passes health checks but has a feature bug can be un-released in under a second via a flag toggle, without any deployment pipeline, without rolling back, and without affecting other features in the same binary.
• In practice, this looks like: An engineer merges code to main on Monday. CI/CD builds and deploys the new version to production on Monday afternoon. The feature is behind a flag, default off. The deploy is boring — no new behavior is exposed. On Wednesday, after the team reviews the deploy metrics and confirms the binary is stable, the flag is enabled for 5% of users. On Thursday, it is at 50%. On Friday, it is at 100%. If at any point the feature misbehaves, the flag is toggled off — no deploy, no rollback, sub-second recovery. The following week, the flag is cleaned up and the branching code is removed.
• Where this matters most in interviews: When discussing deployment strategies, explicitly calling out this distinction signals maturity. “We deploy behind flags, so the deploy itself is a no-op from the user’s perspective. The release is a separate, controlled, instantly reversible decision.” This is how Netflix, GitHub, Facebook, and every high-velocity engineering org operates. Teams that conflate deploy and release tend to deploy less often (because each deploy is scary), which makes deploys bigger, which makes them scarier — a death spiral.

​

7. Your Kubernetes pods are getting SIGKILL’d during rolling deployments, causing dropped requests. How do you diagnose and fix this?

• The root cause is almost always a timing mismatch. When Kubernetes terminates a pod during a rolling update, two things happen simultaneously: (1) the kubelet sends SIGTERM to the container, and (2) the Endpoints controller removes the pod from the Service’s endpoint list. The problem is that the load balancer (kube-proxy, or a cloud load balancer like ALB) takes a few seconds to propagate the endpoint removal. During those seconds, the load balancer is still sending new requests to a pod that is already shutting down.
• Diagnosis steps:
  1. Check if the application handles SIGTERM. Many applications (especially Node.js, Python, and Java apps not configured for graceful shutdown) ignore SIGTERM and continue running until Kubernetes sends SIGKILL at terminationGracePeriodSeconds. The SIGKILL kills in-flight requests with no cleanup. Run kubectl describe pod <pod> and look for last state: Terminated, reason: OOMKilled or Error, exit code: 137 (137 = SIGKILL = 128 + 9).
  2. Check terminationGracePeriodSeconds. The default is 30 seconds. If your application needs more time to drain in-flight requests (e.g., long-running report generation, file uploads, streaming responses), it will be SIGKILL’d before finishing. Increase this to match your longest expected request duration plus a buffer.
  3. Check for a preStop hook. This is the most commonly missing piece. Without a preStop hook, the pod starts shutting down immediately on SIGTERM, but the load balancer has not yet removed it from the endpoint list. New requests arrive at a pod that is shutting down. The fix: add a preStop hook with a sleep of 5-10 seconds. This gives the load balancer time to deregister the pod before the application starts its shutdown sequence.
  4. Check readiness probe configuration. When the pod receives SIGTERM, it should immediately start failing its readiness probe (return 503 on /health). This signals the Endpoints controller to remove it from the Service. If the readiness probe keeps passing during shutdown, the pod stays in the endpoint list longer than it should.
• The fix is a coordinated configuration:

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "10"]
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      periodSeconds: 5
      failureThreshold: 1

• The sequence with this fix: SIGTERM arrives. The preStop hook sleeps 10 seconds (giving the load balancer time to deregister). Meanwhile, the readiness probe starts failing. After 10 seconds, the application’s SIGTERM handler runs: it stops accepting new connections, drains in-flight requests, and exits. If it has not exited by 60 seconds (terminationGracePeriodSeconds), Kubernetes sends SIGKILL.

​

8. Compare Server-Sent Events (SSE), WebSockets, and long-polling. You are building a live dashboard that shows stock prices updating every 500ms. Which do you choose and why?

• For a stock price dashboard with 500ms updates and no client-to-server data flow, I would choose SSE. Here is why:
• The requirement is unidirectional: The server pushes price updates to the client. The client never sends data back (no chat messages, no commands, no interactive state). SSE is purpose-built for this pattern — server-to-client event streaming over a standard HTTP connection.
• SSE advantages for this use case:
  • Built-in reconnection. If the connection drops (mobile network switch, proxy timeout), the browser automatically reconnects and sends a Last-Event-ID header so the server can resume from where the client left off. With WebSocket, you have to build reconnection logic yourself.
  • Works through proxies and CDNs. SSE is plain HTTP, so it works through corporate proxies, CDNs, and firewalls that might block or interfere with WebSocket’s upgrade handshake. For a stock dashboard used by traders behind corporate firewalls, this is a real operational benefit.
  • Simpler infrastructure. SSE connections are standard HTTP connections. Your existing HTTP load balancers (ALB, NGINX) handle them without special configuration. WebSocket requires L4 or WebSocket-aware L7 load balancing, sticky sessions for connection affinity, and careful handling of the upgrade handshake.
  • Easier to scale. SSE is naturally compatible with HTTP/2 multiplexing — many SSE streams over a single TCP connection. The infrastructure is stateless from the load balancer’s perspective. With WebSocket, each connection is stateful and bound to a specific backend server, requiring a pub/sub layer and connection registry to scale.
• When I would switch to WebSocket instead:
  • If the dashboard becomes interactive — users can place trades, set alerts, or send messages — then the client needs to send data to the server, and WebSocket’s bidirectional channel becomes necessary.
  • If update frequency drops below 100ms and latency is critical (high-frequency trading visualization), WebSocket’s lower per-message overhead matters. SSE has a small framing overhead per event that is negligible at 500ms but adds up at sub-100ms intervals.
  • If you need binary data streaming (WebSocket supports binary frames natively; SSE is text-only and would require Base64 encoding, which bloats the payload by ~33%).
• Why not long-polling? At 500ms update intervals, long-polling would mean establishing a new HTTP connection every 500ms (or close to it). That is 2 connections per second per client. With 10K concurrent users, that is 20K connections/second of overhead just for the polling mechanism. The connection setup cost dominates the actual data transfer. SSE and WebSocket both maintain a persistent connection, avoiding this overhead entirely.

​

9. Your team uses GitOps with ArgoCD. A developer pushes a configuration change that passes CI but takes down production when ArgoCD syncs it. How do you prevent this from happening again?

• The root problem is that GitOps auto-sync treats “merged to the config repo” as “approved for production deployment,” but merging to a repo is a code review gate, not a production safety gate. CI can validate syntax and schema, but it cannot predict how a configuration change will interact with live traffic, real data, and production-scale load.
• Layer 1: Strengthen what CI can catch.
  • Schema validation. Every Kubernetes manifest, Helm values file, and Kustomize overlay should be validated in CI with kubeval or kubeconform to catch invalid YAML, unknown API fields, and deprecated API versions. This catches typos and obvious errors.
  • Policy enforcement. Use Open Policy Agent (OPA) / Gatekeeper or Kyverno policies in CI to enforce organizational rules: resource limits must be set, liveness/readiness probes must exist, no containers running as root, image tags must be pinned (no latest). If the bad config change violated a policy (e.g., removed a readiness probe), this gate would have caught it.
  • Dry-run / diff preview. CI should run kubectl diff or ArgoCD’s app diff against the target cluster to produce a human-readable diff of what will change. This diff should be posted as a comment on the PR so reviewers can see the actual Kubernetes resource changes, not just the Kustomize overlay changes. Many misconfigurations are obvious in the diff but invisible in the source YAML.
• Layer 2: Do not auto-sync high-risk changes.
  • Configure ArgoCD with auto-sync for low-risk changes (image tag updates, replica count changes) but manual sync for high-risk changes (resource limit changes, new Ingress rules, RBAC changes, CRD modifications). ArgoCD’s sync policy and resource-level annotations can control this.
  • For changes that affect traffic routing, security, or resource allocation, require an explicit argocd app sync command (or button click) after a human reviews the diff in the ArgoCD UI.
• Layer 3: Progressive sync with canary.
  • Use Argo Rollouts or Flagger alongside ArgoCD. Instead of ArgoCD applying the new configuration to all pods at once, the Rollout controller applies it as a canary: 5% of pods get the new config, metrics are compared against baseline, and the change is promoted or rolled back automatically. This is how you get production-traffic validation for configuration changes, not just code changes.
• Layer 4: Blast radius reduction.
  • If you manage multiple clusters or environments, sync to a staging cluster first. ArgoCD supports wave-based sync across clusters: staging syncs immediately, production syncs only after staging has been stable for a configurable soak period (e.g., 30 minutes).
  • Within a single cluster, use ArgoCD’s sync waves to apply configuration changes in a controlled order: non-critical services first, then critical services, with a health check gate between waves.
• Layer 5: Fast rollback.
  • Enable ArgoCD’s rollback capability (auto-rollback on sync failure). But more importantly, ensure the team has practiced git revert as the standard rollback mechanism. In GitOps, rollback is a Git operation — revert the PR, merge it, and ArgoCD syncs the reverted state. This is fast, auditable, and goes through the same review process.

​

10. You are designing the CI/CD pipeline for a new microservices platform with 15 services. What does the pipeline architecture look like, and how do you handle cross-service dependencies?

• The foundational principle is independent deployability. Each of the 15 services should have its own CI/CD pipeline, its own test suite, its own deployment cadence, and its own version. If deploying Service A requires coordinating with Service B, you have a distributed monolith, not microservices. The pipeline architecture should enforce and enable this independence.
• Per-service pipeline (the inner loop):
  • Trigger: Any push to the service’s directory in the monorepo (or its dedicated repo if using multi-repo). Use path filters in GitHub Actions or only: changes: in GitLab CI to avoid triggering all 15 pipelines on every commit.
  • Stage 1 (30 seconds): Lint + static analysis. Run linters, type checking, and security scanning (Snyk, Trivy for container images). Fail fast on obvious issues.
  • Stage 2 (2-3 minutes): Unit tests + build. Run unit tests in parallel. Build the container image and tag it with the git SHA (e.g., order-service:abc123def). Push to the container registry.
  • Stage 3 (5-10 minutes): Integration tests. This is where cross-service dependencies are tested. Spin up the service’s direct dependencies (databases, message queues) as containers using Docker Compose or Testcontainers. For downstream service dependencies, use contract-based testing (not running the actual service).
  • Stage 4 (10-15 minutes): Deploy to staging + smoke tests. Deploy the new image to a shared staging environment. Run end-to-end smoke tests that exercise the service’s critical paths in the context of the full system. This is the only stage where the service interacts with other real services.
  • Stage 5: Deploy to production. Canary rollout via Argo Rollouts. 5% traffic for 10 minutes, then 25%, then 100%, with automated rollback on error rate or latency degradation.
• Cross-service dependency management (the hard problem):
  • Contract testing with Pact or similar. Each service defines the API contracts it depends on (as a consumer) and the contracts it provides (as a provider). When Service A changes its API, the provider contract test runs against all known consumers. If any consumer would break, the CI pipeline fails before the change is deployed. This catches breaking changes without requiring all services to be deployed together.
  • API versioning. All services expose versioned APIs (/v1/orders, /v2/orders). New versions are additive — old endpoints continue to work. This ensures that Service B running version 1.5 can talk to Service A whether it is running version 2.3 or 2.4. Never remove an API version until all consumers have migrated.
  • Shared library management. If multiple services share a library (common data models, auth middleware, logging), publish it as a versioned package (npm, Maven, PyPI). Services pin to a specific version. Updating the shared library is a two-step process: publish the new library version, then update each service’s dependency individually. Never use floating version ranges (^1.0.0) for shared libraries in production — a library update should be an explicit, tested change per service.
  • Database per service. Each service owns its database. No cross-service database queries. If Service A needs data from Service B, it calls Service B’s API. This eliminates the worst class of cross-service coupling: shared database schemas where a migration in one service breaks another.
• The cross-cutting pipeline (the outer loop):
  • Nightly end-to-end test suite. Runs against the staging environment with all services at their latest deployed versions. Exercises full user journeys (sign up, browse, add to cart, checkout, receive confirmation). This catches integration failures that contract tests miss — timing issues, data format edge cases, behavior under concurrent load.
  • Dependency graph visualization. Maintain a service dependency graph (auto-generated from contract tests or service mesh telemetry). Before deploying a change to a foundational service (auth, user service, payment service), the pipeline checks the dependency graph and optionally triggers smoke tests for downstream services.

​

11. A latency-sensitive service occasionally sees DNS resolution times spike to 5 seconds. This happens randomly and affects about 1% of requests. How do you investigate and fix this?

• First, I would characterize the problem precisely. 5-second DNS spikes on 1% of requests is a classic pattern that points to one of a few specific causes. I would instrument DNS resolution time as a metric (most languages have a way to hook into the resolver — Go has net.Resolver with custom dialers, Java has custom NameResolver implementations). Correlate the spikes with: time of day, specific domains being resolved, specific pods/instances, and whether the spikes cluster or are evenly distributed.
• Most likely cause 1: DNS cache miss hitting a slow upstream resolver. The application (or the OS) caches DNS results according to TTL. When the cache entry expires, the next request to that domain must perform a live DNS resolution. If the upstream resolver (kube-dns/CoreDNS in Kubernetes, or the VPC resolver in cloud environments) is overloaded or the authoritative nameserver is slow, that resolution takes seconds instead of milliseconds. The 1% pattern fits because only the first request after each cache expiry pays the resolution cost — subsequent requests use the refreshed cache.
  • Fix: Configure DNS caching at the application level with a minimum TTL floor (e.g., never cache for less than 30 seconds, even if the record’s TTL is 0). In Kubernetes, tune CoreDNS caching settings. Use dnsConfig in the pod spec to set ndots: 1 (to avoid unnecessary search domain appending, which multiplies DNS queries).
• Most likely cause 2: The ndots problem in Kubernetes. By default, Kubernetes sets ndots: 5 in /etc/resolv.conf. This means any domain with fewer than 5 dots is appended with search domains first. A lookup for api.example.com (2 dots, less than 5) first tries api.example.com.default.svc.cluster.local, then api.example.com.svc.cluster.local, then api.example.com.cluster.local, then finally api.example.com. That is 4 DNS queries before the real one. Each failed query adds latency. If any of those intermediate queries is slow (overloaded CoreDNS), the total resolution time spikes.
  • Fix: Set ndots: 1 in the pod’s dnsConfig for services that primarily call external domains. Or use fully qualified domain names with a trailing dot (api.example.com.) to bypass the search domain appending entirely.
• Most likely cause 3: Resolver overload. In a large Kubernetes cluster, CoreDNS handles DNS for every pod. If you have hundreds of pods making frequent DNS queries, CoreDNS can become a bottleneck. CoreDNS pods have limited CPU/memory, and under load, queries queue up.
  • Fix: Scale CoreDNS horizontally (increase replica count). Enable NodeLocal DNSCache (a DaemonSet that runs a DNS cache on every node, intercepting DNS queries before they hit CoreDNS). This dramatically reduces the load on CoreDNS and provides sub-millisecond DNS resolution for cached entries.
• Less likely but worth checking: The application is using a DNS resolver library that does not cache (every request triggers a fresh resolution), connection-level DNS where each new TCP connection resolves DNS (not using connection pooling), or the authoritative nameserver itself is slow (check with dig @authoritative-ns example.com to measure response time directly).

​

12. Walk me through how you would set up deployment observability from scratch for a team that currently has none. What do you instrument first, and why?

• The way I think about this is in three tiers, where each tier unlocks a capability that the team does not have today. You ship each tier before starting the next, so the team gets value immediately rather than waiting months for a comprehensive system.
• Tier 1 (Week 1-2): Deploy markers + the golden signals. This is the minimum viable deployment observability.
  • Deploy annotations. Every deployment automatically creates an annotation/event in your monitoring system (a vertical line on graphs in Grafana, a deploy marker in Datadog) with the commit SHA, deployer, and a link to the diff. This single addition transforms debugging from “something went wrong at some point” to “something went wrong 3 minutes after deploy abc123.” This is the highest ROI observability investment you can make — it costs almost nothing and immediately correlates metric changes with deploys.
  • The four golden signals (from the Google SRE book): Latency (request duration — track p50, p95, p99), traffic (request rate — requests/second), errors (error rate — 5xx responses / total responses), and saturation (resource utilization — CPU, memory, connection pool usage). Instrument these for every service endpoint. Use Prometheus with a standard metrics library (micrometer for Java, prometheus-client for Python, prom-client for Node). Create one Grafana dashboard per service with these four signals.
  • Basic alerting. Set up two alerts: error rate > 5% for 3 minutes, and p99 latency > 2x the 7-day average. That is it for Tier 1. These two alerts catch the majority of deployment-caused regressions.
• Tier 2 (Week 3-4): Business metrics + structured logging.
  • Business metrics on the deployment dashboard. Identify the 2-3 metrics that represent “the system is working correctly from the user’s perspective.” For an e-commerce platform: orders per minute, checkout conversion rate, payment success rate. For a SaaS product: sign-ups per minute, API calls per minute, feature usage rates. Add these to the same Grafana dashboard alongside the golden signals. Now the team can see technical health AND business health in one view.
  • Structured logging with correlation IDs. Switch from unstructured text logs to JSON-structured logs with fields: timestamp, level, service, request_id, user_id, endpoint, duration_ms, status_code. Inject a correlation ID at the API gateway that flows through every service in the request path. Ship logs to a centralized system (Loki, Elasticsearch, Datadog Logs). Now when a deployment causes a new type of error, you can search logs by error message across all services and trace the request path.
• Tier 3 (Week 5-8): Distributed tracing + automated canary analysis.
  • Distributed tracing (OpenTelemetry). Instrument services with OpenTelemetry to produce traces that show the full journey of a request across services. This is critical for debugging performance regressions that span multiple services — “this endpoint got 200ms slower, and the trace shows the extra time is in the downstream inventory service, which is making a new database query that was not there before.” Ship traces to Jaeger, Tempo, or Datadog APM.
  • Automated canary analysis. Integrate deployment metrics comparison into the CD pipeline. When a canary deploys, automatically compare its golden signals against the baseline. Argo Rollouts with a Prometheus metrics provider can do this. Set promotion/rollback thresholds. This is the capstone — the team now has a system that automatically detects and rolls back bad deployments without human intervention.
• Why this order matters: Tier 1 is achievable by a single engineer in a week and immediately makes every subsequent deployment safer. Tier 2 adds the business context that prevents “metrics are green but the product is broken” scenarios. Tier 3 adds the automation that makes high-velocity deployment sustainable. Trying to jump to Tier 3 without Tier 1 and 2 means you are building automated analysis on a shaky foundation — garbage in, garbage out.

​

Advanced Interview Scenarios

​

13. Your health check endpoint returns 200 OK, but the service is clearly broken — users cannot complete any transactions. How is this possible, and how do you design health checks to prevent it?

​

14. Your team introduces CORS headers for a new frontend domain, and it works in Chrome but not in Safari. API requests fail silently. What is happening?

Access-Control-Allow-Origin: https://app.example.com  (echo the specific origin, never *)
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization, X-Request-ID
Access-Control-Max-Age: 7200  (2 hours — Chrome's cap, so going higher only helps Firefox/Safari)
Vary: Origin

ALLOWED_ORIGINS = {
    "https://app.example.com",
    "https://admin.example.com",
    "https://partner.example.com",
}

def cors_middleware(request, response):
    origin = request.headers.get("Origin")
    if origin in ALLOWED_ORIGINS:
        response.headers["Access-Control-Allow-Origin"] = origin
        response.headers["Vary"] = "Origin"  # CRITICAL — tells caches the response varies by origin

​

15. You are running a microservices architecture with Consul for service discovery. A service starts returning errors because it is routing traffic to instances that were terminated 5 minutes ago. Walk me through the failure.

​

16. Your canary deployment shows BETTER metrics than baseline — 20% lower latency, 50% fewer errors. Should you promote it? The obvious answer is yes. The correct answer is “it depends, and probably investigate first.”

​

17. You join a team that deploys once every two weeks. They are at Level 2 on the CI/CD maturity model — automated tests, manual deploys. Leadership wants daily deployments within 6 months. How do you get there?

​

18. During a blue-green deployment cutover, you switch traffic to Green and immediately get reports of users seeing a mix of old and new UI — some pages show the old version, some show the new version in the same session. What is happening?

​

19. Your API gateway is a single point of failure. It is handling 30K requests/second across all your microservices. Design the resilience strategy.

​

20. Your team has 147 active feature flags. A production incident occurs, and during investigation you discover two flags are interacting in a way nobody anticipated. How do you triage this, and how do you prevent flag interaction bugs going forward?

​

21. You have a service running behind an AWS ALB. Requests are distributed across 10 instances using round-robin. Response times are highly variable — p50 is 15ms but p99 is 2.5 seconds. One engineer suggests switching to least-connections. Another says the problem is not the algorithm. Who is right?

​

22. You need to deploy a breaking API change — v1 to v2 — for a public API with 500 external consumers. Some consumers take months to update their integrations. How do you manage this without breaking anyone?