• Q: Name a trade-off where both options are bad. A: Distributed transactions — two-phase commit gives you consistency but blocks on the slowest participant; eventual consistency gives you speed but requires conflict resolution. You pick your poison based on the business cost of each failure mode.
• Q: When is “no trade-off” the right answer? A: When the decision is trivially reversible. Choosing a logging format is cheap to change — do not over-analyze it. Choosing a database engine is expensive to change — analyze deeply.
• Q: What is the most expensive trade-off mistake you can make? A: Choosing the wrong consistency model for financial data. You cannot un-double-charge a customer.

• Q: Your system handles 1K RPS today. Name three things that will break at 10K. A: Database connection pool (finite connections), DNS TTL cache churn (more unique clients), and log volume (10x writes to disk or log aggregator may throttle).
• Q: When should you NOT plan for scale? A: When you have <100 users and no evidence of growth. Premature scaling infrastructure is the most expensive form of premature optimization.

• Q: Name three things you check in a security review of a new service. A: Trust boundary crossings (where does untrusted input enter?), secrets handling (hardcoded? environment variables? Vault?), and dependency audit (known CVEs in the supply chain).
• Q: What is the most dangerous security misconception? A: “We are too small to be a target.” Automated scanners do not care about your company size — they scan every IP on the internet.

• Q: Latency vs throughput — when do they conflict? A: Batching improves throughput (process 100 items at once) but increases latency for individual items (each waits for the batch to fill). Streaming reduces latency (process immediately) but may reduce throughput (overhead per item).
• Q: What is the first thing you check when an endpoint is slow? A: The database query plan (EXPLAIN ANALYZE). In my experience, 80% of slow endpoints are caused by missing indexes, N+1 queries, or full table scans.

• Q: You have 1 hour to add observability to a service with none. What do you add? A: Request rate, error rate, and P99 latency on the critical path (3 metrics + 1 dashboard), plus structured JSON logging with a correlation ID header.
• Q: What is the difference between an SLI, SLO, and SLA? A: SLI is the measurement (P99 latency = 200ms). SLO is the target (P99 < 300ms, 99.9% of the time). SLA is the contract with consequences (if SLO is breached, customer gets credits).
• Q: When is high cardinality dangerous in observability? A: When you use unbounded values (user IDs, request IDs) as metric labels — this explodes your time series count and crashes Prometheus.

• Q: What must every log line in a distributed system contain? A: Timestamp, severity level, correlation/request ID, service name, and a human-readable message. Without the correlation ID, you cannot trace a request across services.
• Q: When should you NOT log something? A: When it contains PII (emails, passwords, credit card numbers), when it is at DEBUG level in production (noise), or when the volume would overwhelm your log aggregator (per-request body logging at 10K RPS).

• Q: What is the difference between a retryable and non-retryable error? A: Retryable (503, timeout, connection reset) means the operation might succeed on a second attempt. Non-retryable (400 bad request, 404 not found, business rule violation) means retrying will produce the same failure — stop immediately.
• Q: Why add jitter to exponential backoff? A: Without jitter, all clients that failed at the same time retry at the same time (thundering herd). Jitter randomizes the retry window so clients spread out.

• Q: What should happen if your service starts with invalid configuration? A: Fail fast and loud. A service that starts with a bad database URL and silently returns errors for 10 minutes is worse than a service that crashes on startup and triggers an immediate alert.
• Q: What is configuration drift? A: When the actual state of your infrastructure diverges from what your IaC (Terraform, CloudFormation) says it should be — usually because someone made a manual change via the console. Detect it with terraform plan in CI and alert on differences.

• Q: What is blast radius and why does it matter? A: The number of users or features affected when a component fails. A search service failure should not prevent users from checking out. Design boundaries so failures are contained.
• Q: Name a failure mode that monitoring will not catch. A: Silent data corruption — the system returns 200 OK but the data it writes is wrong. This requires data validation checks and reconciliation jobs, not just uptime monitoring.

• Q: Name the three biggest cloud cost traps. A: Data transfer between AZs/regions, idle resources in non-production environments running 24/7, and over-provisioned databases running at 5% utilization.
• Q: When is a more expensive tool the cheaper choice? A: When the operational burden of the cheap tool consumes more engineering hours than the price difference. A managed database at $500/month that requires zero ops time is cheaper than a self-hosted database at$100/month that requires 10 hours/month of DBA work.

• Q: What is the difference between blue-green and canary deployments? A: Blue-green is all-or-nothing with instant rollback (two identical environments, swap traffic). Canary is gradual — send 5% of traffic to the new version, watch metrics, then ramp up. Canary catches issues earlier with lower blast radius.
• Q: When is a feature flag better than a canary deploy? A: When the change is user-facing and you want to control which users see it (by segment, geography, or account), or when rollback needs to be instant without a redeploy.

• Q: You need to remove a field from your API response. What is the safe process? A: Mark it deprecated in documentation, add a sunset date, emit a deprecation warning header, monitor usage of the field (log when consumers read it), and only remove it after usage drops to zero or the sunset date passes.
• Q: How does Protobuf handle backward compatibility better than JSON? A: Protobuf uses field numbers, not names — adding new fields or deprecating old ones does not break existing consumers because unknown fields are silently ignored. JSON APIs require explicit versioning to achieve the same safety.

• Q: What is the practical difference between linearizability and serializability? A: Linearizability is about real-time ordering of individual operations across all clients. Serializability is about transactions appearing to execute in some serial order. A system can be serializable but not linearizable (transactions are ordered, but the order may not reflect wall-clock time).
• Q: Why are distributed transactions so expensive? A: Two-phase commit requires all participants to be available and agree — one slow or failed participant blocks everyone. This is why most modern systems prefer saga patterns with compensating transactions.
• Q: When would you use CRDTs instead of consensus? A: When availability is more important than strong ordering — collaborative editing, distributed counters, shopping cart merge. CRDTs guarantee convergence without coordination but limit the operations you can express.

• Q: A container keeps getting killed with no error logs. What happened? A: OOM Killer. The container exceeded its memory cgroup limit. Check dmesg for “Out of memory: Kill process” entries. Fix: increase the limit, find the memory leak, or tune GC settings.
• Q: What is the difference between a soft and hard file descriptor limit? A: Soft limit is the current effective limit (can be raised by the process up to the hard limit). Hard limit is the maximum (can only be raised by root). If ulimit -n returns 1024 and your service needs 10K connections, you will get “too many open files” errors under load.

• Q: What is a covering index and why does it matter? A: An index that contains all columns needed by a query, so the database never reads the actual table row (no heap fetch). This can make queries 10-100x faster for read-heavy workloads.
• Q: Why does DynamoDB throttle even when you have unused capacity? A: Because capacity is distributed across partitions. If one partition key is “hot” (receives disproportionate traffic), that partition throttles even if other partitions are idle. The fix is a well-distributed partition key, not more provisioned capacity.

• Q: When is Lambda more expensive than a container? A: When invocation frequency is high and steady. Lambda at 100 RPS continuously costs roughly 10x what a Fargate container doing the same work costs. Lambda wins for sporadic, bursty workloads; containers win for steady-state.
• Q: What changed about S3 consistency in 2020? A: S3 became strongly consistent for read-after-write and list operations. Before 2020, you could write an object and immediately get a 404 on a GET. This is no longer the case — S3 is now strongly consistent at no additional cost or latency.

• Q: When would you choose SSE over WebSocket? A: When data flows only from server to client (live feeds, notifications, dashboard updates). SSE is simpler — it uses standard HTTP, works through proxies without special configuration, and auto-reconnects natively in browsers.
• Q: What is the hardest problem in real-time collaborative editing? A: Conflict resolution when two users edit the same content simultaneously. Operational Transformation (OT, used by Google Docs) and CRDTs (used by Figma) are the two main approaches — OT is centralized and order-dependent, CRDTs are decentralized and commutative.

• Q: What do you do when you realize your system design has a flaw mid-interview? A: Call it out explicitly: “I just realized this approach has a problem with X. Let me revise.” Interviewers reward self-correction — it shows the same instinct you need in code review and incident response.
• Q: How do you handle a question you genuinely do not know the answer to? A: “I do not have direct experience with that, but here is how I would reason about it based on what I do know…” Then reason from first principles. A structured “I do not know but here is my thought process” is worth more than a confidently wrong answer.

• Q: OpenTelemetry vs vendor-specific instrumentation — when does it matter? A: OpenTelemetry matters when you might switch vendors (avoiding lock-in) or need to send telemetry to multiple backends simultaneously. Vendor-specific SDKs matter when you need deep, proprietary features (Datadog APM’s code-level profiling, Honeycomb’s high-cardinality exploration) that OTel does not fully support yet.
• Q: When is distributed tracing overkill? A: When you have a monolith or 2-3 services. Structured logs with correlation IDs give you 80% of the debugging value at 20% of the operational cost. Invest in tracing when you have >5 services and cross-service debugging takes >30 minutes.

• Q: PostgreSQL vs MySQL — when does the choice matter? A: PostgreSQL wins when you need complex queries (CTEs, window functions, JSONB), advanced data types, or strong standards compliance. MySQL wins for read-heavy workloads with simple queries where its simpler replication model and MyISAM-era optimizations still provide an edge. For most new projects, PostgreSQL is the safer default.
• Q: When is Redis a primary database vs a cache? A: Redis is a primary database when your data fits in memory, durability requirements are met by RDB/AOF persistence, and the data model maps to Redis data structures (leaderboards with sorted sets, session storage with hashes, rate limiting with counters). It is a cache when the source of truth lives elsewhere and Redis holds a disposable copy.
• Q: DynamoDB vs a relational database — what is the deciding factor? A: Access pattern predictability. If you can define all access patterns upfront, DynamoDB’s single-table design delivers single-digit millisecond latency at any scale. If access patterns are unknown or require ad-hoc queries, a relational database gives you flexibility DynamoDB cannot.

• Q: Kafka vs RabbitMQ — what is the fundamental difference? A: Kafka is a distributed log — messages are durable, replayable, and retained for a configurable period. RabbitMQ is a message broker — messages are delivered and then gone. Use Kafka when you need event sourcing, replay, or multiple consumers processing the same events. Use RabbitMQ when you need task queues, routing patterns, or request-reply.
• Q: When is SQS the right choice over Kafka? A: When you want zero operational overhead on AWS, do not need message replay, and your throughput is <10K messages/second. SQS is a managed queue with no infrastructure to run. Kafka gives you more power but requires cluster management (or MSK, which still needs tuning).
• Q: What does “exactly-once” mean in Kafka, and what are its limits? A: Kafka’s exactly-once guarantee applies within a Kafka transaction boundary (consume-process-produce within Kafka topics). The moment your consumer writes to an external system (a database, an API), you are back to at-least-once and need idempotency keys in the external system.

• Q: Terraform vs Pulumi — when does the choice matter? A: Terraform uses HCL (a DSL) which enforces declarative patterns but limits expressiveness. Pulumi uses real programming languages (TypeScript, Python) which allows loops, conditionals, and abstractions but can lead to imperative spaghetti if undisciplined. Choose Terraform for teams that value convention; choose Pulumi for teams that need programmatic infrastructure (dynamic environments, complex conditionals).
• Q: What is the biggest risk with IaC? A: State file corruption or divergence. Terraform’s state file is the source of truth for what exists in your cloud account. If it gets out of sync (manual console changes, failed applies, concurrent modifications), you can accidentally destroy production resources. Remote state with locking (S3 + DynamoDB) is mandatory for teams.

• Q: Docker vs Kubernetes — when do you need each? A: Docker is for packaging (building reproducible images). Kubernetes is for orchestration (running, scaling, and managing many containers). You always need Docker (or an equivalent). You only need Kubernetes when you have the team size and operational maturity to justify it — typically 15+ engineers with multiple services.
• Q: What does Kubernetes actually give you over ECS or Cloud Run? A: Portability (runs on any cloud), a rich ecosystem (Istio, ArgoCD, Prometheus), and fine-grained control (custom schedulers, operators, CRDs). The cost: operational complexity. If you do not need portability or the ecosystem, ECS/Cloud Run is simpler.

• src/server.c — The main event loop. Follow how a client connection becomes a command execution. This is a masterclass in event-driven architecture without callbacks or async/await.
• src/t_zset.c — The sorted set implementation using skip lists. One of the clearest skip list implementations you will find anywhere. Understand why Redis chose skip lists over balanced trees (simpler to implement, similar performance, easier to reason about concurrently).
• src/dict.c — The hash table implementation with incremental rehashing. Redis cannot block for a full rehash, so it does it one bucket at a time during normal operations. This is how you handle expensive maintenance operations in latency-sensitive systems.
• src/aof.c and src/rdb.c — Persistence strategies. Compare append-only file (durability) with RDB snapshots (compactness). The BGSAVE fork-based snapshot is a beautiful use of copy-on-write semantics.

• src/net/http/server.go — The HTTP server implementation. Follow how ListenAndServe creates a listener, accepts connections, and dispatches to handlers. The Handler interface (a single ServeHTTP method) is one of the most elegant interface designs in any language.
• src/sync/mutex.go and src/sync/waitgroup.go — Concurrency primitives. Compact, well-commented implementations that teach you how mutexes and wait groups actually work at the runtime level.
• src/encoding/json/decode.go — Reflection-based JSON decoding. A practical example of how Go uses reflection (sparingly) and why the performance trade-offs are acceptable for a standard library.
• src/context/context.go — The entire context package is under 500 lines. It is the canonical example of how to propagate cancellation, deadlines, and request-scoped values through a call chain.

• packages/react-reconciler/src/ReactFiberWorkLoop.js — The main work loop. Understand how React breaks rendering into units of work that can be paused and resumed. This is cooperative scheduling implemented in JavaScript.
• packages/react-reconciler/src/ReactFiberBeginWork.js — Where React decides what work to do for each fiber node. Follow how a state update propagates through the fiber tree.
• packages/react-reconciler/src/ReactChildFiber.js — The reconciliation (diffing) algorithm. Understand the heuristics: same type at same position means update, different type means unmount/remount, keys disambiguate reordering.
• packages/shared/ReactTypes.js — The type definitions reveal the mental model: everything is an element, elements form trees, trees are diffed, diffs become DOM mutations.

• kernel/sched/core.c — The CFS (Completely Fair Scheduler) core. Understand how the kernel decides which process runs next. The vruntime concept (virtual runtime that tracks how much CPU each process has consumed) explains why your latency-sensitive service sometimes gets preempted.
• mm/oom_kill.c — The OOM Killer. Under 500 lines. Read how oom_badness() scores processes for termination. This is the code that decides which of your containers dies when the node runs out of memory.
• net/core/sock.c — Socket fundamentals. Follow how SO_RCVBUF and SO_SNDBUF are set and enforced. This explains why your network-heavy service behaves differently under different buffer configurations.
• fs/eventpoll.c — The epoll implementation. This is the foundation of every high-performance event loop (Node.js, Nginx, Redis). Understand how epoll_wait avoids the O(n) scan that killed select and poll at scale.

• source/common/http/conn_manager_impl.cc — The HTTP connection manager. This is where every HTTP request enters Envoy. Follow how it flows through filter chains (the extension mechanism that makes Envoy composable).
• api/envoy/service/discovery/v3/ — The xDS API protobuf definitions. These define how Envoy receives dynamic configuration (routes, clusters, listeners, endpoints) from a control plane. Understanding xDS is understanding the lingua franca of modern service mesh architecture.
• source/common/upstream/cluster_manager_impl.cc — How Envoy manages upstream clusters, health checking, and load balancing. The circuit breaking and outlier detection logic here is what keeps service mesh traffic healthy.

​

Follow-up: How would you decide the threshold for your error rate alert?

​

Follow-up: Your team argues that they need distributed tracing before anything else because “we can’t debug without it.” How do you handle this?

​

Going Deeper: What is the difference between observability and monitoring, and why does the distinction matter?

​

Follow-up: How do you quantify technical debt to convince a product manager it needs to be addressed?

​

Follow-up: When would you argue against paying down technical debt, even when the team wants to?

​

Going Deeper: How does Conway’s Law apply to technical debt decisions?

​

Follow-up: The product team pushes back and says “but what about when we need to scale?” How do you respond?

​

Follow-up: A year later, the team has grown to 25 engineers and you need to extract your first service. How do you decide which one?

​

Follow-up: You have implemented a cache and the hit rate is only 25%. What do you investigate?

​

Follow-up: How do you handle cache invalidation in a microservices architecture where the service that writes data is different from the service that reads it?

​

Follow-up: You mentioned the circuit breaker was configured but never tested. How do you test resilience patterns in production?

​

Follow-up: How do you run a blameless postmortem? What makes a postmortem actually useful vs. a bureaucratic exercise?

​

Follow-up: How do you handle the case where a user writes data and then immediately reads it back, but the read hits a replica that has not received the write yet?

​

Follow-up: Can you give an example where choosing the wrong consistency model caused a real production problem?

​

Going Deeper: How do CRDTs change the consistency trade-off landscape?

• Data model and schema design. If I choose to store user activity as a flat table without a timestamp index, and the primary query is “show me the last 30 days of activity,” I have built a full table scan into the architecture. Fixing this later means a data migration, not a code change.
• Algorithm complexity class. Choosing an O(n^2) algorithm when O(n log n) is available and the dataset will grow to millions of records is a structural mistake, not a premature optimization concern. In a coding interview, if you implement bubble sort and say “I will optimize later,” the interviewer is not impressed.
• Network architecture. Deciding to make 10 sequential API calls when one batched call would work is a latency decision that gets baked into every consumer of your API. Changing this later requires coordinating all consumers.

• Caching. Do not cache until you have measured the bottleneck. The cache adds complexity (invalidation, staleness, cold start) that is only justified when you know the read path is the problem.
• Micro-benchmarks. Do not optimize which JSON serializer is 15% faster until you have confirmed serialization is actually a significant portion of your response time.
• Infrastructure tuning. Connection pool sizes, thread counts, GC settings — these are all deferrable until you have load test data showing they matter.

​

Follow-up: You are in a coding interview and you have an O(n^2) solution that works. The interviewer asks if you can do better. How do you approach this?

​

Follow-up: Your team’s backend engineer says “we should switch from JSON to Protocol Buffers for all our APIs because it is faster.” How do you evaluate this?

​

Follow-up: How do you handle the “I don’t know” moment during a live interview?

​

Follow-up: How do you distinguish between topics you should study deeply before an interview vs. topics where surface-level knowledge is acceptable?

​

Follow-up: What if the migration involves changing a column type — for example, from integer to UUID?

​

Follow-up: How do you handle the case where a migration backfill is too slow to run during normal operations?

​

Follow-up: The junior engineer pushes back and says “but Kubernetes is on every job posting — we need it for our careers.” How do you handle this?

​

Follow-up: At what point would you revisit the decision and consider migrating to Kubernetes?

​

Follow-up: How do you evaluate whether a blog post or tutorial is trustworthy?

​

Follow-up: How do you balance depth vs. breadth when preparing for an interview at a company whose tech stack you are unfamiliar with?

​

Follow-up: The product manager says “security vulnerabilities happen all the time, just ship it and we’ll fix it later.” How do you handle this?

​

Going Deeper: How do you build a culture where security is treated as a first-class concern rather than a last-minute checkbox?

​

Follow-up: How would you set up monitoring to catch this class of problem earlier?

​

Follow-up: The database team says “just add more read replicas” to fix the P99 problem. Why might this not help?

​

Follow-up: How do you prioritize which technical debt to address first in an inherited system?

​

Follow-up: The business wants new features on this inherited system. How do you negotiate time for stabilization work?

​

Follow-up: How do you prevent this from happening again on future projects?

​

Follow-up: How do you handle the team morale impact of walking back a decision the team invested in?

​

Follow-up: How do you design an alerting strategy that avoids alert fatigue while still catching real problems?

​

Follow-up: Your manager wants a dashboard showing 20 metrics for the database. Is that a good idea?

​

Follow-up: The engineering team pushes back, saying cost optimization will slow down feature delivery. How do you respond?

​

Follow-up: What are the most common cost optimization mistakes you have seen?

​

Follow-up: The PM says “we’re losing revenue every day we don’t roll this out.” How do you respond?

​

Follow-up: How do you handle feature flag cleanup? What happens when flags never get removed?

​

Follow-up: How do you handle idempotency when the side effect is calling an external API that is not idempotent?

​

Follow-up: Your idempotency key table has 500 million rows and is growing. How do you manage it?

​

Follow-up: How do you handle the transition period where both the old (divergent) copies and the new (authoritative) source exist?

​

Follow-up: How does Conway’s Law explain why this divergence happened in the first place?

​

Follow-up: How would you design a load test that actually predicts production behavior?

​

Follow-up: Your load test environment has smaller instances than production “to save costs.” Is this a good idea?

• Timing dependency. The test assumes an operation completes within N milliseconds. On a loaded CI runner, it sometimes takes longer. Fix: replace sleep(500) with an explicit wait condition or polling.
• Shared state. Tests run in parallel and share a database, file, or port. Occasionally, they conflict. Fix: isolate test state — use unique database schemas, random ports, or test containers.
• Non-deterministic ordering. The test depends on the order of results from an unordered source (hash map iteration, database results without ORDER BY, concurrent goroutines). Fix: sort results before assertion, or use set-based comparison.
• Real race condition in the application. The test is flaky because the code has a concurrency bug that manifests under specific timing. Fix: fix the bug, not the test. This is the 10% case where the flaky test is actually the most valuable test in your suite.

​

Follow-up: The team argues they do not have time to fix flaky tests because they need to ship features. How do you prioritize this?

​

Follow-up: How do you distinguish between a flaky test and a flaky system?

1. Traffic shape change: new customer launched? Bot spike? Regional shift? Compare today’s traffic distribution vs yesterday.
2. Upstream dependency regression: check p99 of each downstream call. Often a dependency got slower and propagated.
3. Infrastructure: noisy neighbor on shared nodes, disk full, GC pressure, cloud provider incident (check status pages).