Overview
Choosing the right architecture pattern is crucial for building scalable, maintainable systems. Each pattern has trade-offs, and the best choice depends on your specific context: team size, domain complexity, scale requirements, and organizational structure. Think of architecture patterns like city planning: a small town does not need a highway interchange, and a metropolis cannot survive on dirt roads. The pattern must fit the traffic.Monolithic Architecture
✅ Pros
- Simple to develop & deploy
- Easy debugging (single process)
- No network latency between modules
- ACID transactions easy
❌ Cons
- Hard to scale individual parts
- Long deployment cycles
- Technology lock-in
- One bug can crash everything
Microservices Architecture
✅ Pros
- Independent scaling
- Technology diversity
- Fault isolation
- Faster deployments
❌ Cons
- Distributed system complexity
- Network latency
- Data consistency challenges
- Operational overhead
Service Communication
Two fundamental approaches exist, and choosing the wrong one is a frequent source of production pain. Synchronous calls are like phone calls — you wait on the line until the other party responds. Asynchronous messaging is like dropping a letter in the mailbox — you move on with your day and trust it will be handled.Event-Driven Architecture
Event Sourcing
Instead of storing the current state of an entity (like a bank balance), you store every event that led to that state (every deposit and withdrawal). Think of it like an accounting ledger: you never erase entries, you only append new ones. To know the current balance, you replay the ledger from the beginning. This gives you a complete, auditable history and the ability to reconstruct state at any point in time.CQRS (Command Query Responsibility Segregation)
Layered Architecture
The most intuitive architecture pattern — like a layer cake where each layer only talks to the one directly below it. Most web frameworks (Django, Spring, Rails) naturally guide you toward this pattern. It works well until your application grows complex enough that the layers become bloated “god layers” with tangled responsibilities.Comparison Matrix
Hexagonal Architecture (Ports and Adapters)
The core business logic is isolated from external concerns through ports (interfaces) and adapters (implementations). Think of it like a power outlet: the port defines the shape of the plug (the interface), and different adapters (US, EU, UK plugs) can connect without changing the device. Your domain logic is the device — it does not care where the electricity comes from.Domain-Driven Design (DDD) Concepts
Strategic Design
Tactical Design Patterns
Saga Pattern (Distributed Transactions)
For maintaining data consistency across microservices without distributed transactions. In a monolith, you wrap everything in a database transaction and either all changes commit or all roll back. In microservices, there is no single database to wrap. The Saga pattern solves this by breaking a distributed transaction into a sequence of local transactions, each with a compensating action (an “undo”) if something downstream fails. Think of it like booking a vacation: if the hotel reservation succeeds but the flight booking fails, you cancel the hotel reservation (the compensating action).Choreography-Based Saga
Orchestration-Based Saga
In the orchestration approach, a central coordinator (the “orchestra conductor”) tells each service what to do and handles compensation when things fail. This is easier to understand and debug than choreography but creates a single point of coordination.API Gateway Pattern
Backend for Frontend (BFF)
Circuit Breaker Pattern
Prevent cascading failures in distributed systems. Named after electrical circuit breakers in your house: when too much current flows (too many failures), the breaker trips (opens) and stops all current (requests) to prevent a fire (cascading system failure). After a cooldown period, it lets a small test current through (half-open state) to see if the problem is resolved. Without a circuit breaker, a single failing downstream service can exhaust your thread pool and connection pool, causing your healthy service to fail too — a cascading failure that can take down your entire system in minutes.Service Mesh
Architecture Decision Records (ADR)
Document important architectural decisions. ADRs answer the question future engineers will inevitably ask: “Why on earth did we build it this way?” Without ADRs, architectural knowledge lives only in the heads of people who may leave the team. GitHub, Spotify, and ThoughtWorks all use ADRs extensively. Each ADR is a short document (one page) capturing the context, decision, and consequences at the time the decision was made.Comparison Matrix
Interview Deep-Dive
Your team is running a monolith that is starting to show scaling pain. Walk me through how you would decide what to extract into a microservice first, and what you would leave behind.
Your team is running a monolith that is starting to show scaling pain. Walk me through how you would decide what to extract into a microservice first, and what you would leave behind.
- The first step is NOT to extract anything. I would instrument the monolith heavily — APM traces, database query analysis, CPU/memory profiling per module — to find the actual bottleneck. In my experience, “scaling pain” is often one or two hot paths, not the entire system. At one e-commerce company, 80% of the load came from the product catalog search, not from orders or payments.
- Once I have identified the bottleneck module, I evaluate it against three extraction criteria: (1) Does it have a clearly defined bounded context with minimal data coupling to other modules? (2) Does it need to scale independently — meaning its load profile is fundamentally different from the rest? (3) Does the team structure support owning it independently (Conway’s Law)?
- I would draw the dependency graph. If the candidate module makes 15 synchronous calls back into the monolith, extracting it creates a distributed monolith — the worst of both worlds. You get network latency and partial failure modes without any of the independence benefits.
- The extraction itself follows the Strangler Fig pattern: stand up the new service, route a small percentage of traffic to it (canary), run both in parallel, compare results, then cut over. Keep the monolith code intact until the new service has proven itself in production for at least 2-4 weeks.
- What I would leave behind: anything with heavy transactional coupling. If placing an order requires atomically updating inventory, processing payment, and creating a shipment record, keeping those in a monolith with a single database transaction is dramatically simpler and more reliable than a distributed saga across three services.
Explain the Saga pattern to me. When would you use choreography versus orchestration, and what failure modes keep you up at night with each approach?
Explain the Saga pattern to me. When would you use choreography versus orchestration, and what failure modes keep you up at night with each approach?
- A Saga is a sequence of local transactions across multiple services where each step has a compensating action (an undo) if a downstream step fails. It replaces the distributed two-phase commit (2PC) that does not scale in microservices architectures.
- Choreography means each service listens to events and decides what to do next. There is no central coordinator. Service A publishes “OrderCreated,” Service B hears it and processes payment, publishes “PaymentProcessed,” Service C hears that and reserves inventory. The flow is implicit — it lives in the event subscriptions, not in any single piece of code.
- Orchestration means a central saga coordinator tells each service what to do in sequence and handles rollbacks when things fail. The flow is explicit — you can read the orchestrator code and see every step.
- I use orchestration when: the saga has more than 3-4 steps, the team is new to distributed systems, or the compensation logic is complex (partial refunds, conditional rollbacks). The explicit flow makes debugging far easier. I use choreography when: the workflow is simple (2-3 steps), services are owned by different teams who deploy independently, or we need to avoid a single point of failure.
- Failure modes that keep me up at night: (1) In choreography, “ghost events” — a service processes an event, crashes before publishing its output event, and the saga is stuck in limbo. No central coordinator knows it is stalled. You need dead-letter queues and timeout monitors to detect these. (2) In orchestration, the orchestrator itself failing mid-saga — you have charged the customer but the orchestrator dies before reserving inventory. This requires the orchestrator to persist its state (saga log) so it can resume on restart. (3) In both: compensating actions that themselves fail. You tried to refund the payment but Stripe is down. Now you need compensation for your compensation — this is where things get genuinely ugly and you need manual intervention queues.
You are designing a system for a company that processes 50,000 orders per day. A colleague proposes an event-sourced architecture with CQRS. Walk me through your response.
You are designing a system for a company that processes 50,000 orders per day. A colleague proposes an event-sourced architecture with CQRS. Walk me through your response.
- My first reaction is skepticism, and I would push back respectfully. 50,000 orders per day is roughly 0.6 orders per second on average, maybe 5-10 per second at peak. This is trivially handled by a well-indexed PostgreSQL database on a single server. Event sourcing and CQRS add enormous operational complexity — separate read and write models, eventual consistency between them, event schema versioning, replay infrastructure, snapshot management. That complexity has to earn its place.
- The question I would ask is: “What problem are we solving?” If the answer is “scale,” then event sourcing is overkill at this volume by two orders of magnitude. If the answer is “we need a complete audit trail of every state change for regulatory compliance,” then event sourcing starts to make sense because it gives us that for free. If the answer is “our read and write patterns are wildly different — writes are simple but reads require complex aggregations across multiple dimensions,” then CQRS alone (without event sourcing) might help.
- If we do proceed, I would start with CQRS only: separate the read model (optimized denormalized views) from the write model (normalized transactional tables), synchronized via database-level change data capture (CDC) using something like Debezium. This gives us 80% of the benefit with 20% of the complexity. We would only add full event sourcing if we hit a concrete need for temporal queries, audit logs, or the ability to replay and rebuild state.
- The hidden cost most people miss: event schema evolution. On day 1, your OrderCreated event has 5 fields. Six months later, it has 12. You now have millions of stored events in two different schemas, and your event replay logic needs to handle both. Tools like Avro with a schema registry help, but it is still a continuous maintenance burden.
What is the difference between a distributed monolith and a proper microservices architecture? How do teams accidentally end up with a distributed monolith?
What is the difference between a distributed monolith and a proper microservices architecture? How do teams accidentally end up with a distributed monolith?
- A distributed monolith has all the operational complexity of microservices (network latency, partial failures, distributed debugging, separate deployments) with none of the benefits (independent scaling, independent deployment, technology diversity). You cannot deploy service A without also deploying service B because they are coupled through shared databases, synchronous call chains, or shared libraries.
- Teams end up here through three common paths. First, extracting services along the wrong boundaries — splitting by technical layer (a “data service,” an “API service,” a “business logic service”) instead of by business domain. This forces every business operation to traverse all three services synchronously. Second, sharing a database across services — two services reading and writing the same tables means any schema change requires coordinated deployment of both services. You have one deployment unit masquerading as two. Third, using synchronous request chains where service A calls B, which calls C, which calls D. Any single service being slow or down causes the entire chain to fail. This is a synchronous distributed monolith.
- The test for whether you have a distributed monolith: “Can this service be deployed independently without coordinating with other teams?” If the answer is no, you have a distributed monolith regardless of how many Docker containers you run.
- The fix is painful but straightforward: identify the coupling points, and either merge tightly-coupled services back into one (yes, sometimes the right answer is fewer services) or introduce asynchronous communication and separate data ownership. Each service should own its data and expose it only through well-defined APIs.
Walk me through how you would design an API Gateway for a system with 20 backend services. What concerns does it handle, and what are the risks of getting it wrong?
Walk me through how you would design an API Gateway for a system with 20 backend services. What concerns does it handle, and what are the risks of getting it wrong?
- An API Gateway is a single entry point for all client requests that handles cross-cutting concerns so individual services do not have to. The core responsibilities are: routing (mapping /users/* to user-service, /orders/* to order-service), authentication (validating JWT tokens once at the edge, not in every service), rate limiting (protecting backend services from traffic spikes), request/response transformation (aggregating responses from multiple services for mobile clients), and observability (logging every request with a correlation ID for distributed tracing).
- For 20 services, I would use an off-the-shelf gateway like Kong, AWS API Gateway, or Envoy-based solutions rather than building custom. Building your own API gateway is tempting but almost always a mistake — it becomes a critical path component that needs to handle TLS termination, connection pooling, circuit breaking, retries, and load balancing, all at high throughput with low latency. That is a full-time infrastructure project.
- The Backend-for-Frontend (BFF) pattern becomes important at 20 services. A mobile client should not make 6 API calls to render one screen. I would create BFF layers — one for mobile, one for web, one for third-party partners — that aggregate backend calls and return tailored responses. The mobile BFF returns smaller payloads and fewer fields. The web BFF returns richer data.
- Risks of getting it wrong: (1) The gateway becomes a single point of failure — if it goes down, the entire system is unreachable. Mitigation: deploy multiple gateway instances behind a load balancer, with health checks and auto-scaling. (2) The gateway becomes a “god service” where developers dump business logic because “it is easy to add middleware.” The gateway should only handle cross-cutting infrastructure concerns, never business logic. (3) Latency overhead — every request now has an extra network hop. At high throughput (100K+ RPS), the gateway’s connection pool and CPU become bottlenecks. Profile it continuously.