API Gateway Pattern
An API Gateway is the single entry point for all client requests to your microservices. It handles cross-cutting concerns and provides a unified interface. Think of an API Gateway like the reception desk at a large corporate office. Visitors (clients) do not wander the building knocking on individual office doors (services). They go to reception, identify themselves (authentication), get a visitor badge (authorization), and reception directs them to the right floor (routing). If a department is closed, reception tells the visitor immediately (circuit breaking) rather than letting them walk to an empty office. The trade-off: reception adds a stop to every visit (latency), and if reception itself goes down, nobody gets in (single point of failure). That is why production gateways run as horizontally scaled, stateless clusters — you need multiple receptionists.- Understand API Gateway responsibilities
- Implement routing and load balancing
- Add authentication and authorization
- Implement rate limiting and throttling
- Build request aggregation patterns
Why API Gateway?
Without API Gateway
With API Gateway
Your API gateway becomes a bottleneck: p99 climbs to 400ms in the gateway alone, separate from downstream latency. Walk through diagnosis and fix.
Your API gateway becomes a bottleneck: p99 climbs to 400ms in the gateway alone, separate from downstream latency. Walk through diagnosis and fix.
- Profile the gateway. Most gateways have per-middleware timing or you can add it cheaply. Identify which middleware is dominant — usually JWT verification (if doing a remote call), rate limiting (if not using local counters), or request body inspection.
- Short-term wins: cache JWT verification results (with a short TTL, cryptographically valid); move rate limit counters to local memory with periodic flush to Redis; stop inspecting request bodies for anything that does not strictly need them.
- Move heavy logic out of the hot path. Audit logging, analytics enrichment, request/response transforms for a single endpoint, and business-specific rewrites should not be in the gateway. Push to sidecars, async pipelines, or per-service BFFs.
- Scale horizontally. Gateways should be stateless so adding replicas linearly scales throughput. If scaling does not help, there is a shared dependency (Redis, auth service) that needs its own attention.
- Set an SLO: “gateway-only overhead under 10ms at p99.” Alert on breaches. Over time, teams will push back on anything that threatens this budget, which is exactly the pressure you want.
- “Add more replicas until latency drops.” Fails when the bottleneck is a shared dependency (e.g., Redis for rate limits); more gateway replicas just hit the same wall.
- “Remove middleware to speed up.” Fails because some middleware (auth, rate limit) is non-negotiable; the right move is to make it faster, not remove it.
- Netflix Tech Blog’s “Zuul 2: The Netflix Journey to Asynchronous, Non-Blocking Systems.”
- Envoy documentation on filter chains and latency budgets.
- Sam Newman’s Building Microservices, 2nd edition, chapter on API gateways.
API Gateway Responsibilities
Building an API Gateway with Express
Basic Gateway Structure
Before we write a single line of routing code, we need to wire up the middleware pipeline that every request will flow through. The order of middleware matters enormously: security headers must come before body parsing (to reject oversized bodies early), logging must come before auth (so you can see unauthenticated attacks), and error handlers must come last (to catch everything). Without this careful ordering, you get subtle bugs — a missing request ID means you cannot correlate logs across services, a missing CORS header means browsers silently reject responses, and a missing body size limit means a single attacker can OOM your gateway with a 10GB upload. The key tradeoff here is that every middleware adds latency and CPU cost, so you only add what earns its keep — every team eventually wants to add “just one more” middleware and ends up with a 100ms gateway overhead.- Node.js
- Python
Service Routing
Routing is the heart of a gateway: when a request comes in for/api/users/123, the gateway must know to forward it to the user service. This sounds trivial, but production routing has three non-obvious requirements. First, the gateway must rewrite the path so the downstream service does not need to know about the gateway prefix — the user service should receive /users/123, not /api/users/123. Second, authentication context must flow through — once the gateway validates the JWT, it injects headers like X-User-ID so downstream services do not need to re-validate. Third, correlation headers must propagate — without X-Request-ID flowing end-to-end, debugging a distributed trace is impossible. The key tradeoff is that every proxy call adds a TCP hop; use connection pooling and HTTP/2 to amortize this. Without proper routing, you end up with either clients that know all internal service URLs (tight coupling), or a big-ball-of-mud gateway where routing logic and business logic mix.
- Node.js
- Python
Two services with different teams both want to own '/api/payments/*'. How do you decide routing, and what does 'gateway governance' look like?
Two services with different teams both want to own '/api/payments/*'. How do you decide routing, and what does 'gateway governance' look like?
- Recognize this as a governance problem, not a technical one. URL paths are contracts with clients; splitting one path across two services is messy and usually indicates a team boundary drawn in the wrong place.
- Clarify the domain. Is this truly two services (e.g., payments-intake and payments-settlement), or is it one domain with two teams collaborating? In the former, sub-path routing (
/api/payments/intake/*,/api/payments/settlement/*) works. In the latter, find a single team to own the path. - Codify the governance in a gateway config review process. Changes to public URL paths go through a CODEOWNERS-style approval with a small platform team as backstop. No team ships a routing change without this review.
- Use a service catalog. Every public path has a single owning service with a single team. Cross-team changes require explicit re-assignment.
- Sunset paths carefully. If a team wants to take over a path, the old owner runs a parallel route with a deprecation date, telemetry shows the migration, and the switch-over is boring.
/api/v1/payments/* and /api/v2/payments/* can be different services during migration. But they should share a common owner or a well-defined interface between owners.Follow-up 2: “How do you audit routing changes?”Every change to the route table is a git commit in a config repo with review. The gateway reads from that repo (or a derived store). You can git blame any route and find the PR that introduced it.Follow-up 3: “What about canary routing: 10% to service-v2, 90% to service-v1?”Supported by most gateways via weighted routing. Represent the weight in config; monitor by version; automate rollback on error-rate regression. Keep weights simple (increments of 10%) to avoid debugging percentage arithmetic in production.- “Let both teams route to the same path and the gateway picks one.” Fails because non-deterministic routing makes incidents impossible to debug.
- “Add a header to distinguish which service.” Fails because clients cannot reliably set headers without coordination, and this effectively forks the URL contract.
- Mike Amundsen’s “API Paths and Path Hierarchies.”
- Uber Engineering’s “Domain-Oriented Microservice Architecture.”
- Backstage documentation on service catalogs.
Authentication Middleware
Authentication at the gateway is the single most important security boundary in a microservices architecture. The principle is simple: validate once at the edge, then trust inside the perimeter. Without this, every service would need to duplicate JWT validation logic — and when you rotate a key, you have to update every service. But there is a critical tradeoff: if the gateway’s auth logic is wrong, every service is wrong. That is why the code validates the token locally (using the signing key) instead of calling an auth service on every request (which would add 20-50ms per request and create a hard dependency on the auth service being up). If we skip this middleware entirely, every service must re-validate the JWT and enforce public/private route rules, leading to inconsistency — one service forgets to check theexp claim and tokens “never expire” there.
- Node.js
- Python
You are designing auth for a new platform. The team lead says 'just put auth in the gateway and trust downstream.' What do you push back on, and what do you propose?
You are designing auth for a new platform. The team lead says 'just put auth in the gateway and trust downstream.' What do you push back on, and what do you propose?
- Agree on the core: yes, validate the client token at the gateway to avoid duplicating JWT parsing in every service.
- Push back on “trust downstream without verification.” Defense in depth requires services to verify their caller, even if it is another internal service. The mechanism is workload identity (mTLS or SPIFFE), not user identity.
- Propose a two-layer model: user identity (gateway-signed internal header, services verify the signature) and service identity (mTLS certificate issued by an internal CA, rotated frequently). A service authorizes on both: “user U via service S.”
- Address key management: a JWKS endpoint the gateway hosts, services fetch and cache the public keys, rotation is automated with overlapping windows. No manual key distribution.
- Document the threat model: what happens if a service is compromised? If only user identity, the attacker can act as any user. With service identity, the attacker can only act as the compromised service to services that explicitly trust it.
- “Validate JWT at the gateway and trust the internal network.” Fails the “what if one service is compromised” test, and modern threat models (supply chain, zero-day) make internal compromise realistic.
- “Pass the raw JWT everywhere for simplicity.” Fails because it creates a blast-radius problem on compromise and couples service auth to user auth forever.
- BeyondCorp papers from Google on identity-aware proxies.
- SPIFFE/SPIRE documentation and the “Solving the Bottom Turtle” book.
- Istio documentation on authorization policies and peer authentication.
Rate Limiting
Rate limiting exists because a single misbehaving client can bring down your entire platform. One infinite loop in a customer’s script, one bot scraping your API, one bug in a mobile app that retries too aggressively — any of these can 100x your traffic in seconds. The job of rate limiting is to protect your services by rejecting excess requests at the edge, before they consume database connections or CPU. The tradeoff is operational complexity: per-user limits need a shared counter across gateway instances (Redis), and that Redis call adds 1-2ms per request. Skip rate limiting and you get cascading outages — a single bad actor causes database contention, which slows down legitimate users, who retry, which creates more load. The layered approach below (global + per-user + per-endpoint) matters because different attacks look different: a credential-stuffing attack hits/login hard but looks normal globally, while a scraper hits /products from many IPs but stays under per-user limits.
- Node.js
- Python
Design the auth, rate limiting, and routing for a new API: what goes in the gateway vs in the service? Walk through your reasoning.
Design the auth, rate limiting, and routing for a new API: what goes in the gateway vs in the service? Walk through your reasoning.
- State the principle: cross-cutting concerns with no domain knowledge belong in the gateway. Anything touching business rules belongs in the service.
- In the gateway: TLS termination, JWT signature verification, rate limiting (global, per-user, per-endpoint), request routing, correlation ID injection, basic request/response logging, CORS.
- In the service: authorization decisions (can this user do this action on this resource), business-specific validation, domain-level rate limits (e.g., “you can only create 10 projects per org per day”), idempotency key handling, and all business logic.
- Explain the gray areas. Authentication is gateway; authorization is service because it needs domain context. Basic rate limiting is gateway; business-tier rate limiting (“paid customers get 10x”) can be either depending on whether tier is exposed in the JWT.
- Document the contract: the gateway promises to pass a verified
X-User-Idand scopes; the service promises to enforce authorization and business rules. Nothing in either layer assumes the other’s responsibilities.
- “Put everything in the gateway so services can be simple.” Fails because the gateway becomes a monolith with no ownership and blocks every team.
- “Put everything in the services so the gateway is just a proxy.” Fails because every service re-implements auth and rate limiting with subtle differences, and you inevitably get bugs and gaps.
- Sam Newman’s Building Microservices, 2nd edition, on gateway patterns.
- Phil Calçado’s “Pattern: API Gateway / Backends for Frontends.”
- Netflix Tech Blog on Zuul 2 and its plugin scope policy.
Request Aggregation
Aggregation exists because mobile and web clients are penalized by round trips — every HTTP request on a mobile network costs 100-300ms of latency just for the TCP + TLS handshake. If a dashboard needs data from five services, doing that client-side means five sequential (or parallel) mobile round trips versus one trip if the gateway aggregates server-side. The key technique is parallel fan-out usingPromise.all or asyncio.gather — you call all five services simultaneously and wait for the slowest one, not the sum of all. The important nuance is Promise.allSettled versus Promise.all: for a dashboard, if the recommendations service is down, you still want to show orders and cart (partial response), so use allSettled. For an order detail page where missing data breaks the UI, use all and fail fast. The tradeoff is that the gateway now has partial knowledge of downstream service contracts, which couples it to those services — overdo this and your gateway becomes a dumping ground for orchestration logic that belongs in a BFF service.
- Node.js
- Python
Your gateway aggregates data from 6 services for a mobile dashboard. Mobile engineers complain about latency and occasional 500s. Walk through your redesign.
Your gateway aggregates data from 6 services for a mobile dashboard. Mobile engineers complain about latency and occasional 500s. Walk through your redesign.
- Measure. Per-sub-request timing at the gateway; identify the long-tail service and the error rate per service.
- Identify the critical vs. non-critical services. Core dashboard data (account balance, recent transactions) must be present; recommendations, promotions, and usage tips can be missing.
- Move aggregation to a mobile BFF. Gateway just routes
/mobile/dashboardto the BFF; BFF does the fan-out with per-sub-request timeouts (say 300ms each) andallSettled. - For critical services, use
Promise.allwith tight timeouts. For non-critical, useallSettledand drop failures from the response with a flag like{"recommendations": null, "recommendations_available": false}. - Add short-TTL caching for fan-out results where staleness is acceptable. Even a 30-second cache dramatically reduces downstream load.
- “Add caching to the gateway.” Fails because it does not address per-user data and still couples aggregation to the gateway.
- “Reduce the number of services called.” Fails because each service exists for a reason; the right fix is to fail gracefully, not to remove features.
- Phil Calçado’s “Pattern: Backends for Frontends.”
- SoundCloud’s blog post “BFF@SoundCloud.”
- Netflix Tech Blog on device-specific edge services.
Backend for Frontend (BFF) Pattern
Different clients need different APIs.Mobile BFF Implementation
The Mobile BFF exists because mobile clients are bandwidth-constrained and battery-constrained in ways web clients are not. Sending a 2MB product payload when the user’s phone only needs 50KB to render a list item wastes data (costing the user money on metered plans) and battery (the radio stays active longer). The mobile BFF’s job is to be ruthlessly aggressive about payload size: one thumbnail instead of a full image gallery, a short description instead of full HTML, top-3 reviews instead of all reviews. The tradeoff is that mobile BFFs need their own deployment pipeline and their own on-call rotation — that is real operational cost. If you skip the BFF and have the mobile app call services directly, you end up either over-fetching (slow mobile experience) or exposing internal service URLs and auth (security nightmare). A well-built Mobile BFF can reduce payload by 80-90% for common list endpoints.- Node.js
- Python
Web BFF Implementation
The Web BFF has the opposite constraints from the Mobile BFF: bandwidth is plentiful, but round trips are perceived as slow (the user stares at a loading spinner). So the Web BFF leans hard into parallel fan-out — one request to the BFF produces five concurrent calls to downstream services, and the aggregated response fills the entire page at once. The tradeoff is that a single slow service slows the entire page render, but that is usually a better experience than a page that appears in pieces over several seconds. The Web BFF also returns richer data: full descriptions, all images, related products, questions, breadcrumbs — everything the web UI needs for SEO and interactivity. Without a Web BFF, you either make the web client orchestrate all these calls (more client complexity, worse perceived performance) or return identical responses to mobile and web (you slow down mobile, or you under-serve web).- Node.js
- Python
Your team wants to create a new BFF for admin users. The platform team pushes back with 'BFFs are getting out of hand.' How do you decide, and what governance applies?
Your team wants to create a new BFF for admin users. The platform team pushes back with 'BFFs are getting out of hand.' How do you decide, and what governance applies?
- Agree on the concern: BFF proliferation is real and expensive. Each BFF is another deployable, another oncall rotation, another set of cross-cutting infrastructure (auth, logging, tracing) to keep in sync.
- Apply the client-type test: is “admin users” a genuinely different client from existing ones? If admins use the same web app with different permissions, no new BFF — extend the existing web BFF. If admins use a dedicated admin console with very different data shapes, yes.
- Propose governance: a “BFF registry” with explicit justification for each one, a periodic review of whether existing BFFs should merge, and a shared library for cross-cutting concerns so new BFFs start with auth/logging/tracing baked in.
- Define the alternatives: sometimes a BFF is overkill. A dedicated endpoint in the existing web BFF, or a small query service, may be enough. Pick the smallest thing that solves the problem.
- Commit to the cost model: if the team wants a new BFF, they own it end-to-end including oncall, upgrades, and eventual deprecation when no longer needed.
- “BFFs are light, just add one whenever you need shaping.” Fails because the operational cost per BFF is significant and proliferation erodes platform consistency.
- “Use a single API for every client.” Fails because different clients genuinely have different needs and forcing uniformity hurts each of them.
- Phil Calçado’s “Pattern: Backends for Frontends.”
- Spotify Engineering’s “Backstage” and Golden Paths documentation.
- Sam Newman’s Building Microservices, 2nd edition, chapter on BFFs.
Circuit Breaker in Gateway
A circuit breaker exists because cascading failures are the single most destructive failure mode in microservices. When the payment service slows from 50ms to 5 seconds, every gateway request waits 5 seconds, the gateway’s connection pool fills up with pending requests, new requests queue behind them, and within 30 seconds your entire gateway is wedged — even for requests that have nothing to do with payments. A circuit breaker short-circuits this: after N failures, it stops calling the failing service entirely for some cooldown period, returning an immediate error (or a fallback value) and letting the failing service recover without the pressure of retrying traffic. The tradeoff is that a tripped breaker causes some requests to fail that might have succeeded — you are trading “some requests fail immediately” for “no requests succeed because the whole gateway is frozen.” The critical tuning parameters are error threshold (50% is standard), volume threshold (must have enough requests to have statistical signal), and reset timeout (how long to wait before probing again). Without circuit breakers, one slow downstream service takes down your entire fleet in a classic retry-storm pattern.- Node.js
- Python
A gateway circuit breaker trips on every deploy of a downstream service because the first 50 requests on the new instance fail while it warms up. How do you fix this without removing the breaker?
A gateway circuit breaker trips on every deploy of a downstream service because the first 50 requests on the new instance fail while it warms up. How do you fix this without removing the breaker?
- Identify the real problem: the breaker is working correctly; the downstream is presenting unhealthy responses during warm-up. Fixing the breaker is the wrong layer.
- Short-term: raise the volume threshold so small initial failures do not trip, or add a warm-up delay to the breaker’s evaluation window.
- Real fix: the downstream needs proper readiness checks. Kubernetes should route traffic only after
readinessProbesucceeds, and the probe should exercise the full critical path, not just return 200 from a static endpoint. - For even warmer deploys, use prefetch / JIT warmup in the service: on startup, hit caches, compile regexes, warm connection pools before declaring readiness.
- Monitor “newly-deployed-pod error rate” as a separate metric from steady-state error rate. If the two diverge significantly, the deploy process itself needs work.
- “Disable the breaker during deploys.” Fails because it removes the safety net exactly when it is most needed.
- “Lower the breaker sensitivity.” Fails because now the breaker does not trip on real failures either, defeating its purpose.
- Kubernetes documentation on readiness and startup probes.
- Netflix Tech Blog on canary analysis and deploy strategies.
- Resilience4j docs on circuit-breaker configuration for warm-up.
Gateway with Kong
Kong is a popular open-source API Gateway.Programmatic Kong Admin API
Declarative YAML works well for static deployments, but mature platforms need to configure Kong dynamically — onboarding a new tenant should not require a redeploy. The Kong Admin API on port 8001 lets you register services, routes, consumers, and plugins via HTTP. The tradeoff is that dynamic configuration means your “source of truth” is now in Kong’s database instead of git, so you need to either sync back to git or accept that operational drift will happen. Use this approach when you have self-service onboarding; use the declarative file otherwise.- Node.js
- Python
Interview Questions
Q1: What are the key responsibilities of an API Gateway?
Q1: What are the key responsibilities of an API Gateway?
- Request Routing: Route requests to appropriate services
- Authentication/Authorization: Validate tokens, check permissions
- Rate Limiting: Protect services from overload
- Load Balancing: Distribute traffic across service instances
- Request/Response Transformation: Modify headers, body
- Caching: Cache responses for performance
- Monitoring/Logging: Centralized observability
- Circuit Breaking: Prevent cascade failures
- Protocol Translation: REST to gRPC, etc.
- Request Aggregation: Combine multiple service calls
Q2: What is the BFF (Backend for Frontend) pattern?
Q2: What is the BFF (Backend for Frontend) pattern?
- Mobile needs slim responses (bandwidth)
- Web needs rich responses (features)
- Admin needs full data access
- Optimized responses per client
- Independent evolution
- Better team ownership
- Client-specific concerns
- More services to maintain
- Potential code duplication
- Need for shared libraries
Q3: How do you handle authentication in an API Gateway?
Q3: How do you handle authentication in an API Gateway?
- JWT Validation: Gateway validates token, forwards user info in headers
- OAuth 2.0: Gateway handles token introspection
- API Keys: Gateway validates keys, applies rate limits per key
- Session-based: Gateway manages sessions
- Validate token at gateway
- Extract user claims
- Forward as headers to services (X-User-ID, X-User-Roles)
- Services trust gateway (internal network)
- Use mutual TLS between gateway and services
Q4: How do you prevent the API Gateway from becoming a bottleneck?
Q4: How do you prevent the API Gateway from becoming a bottleneck?
- Horizontal Scaling: Multiple gateway instances behind load balancer
- Stateless Design: No session state in gateway
- Efficient Routing: Use performant routing algorithms
- Caching: Cache static/semi-static responses
- Async Processing: Non-blocking I/O
- Connection Pooling: Reuse connections to services
- Track latency added by gateway
- Monitor CPU/memory usage
- Set up auto-scaling based on metrics
- Service mesh for internal traffic
- Edge computing for some operations
Summary
Key Takeaways
- API Gateway is the single entry point
- Handles cross-cutting concerns centrally
- BFF pattern optimizes for different clients
- Circuit breakers prevent cascade failures
- Kong/AWS API Gateway for production
Next Steps
Interview Deep-Dive
'Your API Gateway is adding 50ms of latency to every request. The P99 is 200ms. Product says this is unacceptable for the checkout flow. How do you optimize it?'
'Your API Gateway is adding 50ms of latency to every request. The P99 is 200ms. Product says this is unacceptable for the checkout flow. How do you optimize it?'
'You have a mobile app, a web app, and an admin dashboard. Should you use one API Gateway or multiple? What about the BFF pattern?'
'You have a mobile app, a web app, and an admin dashboard. Should you use one API Gateway or multiple? What about the BFF pattern?'
'How do you implement rate limiting in a distributed system where the API Gateway runs as multiple instances behind a load balancer?'
'How do you implement rate limiting in a distributed system where the API Gateway runs as multiple instances behind a load balancer?'
ratelimit:user:123:minute:2024-01-15T10:30 with a TTL equal to the window size. Redis handles the concurrency. This adds 1-2ms per request for the Redis round-trip, which is acceptable for most use cases.For higher performance, I use a two-tier approach. The first tier is a local in-memory token bucket per user. It handles burst absorption and catches obvious abuse without any network call. The second tier is Redis for accurate distributed counting. The local tier allows up to 80% of the limit, and the remaining 20% is checked against Redis. This means a user could theoretically get 120% of their limit if they perfectly distribute requests across all instances, but that is an acceptable trade-off for removing Redis from the hot path of 80% of requests.For extreme scale (millions of requests per second), I have seen teams use sliding window algorithms with Redis sorted sets, or even move rate limiting into a service mesh sidecar (Envoy) with shared state via a control plane. But for most systems, the simple Redis counter with fixed windows is sufficient and much easier to operate.The gotcha that catches teams: clock skew between instances. If instance A thinks it is 10:30:59 and instance B thinks it is 10:31:01, they use different window keys and the limit doubles for that second. Use NTP synchronization and make your window slightly larger than the stated limit to absorb clock drift.Follow-up: “A customer is complaining they are being rate limited even though they are well within their plan limits. How do you debug this?”I would check three things: First, are they using multiple API keys? Rate limits are typically per-key, so two keys each using half the limit appears fine but is counted separately. Second, check if retry storms are inflating their request count — their client library might be retrying failed requests with aggressive backoff, doubling or tripling their effective request rate. Third, check for shared IP rate limiting — if they are behind a corporate NAT, all employees share one IP, and the IP-based rate limit (separate from user-based) might be the one triggering. The fix depends on the cause: consolidate keys, fix retry logic, or switch from IP-based to authenticated user-based rate limiting.