Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Cloud Service Patterns — AWS in Depth

​

Part I — Serverless Patterns (Lambda / Cloud Functions)

​

1.1 Cold Starts — What Actually Happens

1. Download your code — AWS fetches your deployment package (zip or container image) from S3 or ECR. Larger packages take longer.
2. Create the execution environment — AWS provisions a microVM (Firecracker), allocates memory, sets up networking, and mounts the filesystem.
3. Initialize the runtime — The language runtime starts (Python interpreter, Node.js V8, JVM, .NET CLR). This is where JVM-based languages pay a massive tax.
4. Run your initialization code — Any code outside your handler function executes: import statements, database connection setup, SDK client creation, global variable initialization.
5. Execute your handler — Finally, your actual function logic runs.

• Provisioned Concurrency — Pre-warms a specified number of execution environments. You pay for them whether they are used or not (like reserved instances for Lambda). Use when: your API has strict latency SLAs and cold starts are unacceptable.
• SnapStart (Java only) — AWS takes a snapshot of the initialized JVM after your init code runs, then restores from snapshot on cold start instead of re-initializing. Reduces Java cold starts from seconds to ~200-500 ms. Released late 2022. Use for any Java Lambda function.
• Keep functions small — Smaller deployment packages download faster. Trim unused dependencies. Use Lambda layers for shared libraries.
• Warm-up pings — A CloudWatch Events rule that invokes your function every 5 minutes with a no-op payload. Cheap and effective, but does not guarantee warm instances under concurrent load (one ping keeps one instance warm).
• Choose lightweight runtimes — Go and Rust have the fastest cold starts. Python and Node.js are a good middle ground. Java and .NET are the slowest without SnapStart or equivalent.
• Initialize outside the handler — Database connections, SDK clients, and configuration should be created in the global scope (outside the handler function). This code runs once per cold start and is reused across warm invocations.

# GOOD: Connection created once, reused across invocations
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Orders')

def handler(event, context):
    return table.get_item(Key={'order_id': event['order_id']})

# BAD: Connection created on EVERY invocation
def handler(event, context):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('Orders')
    return table.get_item(Key={'order_id': event['order_id']})

​

1.2 Concurrency Model

• Unreserved — All functions share the account’s concurrency pool. A traffic spike on one function can starve all others.
• Reserved — You allocate a guaranteed number of concurrent executions to a specific function. That function can never exceed its reservation, and no other function can consume its allocation. This is both a guarantee and a throttle.

​

1.3 Event Sources — The Lambda Trigger Taxonomy

​

1.4 Lambda Architecture Patterns

​

1.5 Container Images vs Zip Packages

• Max size: 50 MB zipped, 250 MB unzipped
• Fast deployment (seconds)
• Works with Lambda layers for shared dependencies
• Best for: most Lambda functions, small-to-medium dependency footprints

• Max size: 10 GB
• Uses standard Docker tooling (Dockerfile, docker build, ECR)
• Slower cold starts (larger image to download, though AWS caches aggressively)
• No Lambda layer support (bake everything into the image)
• Best for: ML inference (large model files), functions with massive dependencies, teams with existing Docker workflows, functions that need specific system libraries

​

1.6 Step Functions — Orchestration Done Right

• Workflows with conditional logic (if payment succeeds, do X; if it fails, do Y)
• Long-running workflows that exceed Lambda’s 15-minute timeout
• Workflows that need human approval steps
• Complex retry and error handling across multiple steps
• When you need a visual representation of workflow state for debugging

• Simple fan-out (one event triggers multiple independent actions)
• When steps are truly independent and do not need coordination
• When you want loose coupling between services owned by different teams
• When the “workflow” is really just a pipeline (A -> B -> C with no branching)

​

1.7 Cost Model — When Serverless Stops Being Cheap

​

1.8 Anti-Patterns

• Latency accumulation: Each hop adds cold start risk + network latency. A chain of A -> B -> C -> D with 300ms cold starts becomes 1.2 seconds in the worst case — before any business logic runs.
• Concurrency multiplication: If A calls B synchronously, both A and B consume a concurrent execution for the full duration. A chain of 4 functions processing 100 requests consumes 400 concurrent executions. You hit the account limit 4x faster.
• Cascading timeouts: If D’s timeout is 30 seconds, C must have a timeout > 30 seconds, B > C, A > B. Now A has a 2-minute timeout for what should be a 5-second operation.
• Cost doubling: Every function in the chain is billed for the time it spends waiting for the downstream function. You are literally paying twice (or more) for the same wall-clock time.
• The fix: Use Step Functions for orchestration (each step is invoked independently), SQS between steps for loose coupling, or collapse the chain into a single function if the logic is simple enough.

• Why it happens: Teams install entire frameworks (all of boto3 when they only need S3, all of pandas when they only need CSV parsing), bundle test fixtures, include unused native binaries, or skip tree-shaking in Node.js.
• The fix: Audit dependencies ruthlessly. Use pip install --target with only the packages you need. In Node.js, use bundlers (esbuild, webpack) to tree-shake dead code. Move shared dependencies to Lambda Layers — layers are downloaded once and cached across function updates, so updating your function code does not re-download shared libraries.
• Lambda Layers best practices: Create layers for common SDKs, database drivers, and utility libraries. A layer can be up to 250 MB (unzipped) and a function can use up to 5 layers. Layers are versioned — pin specific versions in production, use $LATEST only in dev.

# Create a Lambda Layer for shared Python dependencies
mkdir -p python/lib/python3.12/site-packages
pip install requests boto3 pyjwt -t python/lib/python3.12/site-packages
zip -r shared-libs-layer.zip python/
aws lambda publish-layer-version \
  --layer-name shared-libs \
  --zip-file fileb://shared-libs-layer.zip \
  --compatible-runtimes python3.12

​

Part II — Storage Patterns (S3 / Blob Storage)

​

2.1 S3 Consistency Model

​

2.2 Storage Classes and Lifecycle Policies

{
  "Rules": [
    {
      "ID": "ArchiveOldLogs",
      "Status": "Enabled",
      "Prefix": "logs/",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" },
        { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
      ],
      "Expiration": { "Days": 2555 }
    }
  ]
}

​

2.3 Multipart Uploads

• Resilience — If one part fails, you retry only that part, not the entire file.
• Parallelism — Upload parts concurrently to saturate your bandwidth.
• Required for files > 5 GB — S3 has a 5 GB single-PUT limit.
• Resumable — Parts already uploaded persist for up to 7 days (configurable). You can resume after a failure.

​

2.4 Pre-Signed URLs — Secure Temporary Access

• Secure file downloads — User requests a file, your API generates a pre-signed GET URL valid for 15 minutes, returns it to the client. Client downloads directly from S3, bypassing your server entirely.
• Direct-to-S3 uploads — Your API generates a pre-signed PUT URL. Client uploads directly to S3. Your server never touches the file data, eliminating a bandwidth and memory bottleneck.
• Temporary media access — Serve images or videos through pre-signed URLs with short expiration. Works with CloudFront signed URLs for even better performance.

import boto3
s3 = boto3.client('s3')

# Generate a download URL valid for 1 hour
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'reports/q4-2025.pdf'},
    ExpiresIn=3600
)

​

2.5 S3 Event Notifications

• Upload image to S3 -> Lambda generates thumbnails -> writes to another S3 bucket
• CSV uploaded to S3 -> SQS message -> worker processes and loads to database
• Log file uploaded -> EventBridge rule -> Step Functions workflow for analysis

​

2.6 S3 Select and Athena — Querying Data in Place

-- S3 Select: filter a CSV in S3 without downloading it
SELECT s.name, s.revenue
FROM S3Object s
WHERE s.revenue > 1000000

• S3 Select — Simple filtering on a single object. Quick, cheap, no setup.
• Athena — Complex queries across many objects. Joins, aggregations, window functions. Best with columnar formats (Parquet, ORC) that reduce data scanned (and therefore cost).

​

2.7 S3 Cost Traps

​

Part III — Compute Patterns

​

3.1 The Compute Decision Matrix

​

3.2 Fargate Spot — The Cost Optimization Cheat Code

​

3.3 Auto-Scaling Patterns

​

3.4 Graviton Processors — ARM on AWS

• 20-40% cheaper for compute-heavy workloads (no code changes needed for interpreted languages)
• Better energy efficiency (lower carbon footprint)
• Same or better single-threaded performance for most workloads

• Native compiled code (C, C++, Go, Rust) needs ARM cross-compilation
• Some third-party software does not have ARM builds yet
• Docker images must be built for ARM (--platform linux/arm64) or use multi-arch manifests

​

Part IV — Messaging and Event Patterns

​

4.1 The Messaging Decision Tree

​

4.2 SQS Deep Dive

​

4.3 EventBridge Patterns

{
  "source": ["my-app.orders"],
  "detail-type": ["OrderPlaced"],
  "detail": {
    "total": [{ "numeric": [">", 1000] }],
    "country": ["US", "CA"]
  }
}

​

4.4 Event-Driven Anti-Patterns

​

Part V — Database Patterns on AWS

​

5.1 RDS vs Aurora

• Availability requirements: Aurora’s storage layer survives losing two copies without read impact and three copies without write impact. RDS Multi-AZ failover takes 60-120 seconds; Aurora failover takes under 30 seconds.
• Read-heavy workloads: Aurora supports up to 15 read replicas (RDS supports 5), with replication lag typically under 20ms (RDS lag can be seconds).
• Auto-scaling storage: Aurora storage grows automatically in 10 GB increments up to 128 TB. No need to provision storage upfront.
• Large databases: For databases over 1 TB, Aurora’s distributed storage outperforms RDS’s EBS-based storage significantly.

• Small databases (< 100 GB) with moderate traffic
• Development and staging environments
• When you need Oracle or SQL Server (Aurora only supports PostgreSQL and MySQL)
• When your workload is write-heavy and does not benefit from Aurora’s read replica architecture

​

5.2 Aurora Serverless v2

​

5.3 ElastiCache — Redis vs Memcached on AWS

​

Part VI — Networking and Security on AWS

​

6.1 VPC Design Patterns

VPC (10.0.0.0/16)
├── Public Subnet A (10.0.1.0/24) -- AZ-a
│   └── ALB, NAT Gateway, Bastion Host
├── Public Subnet B (10.0.2.0/24) -- AZ-b
│   └── ALB, NAT Gateway
├── Private Subnet A (10.0.10.0/24) -- AZ-a
│   └── Application servers, ECS tasks
├── Private Subnet B (10.0.11.0/24) -- AZ-b
│   └── Application servers, ECS tasks
├── Database Subnet A (10.0.20.0/24) -- AZ-a
│   └── RDS Primary
└── Database Subnet B (10.0.21.0/24) -- AZ-b
    └── RDS Standby

• Gateway Endpoints (free): S3, DynamoDB. Use these always.
• Interface Endpoints ($0.01/hour +$0.01/GB): Everything else (SQS, ECR, Secrets Manager, CloudWatch). Use when the cost is less than NAT Gateway traffic.

​

6.2 Security Groups vs NACLs

​

6.3 IAM Least Privilege

• Never use root credentials. Create IAM users or roles for everything.
• Use roles, not long-lived access keys. EC2 instance profiles, ECS task roles, Lambda execution roles — all use temporary credentials that rotate automatically.
• Scope permissions narrowly. "Resource": "*" is almost never necessary. Specify the exact ARN of the resource.
• Use conditions. Restrict by IP range, VPC, time of day, MFA requirement, or source account.

{
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::my-bucket/reports/*"
}

​

6.4 Secrets Manager vs Parameter Store

​

6.5 WAF and Shield

​

6.6 Multi-Account Strategy

• Blast radius containment. A misconfigured IAM policy, a compromised credential, or a runaway resource in production cannot affect staging or other teams’ workloads. Each account is a hard security boundary.
• Cost isolation. Each account generates its own bill. No tagging strategy needed to separate prod from dev — it is structural. Finance gets clear cost attribution by account.
• Service limit isolation. Lambda concurrency limits, API Gateway throttling, EC2 instance limits — all are per-account. A runaway batch job in the data team’s account cannot starve the API team’s Lambda functions.
• Compliance boundaries. PCI-DSS, HIPAA, SOC2, and GDPR all benefit from isolating regulated workloads into dedicated accounts with stricter controls. Auditors love clean account boundaries.
• Independent deployments. Teams can deploy, experiment, and break things in their own accounts without coordination overhead.

• Organizational Units (OUs): Group accounts hierarchically. Common structure:

Root
├── Security OU
│   ├── Log Archive Account (centralized CloudTrail, Config)
│   └── Security Tooling Account (GuardDuty, Security Hub)
├── Infrastructure OU
│   ├── Networking Account (Transit Gateway, DNS, VPN)
│   └── Shared Services Account (CI/CD, container registries, artifact repos)
├── Workloads OU
│   ├── Production OU
│   │   ├── Team A Prod Account
│   │   └── Team B Prod Account
│   └── Non-Production OU
│       ├── Team A Dev Account
│       ├── Team A Staging Account
│       └── Team B Dev Account
└── Sandbox OU
    └── Developer Sandbox Accounts (experimentation, auto-cleanup)

• Consolidated billing: All accounts roll up to a single bill. Volume discounts (Savings Plans, Reserved Instances) apply across the entire organization. You buy RIs in one account and they apply to matching usage in any account.
• Service Control Policies (SCPs): Organization-wide guardrails that override IAM permissions. An SCP is a ceiling, not a floor — even if an IAM policy grants permission, an SCP can deny it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRegionsOutsideAllowed",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2", "eu-west-1"]
        }
      }
    }
  ]
}

• Region restriction: Deny all actions outside approved regions.
• Prevent root user access: Deny all actions by the root user (except account recovery).
• Require encryption: Deny s3:PutObject unless s3:x-amz-server-side-encryption is set. Deny rds:CreateDBInstance unless StorageEncrypted is true.
• Prevent leaving the organization: Deny organizations:LeaveOrganization on all accounts.
• Deny expensive services: Deny ec2:RunInstances for instance types larger than xlarge in sandbox accounts. Prevent accidental p4d.24xlarge launches that cost $32/hour.

• IAM cross-account roles: Account A’s Lambda assumes a role in Account B to read from Account B’s S3 bucket. The trust policy on Account B’s role explicitly allows Account A’s principal.
• Resource-based policies: S3 bucket policies, KMS key policies, and SNS topic policies can grant access to specific external accounts without role assumption.
• AWS RAM (Resource Access Manager): Share VPC subnets, Transit Gateway, and other resources across accounts without duplicating them.
• EventBridge cross-account event buses: Route events from workload accounts to a central observability account for unified monitoring.

• Account vending machine. Automate new account creation with Control Tower Account Factory or a custom pipeline (Terraform + Step Functions). Every new account is provisioned with: baseline SCPs applied via its OU, CloudTrail logging shipped to the centralized log archive, GuardDuty enrolled in the delegated administrator account, VPC with Transit Gateway attachment pre-configured, and a pre-seeded CI/CD role for the owning team. A developer requests an account through a self-service portal; 15 minutes later, the account is ready with all guardrails in place. No tickets, no waiting.
• Drift detection. AWS Config rules and Config Conformance Packs continuously audit every account against your organization’s baseline. Common rules: “all S3 buckets must have versioning enabled,” “no security groups allowing 0.0.0.0/0 inbound on port 22,” “all RDS instances must be encrypted.” When drift is detected, AWS Config sends findings to Security Hub, which triggers SNS notifications or auto-remediation Lambda functions. Without drift detection, your carefully crafted guardrails erode within weeks as engineers work around them.
• Centralized audit and compliance. AWS CloudTrail Organization Trail captures every API call across every account in the organization and delivers logs to the centralized log archive account. The log archive account has an S3 bucket with Object Lock (WORM — Write Once Read Many) so that even a compromised account cannot tamper with its own audit trail. CloudTrail Lake enables SQL-based queries across the organization-wide audit log for investigations.
• Delegated administrator accounts. Instead of running all security tools from the management account (which should have minimal usage), delegate administration to specialized accounts: GuardDuty delegated admin, Security Hub delegated admin, AWS Config delegated admin. This follows the principle of least privilege at the account level — the management account only manages organization structure and SCPs.

1. Your CI/CD platform (GitHub Actions, GitLab CI, CircleCI) issues an OIDC token to the running job. This token contains claims about the job: repository name, branch, workflow, actor.
2. The job calls AWS STS AssumeRoleWithWebIdentity, presenting the OIDC token.
3. AWS validates the token against the OIDC provider’s public keys (configured as an IAM Identity Provider in your account).
4. If valid, AWS returns temporary credentials (access key, secret key, session token) scoped to a specific IAM role with a short TTL (typically 1 hour).
5. The job uses these temporary credentials to deploy. When the job ends, the credentials expire automatically.

# GitHub Actions example -- no stored AWS credentials
name: Deploy
on:
  push:
    branches: [main]
permissions:
  id-token: write  # Required for OIDC
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubDeployRole
          aws-region: us-east-1
          # No access keys stored anywhere -- OIDC federation handles auth
      - run: aws ecs update-service --cluster prod --service api --force-new-deployment

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
      }
    }
  }]
}

​

Part VII — Cost Engineering

​

7.1 Reserved Instances vs Savings Plans vs Spot

1. Cover your steady-state baseline (databases, minimum API capacity) with Savings Plans or RIs.
2. Handle predictable burst capacity with On-Demand (auto-scaling groups).
3. Run fault-tolerant batch work on Spot.
4. Use Compute Savings Plans when you are not sure which instance types you will use — they apply across Lambda, Fargate, and EC2.

​

7.2 Cost Allocation Tags

​

7.3 The Serverless Tax

• NAT Gateway: $77/month for 1 TB of traffic. A t3.nano instance running as a NAT costs ~$4/month (with reduced availability and throughput).
• API Gateway REST: $3.50 per million requests. An ALB ($16/month + $0.008/LCU-hour) is cheaper above roughly 5 million requests/month.
• Managed Kafka (MSK): $200+/month minimum. Self-managed Kafka on EC2 can be cheaper at scale (but operational cost is significant).
• Lambda at scale: As shown in section 1.7, Lambda is 10-40x more expensive than containers at sustained high traffic.

• Your team is small and engineering time is more expensive than AWS bills
• Traffic is variable and unpredictable
• The operational risk of self-managing exceeds the cost savings
• You are optimizing for speed of delivery, not cost efficiency

​

Part VIII — Cloud-Agnostic Patterns

​

8.1 Patterns That Transfer Across Clouds

​

8.2 Patterns That Are Cloud-Specific

• Default failure mode: DynamoDB’s primary failure mode is hot partition throttling: if a single partition key receives disproportionate traffic, requests to that partition are throttled even if the table’s aggregate throughput is well within limits. Global tables add replication lag (typically sub-second, but can spike to seconds during regional degradation). The recovery pattern: monitor ThrottledRequests per partition, use write sharding for hot keys, and design your access patterns to distribute load evenly across the key space.

​

8.3 The Abstraction Layer Decision

• Portable: Resource definitions can target any cloud. Switching providers means rewriting .tf files, not application code.
• Not portable: The resources themselves are cloud-specific. Defining an Aurora cluster in Terraform does not make it portable — it just means your infrastructure definition is in a consistent language.
• Verdict: Use Terraform/Pulumi for consistency and automation, not for cloud portability.

• Portable: Kubernetes manifests (Deployments, Services, ConfigMaps) work on any Kubernetes cluster. Move from EKS to GKE by repointing kubectl.
• Not portable: Ingress controllers, storage classes, load balancer annotations, and IAM integration are cloud-specific. You will still have cloud-specific YAML.
• Verdict: Kubernetes gives you compute portability. Data layer, networking, and identity are still cloud-specific. This is why Kubernetes is popular for multi-cloud — it solves the hardest portability problem (running your applications) while accepting that storage and networking are different.

• Libraries like libcloud, fog, or cloud-agnostic SDK wrappers provide unified APIs across providers.
• The honest truth: These abstractions target the lowest common denominator. You lose the best features of each cloud. In practice, teams use them for basic operations (file upload, queue send) and still use native SDKs for advanced features.
• Verdict: Use for simple, commodity operations (put object, send message). Do not use for complex patterns (DynamoDB streams, BigQuery ML, Lambda event mappings).

​

Interview Questions — Cloud Service Patterns

S3 Upload -> S3 Event Notification -> SQS Queue -> Lambda (processor) -> S3 / DynamoDB
                                         |
                                         v (after 3 failures)
                                      DLQ -> Lambda (alerter) -> SNS -> PagerDuty/Slack

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:Query"
  ],
  "Resource": "arn:aws:dynamodb:us-east-1:123456789:table/Orders"
}

​

Real-World Architecture Examples

​

Example 1: E-Commerce Order Pipeline

User places order
  -> API Gateway -> Lambda (validate order)
  -> DynamoDB (store order, status: PENDING)
  -> SQS (order-processing queue)
  -> Lambda (process payment via Stripe API)
  -> EventBridge (OrderPaid event)
    -> Lambda: send confirmation email (SES)
    -> Lambda: update inventory (DynamoDB)
    -> Lambda: publish to analytics (Kinesis Firehose -> S3)
  -> DynamoDB (update order, status: CONFIRMED)

​

Example 2: Media Processing Pipeline

Client -> Pre-signed URL (PUT) -> S3 (raw-uploads bucket)
  -> S3 Event -> SQS -> Lambda (validate file, extract metadata)
  -> Step Functions workflow:
    -> Parallel:
      -> MediaConvert job: 1080p H.264
      -> MediaConvert job: 720p H.264
      -> MediaConvert job: 480p H.264
      -> Lambda: generate thumbnail from first frame
    -> Lambda: update DynamoDB (status: READY, URLs)
    -> Lambda: send notification (SNS -> user's webhook)

​

Example 3: Real-Time Fraud Detection

Transaction API -> Kinesis Data Streams (partitioned by card_id)
  -> Lambda (enhanced fan-out consumer):
    -> Call ML inference endpoint (SageMaker) for fraud score
    -> If score > 0.8: publish to SNS (block transaction)
    -> If score 0.5-0.8: publish to SQS (human review queue)
    -> All: write to DynamoDB (transaction log)
    -> All: Kinesis Firehose -> S3 (for model retraining)

​

Curated Resources

​

Interview Deep-Dive Questions

​

1. Walk me through exactly what happens inside AWS when a Lambda function cold-starts. Where does the time go, and what levers do you have at each stage?

• Stage 1 — Code download (~50-500ms). The Lambda service fetches your deployment package from an internal S3 cache. Larger packages take longer. Container images are cached closer to the execution fleet via ECR’s lazy-loading (SOCI — Seekable OCI), so only the layers you need at startup are fetched. Lever: Shrink package size. Use tree-shaking in Node.js, strip unused dependencies in Python. For container images, use a minimal base image (alpine or distroless) and order Dockerfile layers so frequently-changing code is last.
• Stage 2 — Microvm provisioning (~50-100ms). AWS uses Firecracker to spin up a lightweight microVM. This is largely outside your control, but it is fast because Firecracker was literally purpose-built for this — it boots a VM in ~125ms. Lever: Almost none directly. But choosing more memory indirectly allocates proportional CPU, which speeds up subsequent stages.
• Stage 3 — Runtime initialization (~10-5000ms). The language runtime starts. Python interpreter: fast. Node.js V8: fast. JVM with class loading and JIT warmup: 3-10 seconds. .NET CLR assembly loading: 1-3 seconds. Lever: Choose a lighter runtime for cold-start-sensitive paths. For Java, SnapStart snapshots the JVM state after init and restores from the snapshot, cutting cold start from seconds to 200-500ms. For .NET, Native AOT compiles ahead-of-time, eliminating CLR startup.
• Stage 4 — Init code execution (~10-2000ms). Your code outside the handler runs: imports, SDK client creation, DB connection establishment, config loading from Parameter Store or Secrets Manager. Lever: This is where you have the most control. Lazy-initialize anything you do not need on every invocation. Cache configuration in memory. Use the AWS SDK’s built-in credential provider instead of making explicit STS calls. If you call Secrets Manager during init, that is a network round-trip on every cold start.
• Stage 5 — Handler execution. This happens on every invocation, warm or cold. Not part of cold start.

​

Follow-up: Your team runs Java on Lambda and the CTO refuses to let you switch runtimes. Cold starts are 6 seconds. What do you do?

• Enable SnapStart immediately. This is the single biggest win for Java Lambda. AWS snapshots the JVM state after your init code runs using CRaC (Coordinated Restore at Checkpoint). On cold start, it restores from snapshot instead of re-initializing the JVM. This typically cuts cold starts from 6s to 400-800ms. It is a configuration flag — zero code changes for most applications.
• Audit your init code. Spring Boot with annotation scanning and component scanning is the biggest offender. If you are using Spring Boot, consider switching to Micronaut, Quarkus, or plain Java (no framework). Quarkus was designed for fast startup and has an explicit Lambda extension. If switching frameworks is off the table, at least disable annotation scanning for packages you do not need.
• Trim the dependency tree. Run mvn dependency:tree and look for transitive dependencies you do not use. Every JAR that the classloader touches adds startup time. Shade only the classes you use with the Maven Shade plugin.
• Provisioned concurrency as the final layer. After SnapStart and init optimization, if cold starts are still above SLA, provision enough warm instances to cover baseline traffic. Use Application Auto Scaling to track the ProvisionedConcurrencyUtilization metric and scale provisioned capacity with demand.
• Measure the cost trade-off. Provisioned concurrency for 50 instances at 512MB costs roughly $20/day. Compare that to the engineering cost of rewriting in another language. Usually, SnapStart plus provisioned concurrency is dramatically cheaper than a rewrite.

​

Follow-up: What are the gotchas with SnapStart that most people miss?

• Uniqueness assumptions break. If your init code generates a random seed, UUID, or unique identifier, the snapshot captures that value. Every restored instance starts with the same “random” value. This breaks security tokens, encryption IVs, and anything that assumes init-time uniqueness. You must use CRaC’s beforeCheckpoint and afterRestore hooks to re-initialize these values. The AWS docs call this out, but most developers miss it until they see duplicate “unique” IDs in production.
• Network connections are stale. A database connection opened during init is in the snapshot but the TCP connection is dead by the time the function restores. You must implement connection validation or re-establishment in afterRestore. Connection pools that do health checks on borrow (like HikariCP’s connectionTestQuery) handle this naturally, but raw connections do not.
• Deterministic encryption becomes non-deterministic. If your init code seeds a CSPRNG (cryptographically secure pseudorandom number generator), the snapshot freezes the seed state. All restored instances start from the same PRNG state. This is a subtle security vulnerability. AWS provides the software.amazon.lambda.snapstart.SnapshotRestore interface to re-seed after restore.

​

2. You are designing a system that processes 500,000 file uploads per day to S3. Each file needs validation, transformation, and loading into a database. Walk me through the architecture.

• Upload path: Clients upload via pre-signed PUT URLs generated by an API (Lambda behind API Gateway). Files land in an S3 bucket partitioned by date prefix (uploads/2026/04/10/). Pre-signed URLs let the client upload directly to S3, so our API never touches the file bytes — this eliminates a huge bandwidth and memory bottleneck.
• Event routing: S3 Event Notifications route to SQS (not directly to Lambda). I put SQS in the middle for three reasons: (1) it buffers during Lambda throttling or cold start bursts, (2) it gives me configurable retry with exponential backoff, and (3) it provides a DLQ for poison messages. I would set the visibility timeout to 6x the Lambda timeout per AWS best practices.
• Processing Lambda: Polls SQS, downloads the file from S3, validates it (schema check, file type, size limits, malware scan if required), transforms it (normalize fields, enrich with lookup data), and writes to the database. I would set reserved concurrency at 50-100 to protect the database from connection storms — this is critical.
• Database writes: If the target is DynamoDB, I would use BatchWriteItem for throughput. If it is Aurora PostgreSQL, I would use RDS Proxy to pool connections from Lambda (otherwise each concurrent Lambda opens its own connection and you exhaust max_connections within minutes). Batch inserts via COPY or multi-row INSERT rather than one-row-at-a-time.
• Error handling: After 3 failed processing attempts, SQS moves the message to a DLQ. A separate Lambda monitors the DLQ depth via CloudWatch Alarm. If the DLQ grows beyond a threshold, it triggers an SNS notification to PagerDuty. I would also write failed records to a separate S3 bucket (failed-uploads/) with the error details for manual reprocessing.
• Idempotency: Files may be delivered more than once (SQS is at-least-once). I would use a DynamoDB table or a database unique constraint on a hash of the file content or the S3 object key to prevent duplicate processing.

​

Follow-up: The product team now says some files are 5GB+. How does that change your architecture?

• Upload path: Files over 5GB require multipart upload. The pre-signed URL approach still works but I need to generate pre-signed URLs for each part (using CreateMultipartUpload, then UploadPart pre-signed URLs, then CompleteMultipartUpload). The client-side SDK handles this — the AWS SDK’s TransferManager in Java or boto3’s upload_file with multipart threshold in Python. I must also add a lifecycle policy to abort incomplete multipart uploads after 7 days to avoid hidden storage costs.
• Processing Lambda timeout: A 5GB file cannot be downloaded, validated, transformed, and loaded within Lambda’s 15-minute timeout. I have two options: (1) use S3 Select to process the file in place if it is CSV/JSON/Parquet — push the filtering to the storage layer, or (2) move processing to an ECS Fargate task that can run for hours. The SQS message would trigger a Step Functions workflow that launches a Fargate task instead of a Lambda.
• Memory constraints: Lambda maxes out at 10 GB memory. A 5GB file loaded entirely into memory leaves little room for processing. For Fargate, I would configure tasks with 8-16 GB memory and stream the file from S3 in chunks rather than loading it all at once.
• The hybrid approach: Keep Lambda for files under 100MB (the vast majority) and route large files to Fargate. The SQS consumer Lambda checks the file size from the S3 event metadata. Small files are processed inline; large files trigger a Step Functions workflow that launches a Fargate task. This optimizes cost (Lambda for the 99% of small files) while handling the edge case (Fargate for the 1% of large files).

​

Follow-up: How do you prevent Lambda from overwhelming your Aurora PostgreSQL database with connections?

• RDS Proxy is the primary solution. RDS Proxy sits between Lambda and Aurora, maintaining a pool of warm database connections. Hundreds of Lambda instances share a pool of, say, 100 database connections. RDS Proxy handles connection multiplexing, authentication caching, and automatic failover. It adds ~1ms of latency, which is negligible. Cost: roughly $21/month for a small instance.
• Reserved concurrency as a guard rail. Even with RDS Proxy, I would set reserved concurrency on the Lambda function to cap the maximum number of simultaneous executions. If the proxy pool is 100 connections and each Lambda holds one connection for 500ms, I would cap Lambda at 200 concurrent executions — this ensures the proxy is never overwhelmed.
• Connection reuse in Lambda. Initialize the database connection outside the handler (in the module scope). The connection persists across warm invocations of the same execution environment. This means a warm Lambda reuses its connection instead of opening a new one per invocation.
• The pattern I would avoid: Opening and closing a connection per invocation. This creates massive connection churn, wastes time on TCP + TLS + auth handshake on every request, and generates load on the database’s process management.

​

Going Deeper: What happens if RDS Proxy itself becomes the bottleneck?

​

3. Your company runs 200 Lambda functions across 5 teams. You are tasked with designing the AWS account strategy. What do you recommend?

• Management Account — AWS Organizations root. No workloads here. Only billing, SCPs, and Control Tower configuration.
• Security OU: Log Archive Account (centralized CloudTrail, Config, GuardDuty findings), Security Tooling Account (Security Hub, IAM Access Analyzer).
• Infrastructure OU: Shared Services Account (CI/CD pipelines, ECR repositories, shared Lambda layers, DNS zones in Route 53), Networking Account (Transit Gateway, VPN, if needed).
• Workloads OU: Split into Production OU and Non-Production OU. Within each, one account per team. So 5 production accounts and 5 development/staging accounts. Total: 10 workload accounts.
• Sandbox OU: Individual developer sandbox accounts with aggressive SCPs (no resources larger than medium, auto-cleanup after 7 days via AWS Nuke).

• Root OU: Deny all actions outside approved regions (us-east-1, eu-west-1). Deny root user access. Require S3 encryption. Prevent leaving the organization.
• Sandbox OU: Deny expensive instance types. Deny production service creation (RDS Multi-AZ, Aurora, large DynamoDB tables).
• Production OU: Deny deletion of CloudTrail logs. Deny disabling encryption. Require MFA for destructive actions.

​

Follow-up: Team A needs to read data from Team B’s DynamoDB table. How do you handle cross-account access without breaking isolation?

• Cross-account IAM role assumption (my default choice). Team B creates an IAM role in their account with a trust policy allowing Team A’s Lambda execution role to assume it. Team A’s Lambda calls sts:AssumeRole, gets temporary credentials, and reads from Team B’s DynamoDB table. The role in Team B’s account has a scoped policy: only dynamodb:GetItem and dynamodb:Query on the specific table, nothing else. This is auditable (CloudTrail logs every role assumption), revocable (delete the trust policy), and granular (limit by IP, VPC, or condition keys).
• DynamoDB Streams to EventBridge (for async data sharing). If Team A does not need synchronous reads but just needs to react to changes in Team B’s data, use DynamoDB Streams in Team B’s account to publish change events to EventBridge. EventBridge cross-account rules forward relevant events to Team A’s event bus. This fully decouples the teams — Team A does not need any access to Team B’s account.
• Shared data layer (for truly shared data). If the DynamoDB table is shared reference data (product catalog, config data), consider putting it in the Shared Services account with cross-account read access for all workload accounts. This is appropriate when the data is organizational, not team-owned.

​

Follow-up: How do you prevent SCP misconfigurations from causing a production outage?

• Test SCPs in a staging OU first. Create a mirror of your production OU structure with a test account. Apply the SCP there, run automated integration tests, and verify nothing breaks before promoting to the production OU.
• Use the SCP simulator. AWS IAM Policy Simulator can evaluate SCPs against specific API calls. Before applying a deny policy, simulate the actions your production services make (Lambda invoke, DynamoDB read/write, S3 access, CloudWatch logging) and verify they are not blocked.
• Deploy SCPs through CI/CD. SCPs should be in version control (Terraform or CloudFormation StackSets), deployed through a pipeline with approval gates and automatic rollback. Never apply SCPs manually through the console.
• Always include a break-glass exclusion. Every deny SCP should have a condition excluding a specific “break-glass” IAM role that can bypass the restriction in emergencies. This role should require MFA and be heavily audited, but it prevents you from locking yourself out.
• Monitor with CloudTrail. Set up CloudWatch alarms on AccessDenied events in CloudTrail. A sudden spike in access denied errors after an SCP change indicates something is broken.

​

4. Explain the difference between SQS, SNS, EventBridge, and Kinesis. When have you used each one, and when have you made the wrong choice?

• SQS is a task queue. One message goes to one consumer. Once processed, it is deleted. Think of it as a to-do list: each item gets assigned to one worker. I use SQS for work distribution — processing uploaded files, handling order fulfillment, running async jobs. The killer feature is simplicity: no shards to manage, no partitioning strategy, nearly infinite throughput for Standard queues, built-in DLQ, and visibility timeout for safe concurrent processing.
• SNS is a megaphone. One message goes to many subscribers (Lambda, SQS, HTTP, email, SMS). It is fan-out with no retention — if the subscriber is down, the message is lost (unless the subscriber is an SQS queue, which buffers it). I use SNS when a single event should trigger multiple independent reactions: “order placed” triggers email, inventory update, and analytics simultaneously.
• EventBridge is SNS with brains. It adds content-based filtering (only route high-value orders to fraud detection), schema registry (know what your events look like), cross-account routing, and archive/replay. I use EventBridge as the default for new event-driven integrations. It is more expensive per event than SNS, but the filtering alone eliminates hundreds of lines of consumer-side if statements.
• Kinesis is a log. Events are ordered within a shard, retained for 1-365 days, and multiple consumers read independently at their own pace from any point in the stream. I use Kinesis for real-time analytics (clickstream data), change data capture (reacting to database changes), and any use case where I need to replay the event history (reprocess last 24 hours after deploying a bug fix).

​

Follow-up: You need exactly-once processing. Which of these gives it to you?

• SQS FIFO claims “exactly-once delivery” within a 5-minute deduplication window. It deduplicates messages with the same MessageDeduplicationId. But “exactly-once delivery” is not “exactly-once processing.” If your Lambda reads the message, processes it, writes to the database, and then crashes before deleting the message from SQS, the message becomes visible again and gets processed a second time. You still need idempotency in your consumer.
• Kinesis is explicitly at-least-once. Checkpointing happens after processing. If the consumer crashes between processing and checkpointing, the record is reprocessed.
• SNS and EventBridge are at-least-once. Retry logic can deliver the same event multiple times.

​

Going Deeper: You mentioned Kinesis shard blocking. Walk me through what happens and how to prevent it.

• bisectBatchOnFunctionError: true — After a failure, Lambda splits the batch in half and retries each half. This narrows down to the single failing record through binary search rather than blocking on the entire batch.
• maxRetryAttempts — Set a maximum number of retries (e.g., 3-5). After exhausting retries, the failing record is sent to the on-failure destination.
• On-failure destination — Route failed records to an SQS queue or SNS topic for investigation. This is your DLQ equivalent for stream sources.
• Error handling in your function — Catch exceptions per record within the batch. Process what you can, collect failures, and use batchItemFailures response (a relatively newer Lambda feature) to report which specific records failed. Lambda retries only the failed records, not the entire batch.
• maxRecordAge — Set a maximum age for records. If a record is older than this threshold, Lambda skips it and sends it to the on-failure destination. This prevents ancient records from blocking the shard indefinitely.

​

5. You are building a new microservice. A colleague says “just put it on Lambda” and another says “use Fargate.” How do you make this decision?

• What is the traffic pattern? If it is spiky (zero at night, peak during the day), Lambda’s scale-to-zero saves money. If it is steady 24/7 traffic, Fargate’s always-on pricing wins.
• What is the latency SLA? If p99 must be under 100ms, Lambda cold starts are a risk unless I pay for provisioned concurrency. Fargate is always warm.
• How long does a single request take? Lambda has a 15-minute hard timeout. If the service does long-running work (video processing, ML training, large file transformations), Lambda is out.
• What are the dependency requirements? If the service needs 2GB of ML model files, Lambda’s container image path works but cold starts will be painful. Fargate handles large images without cold start penalties.
• Does it talk to a relational database? Lambda-to-RDS without RDS Proxy is a foot-gun (connection storms). Fargate maintains persistent connection pools naturally.
• What does the team know? If the team has deep container experience with existing Dockerfiles, CI/CD pipelines, and monitoring, Fargate is lower friction. If the team is small and does not want to manage containers, Lambda is simpler.

​

Follow-up: At what traffic volume does the cost crossover from Lambda to Fargate typically happen?

• Below 1 million requests/month: Lambda wins overwhelmingly. The free tier alone covers most of this. A Fargate task running 24/7 costs at minimum ~$9/month regardless of traffic.
• 1-10 million requests/month: Comparable cost. Lambda is roughly $10-80/month. A single Fargate task handling this load costs ~$9-20/month. The difference is not worth optimizing for.
• 10-100 million requests/month: Fargate wins on raw compute cost. Lambda at 50M requests/month with 256MB/200ms costs ~$200/month. Two Fargate tasks can handle the same load for ~$40/month.
• Above 100 million requests/month: Fargate or ECS on EC2 is dramatically cheaper. At this scale, consider EC2 with Savings Plans — you might save another 50-70%.

​

Follow-up: What about App Runner? When does that fit?

​

6. A critical production service depends on DynamoDB. The table is getting hot-partition throttling at 3 AM. Walk me through your diagnosis and fix.

1. Check CloudWatch Contributor Insights. Enable it on the table — it shows the most frequently accessed and throttled partition keys. This immediately tells you which keys are hot. If the top key accounts for 30% of all reads at 3 AM, you have found the problem.
2. Correlate with application behavior. What runs at 3 AM? Batch jobs, cron tasks, data exports, report generation. A nightly job scanning a specific partition key range (e.g., all orders from yesterday using a partition key of date=2026-04-09) concentrates all traffic on one partition.
3. Check if it is on-demand or provisioned. On-demand tables can still throttle if traffic exceeds the table’s previous peak by more than 2x within 30 minutes. Provisioned tables throttle when consumed capacity exceeds allocated capacity per partition.

• Immediate: Increase provisioned capacity or switch to on-demand. This buys time but does not fix the root cause if the partition key design is skewed.
• Short-term: Add write sharding to the hot key. Append a random suffix (e.g., date=2026-04-09#3) to spread writes across multiple physical partitions. The batch reader scatters-gathers across all suffixes. This is the standard DynamoDB hot-partition pattern.
• Medium-term: Redesign the access pattern. If the 3 AM job scans by date, and the partition key is date, every daily scan hits one partition. Restructure the key to include another dimension (e.g., PK = customer_id, SK = date) so that the scan is distributed. Use a GSI if query access patterns require the date-first lookup.
• Architectural: DAX (DynamoDB Accelerator). If the 3 AM traffic is read-heavy and reads the same data repeatedly, put DAX in front of the table. DAX caches reads at the item and query level, absorbing the burst without hitting the underlying partition.

​

Follow-up: The hot key is a single global counter that multiple services increment. How do you handle that?

• Choose shard count based on expected writes per second. Each shard can handle ~1,000 WCUs, so 10 shards handle ~10,000 increments per second.
• The trade-off is read complexity: getting the current total requires reading all shards. For use cases where you only need an approximate count or can tolerate 1-second staleness, cache the total in a separate item and update it periodically.
• If exact real-time counts are needed, use DynamoDB Streams to aggregate changes into a single total asynchronously.

​

7. Your S3 data transfer bill is $15,000/month. How do you bring it down?

• S3 to internet (most expensive at $0.09/GB)
• S3 to CloudFront (cheaper at $0.00/GB for origin fetches in most regions)
• S3 cross-region (e.g., replication to another region at $0.02/GB)
• S3 to EC2/Lambda in the same region (free — but going through NAT Gateway costs $0.045/GB)

​

Follow-up: You put CloudFront in front of S3 and cache hit ratio is only 20%. Why, and how do you fix it?

• Unique URLs per user. If each request includes a unique query string parameter (session token, timestamp, user ID), CloudFront treats each URL as a unique cache key. Fix: Configure CloudFront to ignore irrelevant query strings in the cache key. Whitelist only query parameters that actually change the content (e.g., size=thumbnail vs size=full).
• Low TTL or Cache-Control: no-cache headers. If S3 objects have no Cache-Control header or set max-age=0, CloudFront does not cache them (or caches very briefly). Fix: Set appropriate Cache-Control headers on S3 objects. Static assets: max-age=31536000, immutable. Dynamic content: use a shorter TTL but still cache (even max-age=60 dramatically reduces origin load).
• Long-tail content. If you serve millions of unique objects and each is accessed once per day, even perfect caching does not help because the first request is always a miss. Fix: This is a content distribution problem, not a caching problem. Consider pre-warming the cache for popular content or accepting the miss rate and focusing on reducing object sizes instead.
• Single edge location. If all traffic comes from one region but CloudFront distributes across hundreds of edge locations, each edge sees low traffic and evicts cached content quickly. Fix: Use CloudFront’s Price Class to restrict to fewer edge locations (higher hit rate per location) or use Origin Shield (an additional caching layer between edge locations and S3 that centralizes cache fills).

​

8. Your team wants to go multi-region active-active. When would you argue against it?

• The business does not actually need zero-downtime failover. Most services can tolerate 30-60 seconds of downtime during an active-passive failover. Ask the business: “What is the cost per minute of downtime?” If the answer is less than the engineering cost of maintaining active-active, it is not worth it. For most SaaS companies, active-passive with Route 53 health checks and automated failover provides 99.95%+ availability.
• The data layer has strong consistency requirements. Active-active means writes happen in multiple regions simultaneously. For eventually consistent data (user preferences, analytics, content caches), this is manageable. For strongly consistent data (financial balances, inventory counts, sequential ordering), you enter split-brain territory. DynamoDB Global Tables give you eventual consistency with ~1 second replication lag. That 1-second window is where duplicate orders, double-charges, and inventory over-sells live.
• The team does not have the operational maturity to run it. Active-active requires: per-region monitoring and alerting, automated failover testing, data conflict resolution strategies, region-aware routing, region-specific deployment pipelines, and engineers who can debug cross-region replication issues at 3 AM. If the team has not mastered single-region operations (observability, incident response, deployment automation), adding a second region multiplies problems rather than solving them.
• The cost does not justify the benefit. Active-active roughly doubles your infrastructure cost (two of everything) plus adds 30-50% operational overhead (cross-region replication, routing, testing). For a service doing $1M ARR, spending$500K on active-active infrastructure is absurd. For a service doing $100M ARR where an hour of downtime costs$500K, it is a no-brainer.

• Regulatory requirement (EU data must be served from EU, US from US) — but this is often better solved with region-pinned routing, not true active-active.
• Latency-sensitive global user base where 200ms cross-ocean latency is unacceptable.
• The business genuinely cannot tolerate the 30-60 seconds of active-passive failover (financial trading, real-time gaming, emergency services).

​

Follow-up: If you do go active-active, how do you handle data conflicts?

• Last-writer-wins (LWW). DynamoDB Global Tables use this by default — the write with the latest timestamp wins. This is fine for data where the most recent value is always correct (user profile updates, session data, preferences). It is dangerous for counters, balances, or any additive data where both writes carry information.
• Application-level conflict resolution. Instead of overwriting, design your data model so writes are additive. Use event sourcing: instead of updating a balance, append a debit/credit event. Both regions can append independently, and the balance is derived by replaying events. Conflicts become “concurrent events” that are both valid.
• Region-pinning with failover. Assign each entity (user, account, order) a “home” region. All writes for that entity go to its home region. The other region serves reads from the replica. On failover, the secondary region promotes to writer. This avoids conflicts entirely at the cost of cross-region write latency for entities whose users are in the “wrong” region.
• CRDTs (Conflict-Free Replicated Data Types). For specific data structures (counters, sets, flags), CRDTs guarantee that concurrent updates from different regions merge deterministically without conflicts. This is theoretically elegant but requires redesigning your data model around CRDT-compatible structures. Practically, it works for counters (G-Counter), boolean flags (LWW-Register), and sets (OR-Set). It does not work for arbitrary relational data.
• My recommendation for most teams: Region-pinning with failover. It sidesteps the conflict problem entirely, is simple to reason about, and gives you active-active read scalability with single-region write simplicity. True active-active writes to the same data from multiple regions should be reserved for teams with deep distributed systems expertise.

​

9. Explain VPC Endpoints — Gateway vs Interface. When do you use each, and what is the cost impact?

• Available only for S3 and DynamoDB.
• Free. No hourly charge, no data processing charge.
• Implemented as a route table entry. You add the endpoint, associate it with your private subnet route tables, and traffic to S3/DynamoDB is routed directly through the AWS backbone — it never leaves the AWS network and never touches your NAT Gateway.
• You should enable these on every VPC with private subnets. There is zero reason not to.

• Available for 100+ AWS services (SQS, SNS, ECR, CloudWatch Logs, Secrets Manager, KMS, etc.) and third-party services.
• Cost: ~$0.01/hour per AZ (~$7.30/month per AZ) plus $0.01/GB of data processed.
• Implemented as an ENI (Elastic Network Interface) in your subnet. The endpoint gets a private IP address in your VPC, and traffic to the service resolves to that private IP via a private hosted zone.
• Use when: the data processing savings through avoiding NAT Gateway ($0.045/GB) exceed the endpoint hourly cost. For high-traffic services (ECR image pulls, CloudWatch Logs ingestion), the break-even is usually within the first few GB per month.

​

Follow-up: You have Interface Endpoints for ECR in two AZs but container image pulls are still going through the NAT Gateway. What went wrong?

• Private DNS is not enabled. When you create an Interface Endpoint, you can enable “Private DNS.” This creates a private hosted zone that overrides the public DNS name (e.g., api.ecr.us-east-1.amazonaws.com) to resolve to the endpoint’s private IP address. If private DNS is not enabled, your ECS tasks still resolve the ECR hostname to its public IP, which routes through the NAT Gateway.
• DNS resolution is not enabled on the VPC. The VPC must have enableDnsSupport and enableDnsHostnames set to true for private hosted zones to work. Check VPC settings.
• There are actually two ECR endpoints required. ECR requires both com.amazonaws.region.ecr.api (for API calls like authentication) and com.amazonaws.region.ecr.dkr (for Docker image layer downloads). Missing either one causes partial or complete fallback to the public endpoint. You also need an S3 Gateway Endpoint because ECR stores image layers in S3.
• Security group on the endpoint is too restrictive. Interface Endpoints have security groups. If the security group does not allow inbound HTTPS (port 443) from your ECS tasks’ security group, the connection fails and the SDK falls back to the public endpoint.
• The endpoint is in the wrong AZs. If your ECS tasks run in AZs where the endpoint does not have an ENI, traffic from those AZs goes through the public path. Create the endpoint in all AZs where your tasks run.

​

10. You are designing the CI/CD pipeline for a team with 30 Lambda functions. What is your deployment strategy?

1. On PR: Lint, unit test, sam build to verify the functions compile/package. Run integration tests against a dev-stage AWS account.
2. On merge to main: Build all changed functions, push artifacts to S3 (zip) or ECR (container images), deploy to staging account.
3. Staging validation: Run smoke tests against the staging deployment. If all pass, proceed to production.
4. Production deployment: Deploy with traffic shifting using Lambda aliases and CodeDeploy integration.

• Each function has a live alias pointing to the current production version.
• On deployment, I publish a new version and shift the live alias using CodeDeploy with a canary or linear strategy:
  • Canary10Percent5Minutes: Route 10% of traffic to the new version. If CloudWatch alarms (error rate, latency p99, custom business metrics) trigger within 5 minutes, auto-rollback. If clean, shift to 100%.
  • Linear10PercentEvery1Minute: For less risky changes, shift 10% every minute over 10 minutes.
• The rollback is instant — just re-point the alias to the previous version. No re-deployment needed.

• Publish the shared layer as a versioned Layer. Pin production functions to a specific layer version (not $LATEST).
• Layer updates go through the same canary deployment. Deploy the new layer to one function first, validate, then roll out to all functions.

​

Follow-up: How do you handle database schema migrations in this serverless CI/CD pipeline?

• Dedicated migration Lambda. A Lambda function whose sole purpose is running database migrations. It is triggered as a custom resource in CloudFormation/CDK or as a step in the CI/CD pipeline (invoke via AWS CLI after deployment). It connects to the database via RDS Proxy, runs the migration scripts (using a tool like Flyway, Alembic, or Knex), and reports success/failure.
• Backward-compatible migrations only. Since Lambda functions are deployed with canary traffic shifting, the old version and new version run simultaneously during deployment. The database schema must be compatible with both versions. This means: add columns (nullable or with defaults), never rename or drop columns during deployment. Use a two-phase migration: Phase 1 deploys the new schema (additive only) and the new code that can use both old and new schema. Phase 2 (days later, after all traffic has shifted) deploys a cleanup migration that removes the old columns.
• Migration as a pre-deploy step. In the CI/CD pipeline, run the migration Lambda before deploying the new function code. If the migration fails, the deployment stops and the old code continues running against the old schema. If the migration succeeds, deploy the new code that uses the new schema.
• For DynamoDB: There are no schema migrations in the traditional sense (DynamoDB is schemaless). But adding a new GSI takes time and consumes write capacity. Add new GSIs in a separate deployment step, monitor the backfill progress, and only deploy the code that queries the new GSI after the backfill completes.

​

Going Deeper: How do you test 30 Lambda functions locally before deploying?

​

11. What is the most expensive mistake you have seen (or made) on AWS, and what did you learn from it?

• Always use prefix filters on S3 event notifications. Source prefix and destination prefix must be different, and the event notification must filter to only the source prefix.
• Set reserved concurrency on every Lambda function. If this function had reserved concurrency of 10, the loop would have been throttled and the blast radius contained.
• AWS now has recursive loop detection for Lambda-S3-Lambda loops (released 2023). But do not rely on detection — design to prevent recursion.
• Set up AWS Budgets with action thresholds. A budget action can automatically apply an SCP that denies Lambda invocations if the monthly spend exceeds a threshold. We now have a $500 daily anomaly alert on every account.
• Use a separate destination bucket. The simplest architectural fix is never writing output to the same bucket as input. Two buckets, one event notification, zero recursion risk.

​

Follow-up: How do you set up cost guardrails to prevent runaway bills on a new AWS account?

• AWS Budgets: Create a monthly budget with alerts at 50%, 80%, and 100% of expected spend. At 100%, trigger a budget action that notifies via SNS and optionally applies an SCP restricting resource creation.
• Cost Anomaly Detection: Enable it on the billing account. It uses ML to detect unusual spending patterns and alerts within hours rather than waiting for the end of the month. Configure alerts for both percentage-based (50% above forecast) and absolute-dollar thresholds ($100 anomaly).
• SCPs for sandbox accounts: Deny creation of expensive resources (large EC2 instances, large RDS instances, Redshift clusters, SageMaker endpoints). Deny actions in regions you do not use.
• Lambda concurrency limits: Set account-level concurrency limits in non-production accounts to a low number (100-200). This prevents any single runaway function from generating millions of invocations.
• S3 lifecycle policies from day one: Abort incomplete multipart uploads after 3 days. Expire old object versions after 30 days. Transition to cheaper storage classes on a schedule.
• Tag enforcement: Use AWS Organizations Tag Policies to require team and environment tags on all resources. Untagged resources generate unattributable costs that nobody optimizes.

​

12. You need to migrate a monolithic application from EC2 to a cloud-native architecture. How do you approach this?

• Database: Do not try to split the database first. That is the hardest part. Start with the monolith and new services sharing the same database (via RDS Proxy if needed). Extract service-specific tables into their own databases (DynamoDB, separate RDS instances) later, once the service boundaries are stable.
• Authentication: Extract auth into a shared service early (Cognito, Auth0, or a custom auth service). Every new microservice needs auth, and having a consistent auth layer prevents each service from reimplementing it differently.
• Shared data: Use an event bus (EventBridge) from the beginning. When the monolith writes an order, it publishes an OrderCreated event. New services subscribe to events rather than querying the monolith’s database. This decouples the extraction pace from the data migration pace.

​

Follow-up: The CTO wants the migration done in 3 months. How do you push back?

​

Going Deeper: How do you handle the database during migration — do you split it early or late?

​

Advanced Interview Scenarios

​

13. Your Lambda function works fine in dev but randomly times out in production — about 5% of invocations hit the 30-second limit. CloudWatch shows the function uses only 60% of allocated memory. What is going on?

• “Increase the timeout to 60 seconds or give it more memory.”
• “It is probably cold starts causing the timeouts.”
• “Add retry logic in the function.”

• Step 1: Enable X-Ray tracing. X-Ray shows the time breakdown per invocation. When I had this exact problem on a payment processing Lambda, X-Ray revealed that 5% of invocations spent 28 seconds waiting on an HTTP call to a third-party fraud detection API. The API had a long-tail latency problem — p50 was 200ms but p99 was 29 seconds. Without X-Ray, CloudWatch only shows total duration, which is useless for multi-step functions.
• Step 2: Check the CPU angle. Lambda allocates CPU proportional to memory. At 128MB, you get roughly 1/10th of a vCPU. If the function does CPU-intensive work (JSON parsing of large payloads, image manipulation, crypto operations), it can be CPU-starved even with memory headroom. I once debugged a Node.js Lambda that parsed a 5MB JSON payload — at 128MB memory, parsing took 8 seconds. At 1024MB memory, it took 400ms. The fix was increasing memory from 128MB to 512MB, which cut p99 latency from 12 seconds to 1.2 seconds. The extra memory was irrelevant — it was the CPU that mattered.
• Step 3: Check DNS resolution and connection establishment. Lambda in a VPC resolves DNS through the VPC’s DNS resolver, which can be slow under load. Also check if the function creates new TCP/TLS connections per invocation instead of reusing them. A TLS handshake to a downstream service adds 100-300ms, and if the downstream has connection limits, the handshake can queue for seconds.
• Step 4: Check for connection pool exhaustion on downstream services. If the Lambda talks to RDS without RDS Proxy, and 200 concurrent Lambda instances each hold a connection, Aurora’s max_connections might be exhausted. New connections queue until one frees up, causing sporadic timeouts on the functions waiting for a connection. CloudWatch metric DatabaseConnections on the RDS side confirms this.
• Step 5: Set explicit socket timeouts. The AWS SDK defaults to no socket timeout in some configurations. If a downstream service silently drops a connection (no RST, no FIN), the Lambda hangs until its own timeout. I always set connectTimeout: 3000, socketTimeout: 10000 on every HTTP client. This converts a 30-second timeout into a 10-second error with a clear message.

​

Follow-up: How would you differentiate between a CPU-bound timeout and a network-bound timeout without X-Ray?

​

Follow-up: You discover it is CPU-bound. Increasing memory from 128MB to 1769MB gives you a full vCPU. But the function only needs 80MB of memory. Are you paying for waste?

​

14. Your team chose Step Functions to orchestrate an order fulfillment workflow. Six months later, the monthly Step Functions bill is $8,000 and growing. The CTO asks why “serverless” is so expensive. Diagnose and fix.

• “Step Functions is just expensive, we should rewrite it.”
• “Move everything to Lambda and SQS.”
• “The CTO should expect high costs for serverless at scale.”

• Pull the Step Functions CloudWatch metrics. ExecutionsStarted tells me volume. StateMachinesCount and per-machine execution counts tell me if it is one workflow or many. Most importantly, look at the number of states per execution — StatesExecuted / ExecutionsStarted gives average transitions per workflow.
• Check for “chatty” workflows. I have seen teams put every micro-operation as a separate Step Functions state: validate input (1 transition), check inventory (2), reserve inventory (3), process payment (4), handle payment failure choice (5), update order (6), send email (7), update analytics (8), log completion (9). That is 9 transitions per order. But the real killer is retry states and map states. A Map state iterating over 100 line items in an order fires 100+ transitions per order. If each line item has 3 states, a 100-item order costs 300+ transitions.
• Identify the fix — Standard to Express migration for eligible workflows. Express Workflows charge per execution and duration ($0.00001667/GB-second), not per transition. For the order fulfillment example: a workflow running 1 million times/month at 5 seconds average with 64MB memory costs roughly$0.83/month on Express. The same workflow on Standard with 10 transitions costs $250/month. Express is 300x cheaper for high-volume, short-duration workflows.
• The catch with Express Workflows: They have a 5-minute maximum duration, are asynchronous by default (no waiting for the result), and do not record execution history in the console (you must log to CloudWatch). If the order workflow takes longer than 5 minutes (waiting for payment confirmation, external API calls), Express does not work and you need a hybrid approach.
• Hybrid approach: Use Express for the high-volume, fast inner loop (process each line item) and Standard for the outer orchestration (order lifecycle with human approval, long waits). The Express workflow is called as a nested workflow within the Standard outer workflow.

​

Follow-up: When would you replace Step Functions entirely with event choreography (SQS + EventBridge)?

​

Follow-up: How do you test Step Functions workflows locally?

​

15. An S3 bucket contains 50TB of customer data. A security audit reveals the bucket was publicly accessible for 72 hours due to a misconfigured bucket policy. The data was not encrypted at rest. Walk me through your incident response AND how you prevent recurrence.

• “Remove the public policy immediately and enable encryption.”
• “Check CloudTrail to see who changed the policy.”
• “Turn on S3 Block Public Access.”

• Enable S3 Block Public Access at the account level, not just the bucket level. This overrides any bucket policy, ACL, or access point policy that grants public access. It is a single API call: aws s3control put-public-access-block --account-id 123456789 --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true. This immediately locks down every bucket in the account.
• Enable default encryption on the bucket with SSE-S3 or SSE-KMS. Existing objects are not retroactively encrypted — you must re-upload or use S3 Batch Operations to copy objects in place with encryption. For 50TB, S3 Batch Operations takes hours but runs server-side.
• Restrict bucket access to only specific IAM roles via an explicit deny bucket policy.

• Pull S3 Server Access Logs for the bucket during the 72-hour window. These logs show every GET, PUT, LIST request including source IP, requester, and user agent. If access logging was not enabled (common oversight), check CloudTrail S3 data events — but these are only available if data event logging was enabled. If neither was enabled, you have a visibility gap.
• Analyze access logs for anomalous IPs, bulk downloads, or programmatic access patterns (many GET requests in rapid succession from non-internal IPs). Tools: Athena query over the access logs in S3, or import into a SIEM (Splunk, Elastic).
• Check AWS CloudTrail for the PutBucketPolicy or PutBucketAcl event that opened the bucket. This tells you who changed the policy, from which IP, and at what time. Determine if it was intentional (developer testing who forgot to revert) or compromised credentials.

• If the data contains PII (names, emails, financial records), this likely triggers GDPR Article 33 (72-hour notification to the supervisory authority), CCPA breach notification, HIPAA breach notification, or industry-specific requirements. Notify the legal team immediately. The notification clock starts when you discover the breach, not when it occurred.
• Determine the scope: which customers’ data was in the bucket, what data fields were exposed, was any data actually downloaded by unauthorized parties. The access logs from Track 2 determine this.

• SCP: Deny public S3 access at the organization level. Apply an SCP that denies s3:PutBucketPolicy and s3:PutBucketAcl when the policy grants public access. This makes it structurally impossible for any account in the organization to make a bucket public, regardless of individual IAM permissions.
• AWS Config rule: s3-bucket-public-read-prohibited. This continuously monitors all buckets and alerts (or auto-remediates) if any bucket becomes public. Set it to auto-remediate by invoking a Lambda that re-applies the Block Public Access configuration.
• Macie for sensitive data discovery. Enable Amazon Macie on all S3 buckets to automatically discover and classify sensitive data (PII, financial data, credentials). Macie would have flagged this bucket as containing sensitive data that was publicly accessible.
• S3 Block Public Access as the default. Enable Block Public Access at the organization level via SCP so every new account inherits the restriction. The only way to make a bucket public is to first remove the organization-level block — which requires Security OU approval.
• Mandatory encryption via SCP. Deny s3:PutObject when s3:x-amz-server-side-encryption is not set. This forces encryption on all new objects organization-wide.

​

Follow-up: The developer says they needed the bucket to be public for a legitimate use case (serving static assets for a marketing site). How do you accommodate that safely?

• Dedicated bucket with a name that clearly indicates it is public (e.g., company-public-assets-prod).
• Block Public Access is disabled on only this bucket, with a documented exception approved by the security team.
• The bucket contains only static assets (HTML, CSS, JS, images). No customer data, no PII, no application data.
• A bucket policy with a Condition restricting uploads to only the CI/CD pipeline’s IAM role — no developer can upload directly.
• CloudFront in front of the bucket with OAC (Origin Access Control), so the bucket is not directly accessible — only CloudFront can read from it.
• AWS Config rule monitors this specific bucket and alerts if any object matching sensitive data patterns (SSN regex, email regex, etc.) is uploaded.

​

16. You inherit a system with 40 Fargate tasks running 24/7 in production. The monthly AWS bill is $12,000. The CTO asks you to cut it by 50%. How?

• “Buy Reserved Instances or Savings Plans.”
• “Move to Lambda.”
• “Use Spot instances.”

• Pull CloudWatch metrics for every task: CPU utilization average and p99, memory utilization average and peak. In my experience, 60-70% of Fargate tasks are over-provisioned. I have seen teams running 2 vCPU / 4GB tasks at 8% average CPU and 15% memory utilization because they copied the task definition from a template and never revisited it.
• AWS Compute Optimizer provides right-sizing recommendations for ECS tasks. It analyzes 14 days of CloudWatch metrics and suggests optimal CPU/memory configurations. It is free and takes 5 minutes to check.
• For a task running at 8% CPU / 15% memory on 2 vCPU / 4GB, dropping to 0.5 vCPU / 1GB cuts cost by 75% for that task. Across 40 tasks, this alone could save $4,000-6,000/month.

• If those 40 tasks include staging, dev, and QA environments running 24/7, schedule non-production environments to run only during business hours (8am-8pm weekdays = 60 hours/week vs 168 hours/week). Use ECS Scheduled Scaling or a Lambda that scales desired count to 0 at night and back up in the morning. This alone cuts non-prod costs by 64%.
• Better yet, use Fargate Spot for non-production. 70% discount with the only risk being occasional task termination — which is acceptable in dev/staging.

• Switch task definitions from x86 (X86_64) to ARM (ARM64) and use Graviton-based Fargate tasks. This is a single line change in the task definition if you build multi-arch Docker images. Graviton Fargate is 20% cheaper than x86 Fargate for the same CPU/memory, with equal or better performance. For interpreted languages (Python, Node.js, Ruby), no code changes are needed. For compiled languages (Go, Rust, Java), you rebuild for linux/arm64.

• Only after right-sizing and scheduling, commit to a Compute Savings Plan. A 1-year no-upfront Compute Savings Plan saves ~20%. A 3-year partial-upfront saves ~35%. Compute Savings Plans apply across Fargate, Lambda, and EC2, so they flex as your architecture evolves. I would commit to covering only 70% of the right-sized baseline — leave 30% on-demand for flexibility.

• Are any tasks doing batch processing that could be moved to Lambda (no idle cost between batches)?
• Are any tasks running singleton workers (one task processing a queue) that could be Lambda SQS consumers?
• Are there sidecar containers (log forwarders, metrics agents) that could be replaced with Firelens (built into ECS) or CloudWatch agent (no sidecar needed)?

​

Follow-up: How do you prevent the costs from creeping back up over the next 6 months?

• Weekly cost review. A 15-minute team meeting every Monday that reviews the Cost Explorer dashboard. Not a detailed analysis — just “are we trending up or down vs last week?” Anomalies get investigated immediately.
• AWS Budgets per service and per team. Set a monthly budget that is 10% above the optimized baseline. Alert at 80%, 90%, and 100%. At 100%, require a justification ticket for the increase.
• Right-sizing automation. A monthly Lambda function that runs Compute Optimizer recommendations and posts them to Slack. If a task has been over-provisioned for 30+ days, it opens a Jira ticket automatically.
• Tagging enforcement. Require cost-center and team tags on all ECS services via AWS Config rules. Untagged resources cannot be attributed and therefore cannot be optimized.
• Governance in the deployment pipeline. The CDK/Terraform pipeline runs a cost estimation tool (Infracost) on every PR. If the estimated monthly cost increase exceeds a threshold ($100/month), the PR requires a finance-team approval label.

​

17. A teammate says: “We should use DynamoDB for everything — it scales infinitely and costs less than RDS.” Where are they right, and where is this dangerously wrong?

• “They are right — DynamoDB handles any scale and is fully managed.”
• “They are wrong — relational databases are always better for complex queries.”

• Scaling. DynamoDB on-demand mode genuinely scales to virtually unlimited throughput. I have seen tables handling 400,000 reads/second with single-digit millisecond latency. You do not manage shards, replicas, or connection pools. It just works.
• Operational simplicity. No patching, no vacuuming, no connection pool tuning, no replica lag monitoring. For a small team, this operational savings is massive — it is easily worth $20K/year in engineering time you do not spend.
• Performance at scale. For key-value and simple query patterns, DynamoDB’s latency is consistently 1-5ms regardless of table size. An RDS PostgreSQL query that returns in 2ms at 1GB can return in 200ms at 1TB without careful indexing and tuning.

• “Costs less than RDS” is often false. DynamoDB on-demand pricing is $1.25 per million write request units and$0.25 per million read request units. A table with 10,000 writes/second sustained costs $1.25 * 10,000 * 86,400 / 1,000,000 * 30 =$32,400/month. An Aurora db.r6g.xlarge instance handling the same write throughput costs roughly $800/month. DynamoDB is cheaper for spiky, low-baseline workloads. It is dramatically more expensive for sustained high-throughput workloads unless you use provisioned capacity with reserved pricing, which erodes the “no capacity planning” benefit.
• “For everything” ignores the access pattern constraint. DynamoDB requires you to know your access patterns at design time. You design the partition key, sort key, and GSIs around specific queries. If the product team adds a new query pattern 6 months later (e.g., “find all orders by product ID across all customers”), you either already have a GSI for it or you add one (which consumes additional write capacity on every write to the table). In PostgreSQL, you add an index. In DynamoDB, you redesign your data model. I have been on a team where a “simple” new reporting requirement forced a complete DynamoDB table redesign and data migration because the existing single-table design could not support the new access pattern.
• Analytical queries are a non-starter. “Show me total revenue by region for the last 90 days” is a full table scan in DynamoDB. You export to S3 and query with Athena. In PostgreSQL, it is a 200ms query with a covering index. If your application needs even basic ad-hoc querying or reporting, DynamoDB alone is insufficient.
• Transactions are limited. DynamoDB transactions span up to 100 items in a single request and work only within a single table (or across tables in the same region). There is no equivalent to a multi-statement PostgreSQL transaction with savepoints, rollbacks, and isolation levels. For workflows like “transfer $100 from account A to account B while recording an audit log entry” where all three writes must succeed atomically, DynamoDB transactions work. For “insert an order, create 50 line items, update inventory for each product, and record 50 ledger entries atomically,” you exceed the 100-item limit.

​

Follow-up: When would you use DynamoDB single-table design versus multi-table design?

​

18. It is 2 AM. PagerDuty wakes you up. Your ECS Fargate service is returning 503 errors. CPU and memory look fine. Tasks are running. What is happening and how do you triage?

• “Check the application logs for errors.”
• “Restart the tasks.”
• “Scale up to more tasks.”

• Check the ALB target group health. If targets show unhealthy, the ALB is pulling tasks out of rotation. The health check might be failing even though CPU/memory are fine — the application could be returning 500 on the health check endpoint due to a downstream dependency failure (database unreachable, config service down).
• Check HealthyHostCount and UnHealthyHostCount CloudWatch metrics on the target group. If healthy count dropped to zero, the ALB has no targets to route to and returns 503 by default.

• HTTPCode_ELB_503_Count metric confirms the ALB is generating the 503s (as opposed to the application returning 503 via the ALB). A 503 from the ALB means: no healthy targets, all targets are deregistering, or the target group has no registered targets.
• RequestCount on the target group — are requests even reaching the targets? If request count is zero but the ALB is receiving traffic, the issue is target registration or health checks.
• Check if a deployment just happened. ECS rolling deployments deregister old tasks and register new ones. If the new task definition has a bug (crashes on startup, fails health check), ECS keeps trying to start new tasks while the old ones are draining. During the crossover, healthy target count can drop to zero.

• aws ecs describe-services — check runningCount vs desiredCount. If running is less than desired, ECS is trying to start tasks but they are failing. Check events on the service for messages like “task failed to start” or “ECS was unable to place a task.”
• Check stopped tasks: aws ecs list-tasks --desired-status STOPPED. Look at stoppedReason — common culprits: “CannotPullContainerError” (ECR access issue, usually a VPC endpoint or NAT Gateway problem), “ResourceNotFoundException” (Secrets Manager or Parameter Store secret was deleted), “OutOfMemoryError” (container killed by OOM despite CloudWatch showing memory below limit — this happens when the container’s hard memory limit is hit but the task-level memory is not).
• Check the ECS task definition — did someone push a bad container image to ECR? Did the entrypoint script change? A new image that crashes on startup will pass docker pull but fail immediately, and ECS will cycle through start-crash-restart.

• Security group on the tasks — did someone modify it? If the ALB’s security group cannot reach the task’s security group on the container port, health checks fail and the ALB returns 503.
• NACLs on the subnets — less common but devastating. A NACL change that blocks ephemeral ports (1024-65535) breaks ALB-to-task communication.
• NAT Gateway — if the tasks need to pull config from Secrets Manager or Parameter Store at startup, and the NAT Gateway is down or overloaded, tasks start but hang during initialization and fail health checks.

​

Follow-up: How do you distinguish between a deployment-caused outage and an infrastructure-caused outage in under 2 minutes?

​

19. You are designing an event-driven system with EventBridge. A colleague says you do not need to worry about schema evolution because “events are just JSON.” Why is this a ticking time bomb, and how do you defuse it?

• “JSON is flexible, consumers just ignore fields they do not know.”
• “We will version our events when we need to.”
• “EventBridge schema registry handles this automatically.”

• The phantom field break. Producer adds a field order.discount_percentage as a float. Consumer A expects it. Consumer B (written 6 months ago) does not know about it and is fine. Consumer C (legacy Python service) receives the event and passes it to a function that chokes because it deserializes the entire payload and feeds it to a pandas DataFrame that cannot handle the new column. Nothing in the event contract told Consumer C this would happen.
• The type change time bomb. Producer changes order.amount from integer cents (1599) to a string with currency ("$15.99"). Every consumer that parses amount as an integer breaks silently — no exception, just wrong calculations. I have seen this exact scenario cause a $47,000 billing discrepancy over 3 days before anyone noticed.
• The required-to-optional change. Producer decides customer.phone is optional and starts sending events without it. Consumer D (the SMS notification service) dereferences a null pointer and crashes. Its SQS consumer dies, messages pile up in the DLQ, and nobody notices until 10,000 customers complain they did not get order confirmations.

• Event schema registry with enforcement. EventBridge Schema Registry discovers schemas automatically from live events. But discovery is not enough — you need enforcement. Define schemas explicitly (JSON Schema or OpenAPI), version them, and validate events at the producer before publishing. The producer calls a schema validation library before putting the event on the bus. If validation fails, the event is rejected, not published. This catches breaking changes at the source.
• Schema versioning with semantic conventions. Every event type gets a version in its detail-type: OrderPlaced.v2. Consumers subscribe to the version they support. When the producer needs a breaking change, it publishes OrderPlaced.v3 alongside v2. Consumers migrate at their own pace. The producer deprecates v2 only after all consumers have migrated (tracked via EventBridge metrics on rule invocations — if the v2 rule has zero invocations for 30 days, it is safe to remove).
• Backward-compatible changes only by default. Adding new optional fields is always safe. Removing fields, renaming fields, or changing field types is a breaking change that requires a version bump. This is the same contract discipline as API versioning, and it needs to be enforced in code review — not left to developer goodwill.
• Consumer contract testing. Each consumer publishes a “contract test” that defines the minimum fields and types it requires from each event type. The CI pipeline for the producer runs all downstream consumer contract tests before deploying. If the producer’s schema change breaks any consumer contract, the pipeline fails. This is the Pact testing model applied to events.
• Dead letter queue per consumer. Every EventBridge rule that invokes Lambda or SQS should have a DLQ. When a consumer fails to process an event (possibly due to a schema change), the event goes to the DLQ rather than disappearing. Monitor DLQ depth — a sudden spike after a producer deployment is a strong signal that a schema change broke something.

​

Follow-up: How do you handle the EventBridge schema registry’s limitations in practice?

• It discovers schemas reactively (an event must flow through the bus before the schema appears). It does not enforce schemas — a producer can publish any JSON.
• Schema versions in the registry are discovered versions, not semantic versions. Adding a new field creates a “new version” even if it is backward compatible.
• There is no built-in mechanism to reject events that do not match a schema.

​

20. Your company runs a SaaS product with 200 tenants on a shared infrastructure. One tenant starts sending 50x their normal traffic. Within minutes, all other tenants experience degraded performance. Diagnose the failure and design the fix.

• “Rate limit the noisy tenant.”
• “Give each tenant their own infrastructure.”
• “Scale up to handle the increased load.”

• API Gateway: If all tenants hit the same API Gateway, the noisy tenant’s 50x spike may be consuming the account-level API Gateway throttle limit (10,000 requests/second default). All tenants share this limit. When exceeded, every tenant gets 429 errors.
• Lambda: If backend Lambda functions have no reserved concurrency, the noisy tenant’s requests consume the account’s 1,000 concurrent execution limit. Other tenants’ requests are throttled with 429.
• DynamoDB: If tenants share a DynamoDB table, the noisy tenant’s requests may hot-partition the table (especially if tenant ID is the partition key and one tenant dominates throughput). Adaptive capacity helps but cannot fully compensate for a 50x spike.
• SQS: If tenants share an SQS queue, the noisy tenant’s messages flood the queue. Consumer Lambda processes messages in order of arrival, starving other tenants’ messages.

• API Gateway per-tenant throttling. Use API Gateway usage plans with API keys. Each tenant gets an API key with a throttle limit (e.g., 100 requests/second) and a burst limit. When Tenant X exceeds their limit, only they get 429 errors. Other tenants are unaffected. This is the first line of defense and the cheapest to implement.
• Lambda concurrency isolation. Create a Lambda function alias per tenant tier (free, standard, enterprise). Set reserved concurrency: free tier functions get 50 concurrent executions, standard gets 200, enterprise gets 500. The noisy tenant’s tier exhausts its own reserved pool and is throttled without impacting other tiers. For critical-path functions, I would also consider per-tenant Lambda functions for the largest tenants (top 5-10 by revenue), each with their own reserved concurrency.
• DynamoDB tenant-aware design. Partition key should include tenant ID (e.g., PK = TENANT#123#ORDER#456). This naturally distributes each tenant’s data across different partitions. But if a single tenant’s throughput exceeds a partition’s capacity, add write sharding within the tenant key. Also: use DynamoDB on-demand mode so individual tenant spikes are absorbed without provisioned capacity limits, and set per-tenant item-level monitoring via Contributor Insights.
• SQS queue-per-tenant or priority queuing. For the noisy-neighbor SQS problem, the cleanest solution is a queue-per-tenant for the top 20 tenants (by volume) and a shared queue for the remaining 180. Each per-tenant queue has its own Lambda consumer with independent reserved concurrency. The shared queue has a Lambda consumer that processes small tenants. If a top-20 tenant spikes, only their queue backs up. Alternative: a single queue with tenant-aware consumer logic that implements fair scheduling (round-robin across tenant IDs in the batch).
• Rate limiting at the application layer. Beyond API Gateway throttling, implement a token bucket rate limiter in the application code (backed by Redis INCR with TTL). Each tenant has a bucket with a per-second and per-minute limit based on their plan tier. This catches bursts that exceed the API Gateway throttle (which operates at the API key level, not the business logic level).

​

Follow-up: How do you detect the noisy tenant automatically before other tenants are impacted?

• Per-tenant request counting in real-time. A Lambda@Edge function (or API Gateway access logging to Kinesis) that increments a Redis sorted set keyed by tenant ID with TTL-based windows. A background Lambda samples the sorted set every 30 seconds and fires a CloudWatch custom metric RequestsPerTenant. A CloudWatch anomaly detection alarm triggers when any tenant exceeds 3 standard deviations from their 7-day baseline.
• API Gateway access logs to Athena. API Gateway access logs include the API key (tenant) and can be queried in near real-time via Kinesis Firehose to S3 and Athena. A scheduled query every 5 minutes identifies tenants exceeding their normal traffic by 5x+.
• Automatic throttling. When the anomaly detection alarm fires, a Lambda function automatically lowers the offending tenant’s API Gateway usage plan throttle to their normal baseline + 20% buffer. This contains the blast radius within minutes, without human intervention. A Slack notification tells the on-call engineer what happened so they can investigate whether the traffic is legitimate or malicious.

​

21. You are reviewing a pull request that adds "Action": "s3:*", "Resource": "*" to a Lambda execution role. The developer says “I will tighten it later.” How do you handle this technically and interpersonally?

• “Block the PR and tell them to fix it.”
• “It is fine for now, we can tighten it later.”
• “Add a TODO comment and merge.”

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket"
  ],
  "Resource": [
    "arn:aws:s3:::specific-bucket-name",
    "arn:aws:s3:::specific-bucket-name/*"
  ]
}

• IAM Access Analyzer as a CI check. AWS IAM Access Analyzer has a policy validation API. Add a CI step that runs aws accessanalyzer validate-policy on every IAM policy in the CDK/Terraform code. It flags overly permissive actions and wildcard resources as findings. The PR cannot merge with ERROR-level findings.
• SCP as a safety net. An SCP that denies s3:PutBucketPolicy, s3:DeleteBucket, and s3:PutBucketPublicAccessBlock for Lambda execution roles (identified by a naming convention or tag condition). This means even if a wildcard policy slips through, the most dangerous S3 actions are blocked at the organization level.
• CDK/Terraform policy templates. Create reusable IAM policy modules for common patterns (S3 read-only, S3 read-write to specific bucket, DynamoDB CRUD on specific table). Developers import the module instead of writing raw JSON. This is faster than writing a custom policy and correct by default.

​

Follow-up: The developer pushes back: “IAM Access Analyzer is too strict, it flags everything. It slows us down.” How do you respond?

​

22. A service processes 2 million SQS messages per day. Occasionally, the same message is processed twice, causing duplicate charges to customers. The team says “SQS is at-least-once, duplicates are expected.” Is that an acceptable answer?

• “Switch to SQS FIFO for exactly-once delivery.”
• “SQS duplicates are rare, the business should accept them.”
• “Add a database check before processing each message.”

• Idempotency key. Every message carries a unique idempotency key (the SQS MessageId, a hash of the message body, or a business-level identifier like order_id + charge_type). Before processing, the consumer checks if this key has been processed before.
• The atomic check-and-process pattern. This is where most implementations go wrong. A naive approach — “read from database, check if key exists, if not process, then write key” — has a race condition. Two concurrent Lambda instances receive the same duplicate message, both check the database simultaneously, both see “not processed,” both process, both write the key. You still get a duplicate.

# DynamoDB: Conditional put that fails if key already exists
try:
    idempotency_table.put_item(
        Item={
            'idempotency_key': message_id,
            'processed_at': timestamp,
            'result': None,  # Will be updated after processing
            'ttl': int(time.time()) + 86400 * 7  # Expire after 7 days
        },
        ConditionExpression='attribute_not_exists(idempotency_key)'
    )
except ConditionalCheckFailedException:
    # Already processed -- skip
    return {'statusCode': 200, 'body': 'Duplicate, skipped'}

# Process the message (charge the customer)
result = process_charge(message)

# Update the idempotency record with the result
idempotency_table.update_item(
    Key={'idempotency_key': message_id},
    UpdateExpression='SET result = :r',
    ExpressionAttributeValues={':r': json.dumps(result)}
)

• TTL for cleanup. Idempotency keys should not live forever. Set a DynamoDB TTL to expire keys after 7 days (or whatever your SQS message retention is). This keeps the table from growing unbounded.
• What about SQS FIFO? SQS FIFO provides exactly-once delivery within a 5-minute deduplication window. But it does not provide exactly-once processing. If the consumer crashes after processing but before deleting the message, the message reappears after the visibility timeout. You still need the idempotency pattern. FIFO reduces the probability of duplicates but does not eliminate it. For 2 million messages/day (23 messages/second), SQS FIFO works within its throughput limits. But I would still implement idempotency because the cost is negligible and the protection is absolute.
• The broader pattern. Idempotency is not just for SQS. API endpoints should be idempotent (Stripe’s Idempotency-Key header is the gold standard). Event processors should be idempotent. Anything that can be retried should be idempotent. I treat idempotency the same way I treat input validation — it is not optional, it is a fundamental requirement for any operation that has side effects.

​

Follow-up: What happens if the Lambda crashes between the idempotency write and the actual processing? The key is recorded but the charge never happened.

​

Follow-up: How do you size the DynamoDB table for the idempotency store at 2 million messages/day?