Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Part XII — Testing and Quality Engineering

​

Real-World Story: How Google Tests at Scale

​

Real-World Story: The Knight Capital Disaster — $440 Million in 45 Minutes

​

Chapter 19: Testing Strategy

// Stub: always returns the same user regardless of input
const userServiceStub = {
  getUser: (id) => ({ id, name: "Test User", email: "test@example.com" })
}
// Use when: you need a dependency to return specific data for your test scenario

// Mock: verify that the email service was called with the right arguments
const emailServiceMock = mock(EmailService)
orderService.placeOrder(order)
verify(emailServiceMock).sendConfirmation("user@example.com", order.id)
// Use when: the behavior you are testing IS the interaction (e.g., "placing an order sends an email")

// Fake: an in-memory database that actually stores and retrieves data
class FakeUserRepository {
  users = new Map()
  save(user) { this.users.set(user.id, user) }
  findById(id) { return this.users.get(id) || null }
}
// Use when: you need a real-ish dependency but cannot use the actual one (database, file system)

// Spy: track calls to the real logger without replacing it
const loggerSpy = spy(realLogger)
orderService.placeOrder(order)
assert(loggerSpy.info).wasCalledWith("Order placed", { orderId: order.id })
// Use when: you want the real behavior but also want to verify what happened

​

19.1 The Test Pyramid

​

Testing Strategy Decision Matrix

​

19.2 Unit Testing

​

19.3 Integration Testing

​

The Testcontainers Pattern in Practice

// Java + JUnit 5 + Testcontainers example
@Testcontainers
class OrderRepositoryIntegrationTest {

    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:16")
        .withDatabaseName("testdb")
        .withUsername("test")
        .withPassword("test");

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgres::getJdbcUrl);
        registry.add("spring.datasource.username", postgres::getUsername);
        registry.add("spring.datasource.password", postgres::getPassword);
    }

    @Test
    void shouldPersistOrderWithLineItems() {
        // Arrange: create an order with 2 line items
        Order order = new Order("customer-123", List.of(
            new LineItem("SKU-A", 2, Money.of(10.00)),
            new LineItem("SKU-B", 1, Money.of(25.00))
        ));

        // Act: save and retrieve
        orderRepository.save(order);
        Order loaded = orderRepository.findById(order.getId());

        // Assert: full round-trip works with real Postgres
        assertThat(loaded.getLineItems()).hasSize(2);
        assertThat(loaded.getTotal()).isEqualTo(Money.of(45.00));
    }
}

// Node.js + Jest + Testcontainers example
const { PostgreSqlContainer } = require("@testcontainers/postgresql");

let container;
let db;

beforeAll(async () => {
  container = await new PostgreSqlContainer("postgres:16").start();
  db = createPool({ connectionString: container.getConnectionUri() });
  await runMigrations(db); // apply your real schema
}, 60_000);

afterAll(async () => {
  await db.end();
  await container.stop();
});

test("order persists with line items", async () => {
  const orderId = await createOrder(db, {
    customerId: "cust-123",
    items: [{ sku: "SKU-A", qty: 2, price: 10.0 }]
  });

  const order = await getOrderById(db, orderId);
  expect(order.items).toHaveLength(1);
  expect(order.total).toBe(20.0);
});

​

19.4 End-to-End Testing

​

19.5 Contract Testing

​

Pact Contract Testing — A Practical Example

1. Consumer side — The consumer writes a test that declares: “When I call GET /users/123, I expect a response with { id, name, email }.”
2. Pact generates a contract — The test produces a JSON “pact file” encoding this expectation.
3. Provider side — The provider runs the pact file against its real API. If the response matches the contract, the test passes. If a field was renamed or removed, it fails.
4. Pact Broker — A central server stores contracts and verification results so teams can see compatibility at a glance.

// Consumer-side Pact test (Node.js)
const { PactV3 } = require("@pact-foundation/pact");

const provider = new PactV3({
  consumer: "OrderService",
  provider: "UserService",
});

describe("User API contract", () => {
  it("returns user details", async () => {
    provider
      .given("user 123 exists")
      .uponReceiving("a request for user 123")
      .withRequest({ method: "GET", path: "/users/123" })
      .willRespondWith({
        status: 200,
        body: {
          id: 123,
          name: "Jane Doe",
          email: "jane@example.com",
        },
      });

    await provider.executeTest(async (mockServer) => {
      const user = await fetchUser(mockServer.url, 123);
      expect(user.name).toBe("Jane Doe");
    });
  });
});

​

19.6 Performance Testing

​

19.7 Flaky Tests

​

Systematic Approach to Flaky Tests

1. Detect — Track test results over time. Flag any test that has both passed and failed on the same commit within a window (e.g., 7 days). Tools like BuildPulse, Datadog Test Visibility, or a simple CI report can surface these.
2. Quarantine — Move known-flaky tests to a separate suite that runs but does not block the pipeline. This restores trust in the main suite immediately while you investigate.
3. Classify the root cause. The most common culprits:
   • Shared mutable state — Tests depend on data left behind by a previous test. Fix: isolate test data, use transactions that roll back, or reset state in beforeEach.
   • Timing and async races — Tests assume something completes within an arbitrary time. Fix: use explicit waits (waitFor, polling assertions) instead of sleep.
   • Non-deterministic ordering — Tests pass when run in one order but fail in another. Fix: run tests in random order during CI to catch these early (jest --randomize, pytest -p randomly).
   • External dependency — Test hits a real network service that is intermittently slow or down. Fix: stub external dependencies at the network boundary.
   • Date/time sensitivity — Tests break at midnight, on DST transitions, or on January 1st. Fix: inject a clock and freeze time in tests.
4. Fix and verify — After fixing, run the test 50-100 times in a loop to confirm stability before removing it from quarantine.
5. Prevent recurrence — Add deterministic ordering to CI, enforce test isolation in code review, and track flaky-test metrics on your engineering dashboard.

​

Flaky System vs Flaky Test

1. Does the test fail across different environments (local, CI, different OS)? If it only fails in CI, it is more likely a flaky test (environmental dependency). If it fails everywhere at roughly the same rate, it is more likely a flaky system.
2. Does the test failure message describe a wrong result or a timeout? Wrong results (unexpected value, assertion mismatch) often indicate a real race condition. Timeouts often indicate an environmental issue.
3. Does the test have a sleep, hardcoded port, shared global state, or direct Date.now() call? If yes, the test is the likely culprit.
4. Does the failure rate correlate with system load? If the test fails more often when CI machines are under heavy load, you may have a real concurrency issue that only surfaces under contention.

​

Release Confidence — The Metric That Matters

• Deployment frequency: How often does the team deploy to production? High confidence enables daily or continuous deployment. Low confidence forces weekly or bi-weekly batch releases with manual QA gates.
• Defect escape rate: How many bugs reach production per deployment? Track this as a rolling average. Healthy teams see <0.5 escaped defects per sprint.
• Mean time to recover (MTTR): When a defect escapes, how fast do you detect and fix it? This measures whether your observability and rollback mechanisms work.
• Change failure rate: What percentage of deployments cause a degradation or rollback? DORA research shows elite teams have a change failure rate of 0-15%.

​

19.8 Testing Microservices

​

Contract Testing Between Services

• Request/response shapes for synchronous HTTP/gRPC calls
• Event schemas for asynchronous messages (Kafka, SQS, EventBridge)
• Error response formats (status codes, error body structure)
• Authentication and authorization header requirements

• Business logic (that belongs in the provider’s unit tests)
• Performance characteristics (that belongs in load tests)
• Every possible edge case (focus on the interactions your consumer actually uses)

​

Integration Test Environments

// Hybrid approach: real DB + stubbed downstream services
const { PostgreSqlContainer } = require("@testcontainers/postgresql");
const { setupWireMock } = require("./test-helpers/wiremock");

let db, userServiceStub;

beforeAll(async () => {
  // Real database — tests actual SQL, migrations, constraints
  db = await new PostgreSqlContainer("postgres:16").start();
  await runMigrations(db.getConnectionUri());

  // Stubbed downstream service — responses verified by Pact contracts
  userServiceStub = await setupWireMock({
    port: 9001,
    stubs: [
      { request: { method: "GET", url: "/users/123" },
        response: { status: 200, body: { id: 123, name: "Jane Doe" } } }
    ]
  });
});

test("order creation calls user service and writes to DB", async () => {
  const order = await createOrder(db, { userId: 123, items: [{ sku: "A", qty: 1 }] });
  expect(order.status).toBe("confirmed");
  expect(order.userName).toBe("Jane Doe");
  // Verify the order was persisted in the real database
  const persisted = await getOrderById(db, order.id);
  expect(persisted).toBeDefined();
});

​

Service Virtualization Deep Dive

• Third-party APIs (payment gateways, identity providers) where you cannot control availability or test data
• Services owned by other teams that may not have a stable test environment
• Legacy systems that are difficult to provision locally

• WireMock — HTTP API simulation. Record real traffic, then replay it. Supports request matching, response templating, and fault injection (delays, connection resets). Available as a Java library, standalone JAR, or Docker container.
• Mountebank — Protocol-agnostic service virtualization (HTTP, TCP, SMTP). More flexible than WireMock for non-HTTP protocols.
• Hoverfly — Lightweight service virtualization with capture-and-replay workflow. Good for Go and Java ecosystems.
• LocalStack — Simulates AWS services locally (S3, SQS, DynamoDB, Lambda, etc.). Essential for testing cloud-native services without incurring AWS costs.

​

19.9 Feature Flag Testing

​

Testing Both Flag States

// Feature flag testing pattern — test BOTH states explicitly
describe("checkout flow", () => {
  describe("with new payment UI disabled (flag OFF — current production state)", () => {
    beforeEach(() => {
      featureFlags.set("new-payment-ui", false);
    });

    test("renders legacy payment form", () => {
      render(<CheckoutPage />);
      expect(screen.getByTestId("legacy-payment-form")).toBeInTheDocument();
      expect(screen.queryByTestId("new-payment-form")).not.toBeInTheDocument();
    });

    test("processes payment through legacy gateway", async () => {
      const result = await processPayment({ amount: 50.00, method: "card" });
      expect(result.gateway).toBe("legacy-stripe-v1");
    });
  });

  describe("with new payment UI enabled (flag ON — gradual rollout)", () => {
    beforeEach(() => {
      featureFlags.set("new-payment-ui", true);
    });

    test("renders new payment form", () => {
      render(<CheckoutPage />);
      expect(screen.getByTestId("new-payment-form")).toBeInTheDocument();
      expect(screen.queryByTestId("legacy-payment-form")).not.toBeInTheDocument();
    });

    test("processes payment through new gateway", async () => {
      const result = await processPayment({ amount: 50.00, method: "card" });
      expect(result.gateway).toBe("stripe-v2");
    });
  });
});

​

The Knight Capital Lesson Applied

​

Flag Cleanup as Part of Test Maintenance

1. Create — Add the flag with a clear purpose, owner, and expected removal date in your flag management tool (LaunchDarkly, Unleash, Flagsmith, or even a config file).
2. Test — Write tests for both states. Tag the tests with the flag name so they can be found later.
3. Roll out — Gradually enable for increasing percentages of users. Monitor error rates and key metrics at each stage.
4. Bake — Once the flag is 100% on and has been stable for a defined period (e.g., two weeks), mark it for cleanup.
5. Remove — Delete the flag, remove the conditional code paths, delete the tests for the “off” state, and simplify the codebase.

• Flag expiry dates. Some flag systems (LaunchDarkly, custom implementations) support setting an expiry date. After that date, the flag appears in a “stale flags” report.
• Lint rules. Write a custom ESLint/Pylint rule that detects flag checks older than a threshold (based on git blame dates).
• Quarterly flag audits. Review all active flags. For each one, ask: “Is this still being rolled out, or is it permanent?” If permanent, remove the flag and the conditional logic.

​

Interview Questions — Testing Strategy

“You inherit a CI pipeline that runs 2,200 tests in 38 minutes. Developers routinely push to main without waiting for CI. Last week a pricing bug shipped to production despite 88% test coverage. You have one sprint to improve the situation. Write a concrete plan with specific actions for Week 1 and Week 2, and explain what you would measure to know the plan is working.”

​

Hermetic CI and Test Data Management

1. Infrastructure per run. Use Testcontainers, ephemeral Kubernetes namespaces, or Docker Compose to spin up all dependencies (database, cache, message broker) fresh for each CI run. Destroy them afterward. No shared infrastructure.
2. Deterministic test data. Test data is created by the test itself (factory functions, fixtures, seeders), not loaded from a shared source. Every test run starts from the same known state.
3. No network calls to external services. All external APIs are stubbed or contract-tested. If the test relies on a third-party API being available, it is not hermetic.
4. Pinned dependencies. Lock files (package-lock.json, go.sum, Cargo.lock) are committed and CI installs from the lock file, not from latest. A dependency update that breaks your build should show up in a deliberate update PR, not in a random CI run.
5. Time and randomness control. Tests that use Date.now() or Math.random() inject a controlled clock and a seeded random generator. Same inputs, same outputs, every time.

​

Test Data Management at Scale

​

Hermetic CI at Scale — The Hard Problems Nobody Mentions

​

Test Data Lifecycle Management

​

Deploy Verification and Shadow Reads

1. Health check. Every service exposes a /health endpoint that verifies it can reach its critical dependencies (database, cache, downstream services). The deployment pipeline checks this endpoint on every new instance before routing traffic.
2. Smoke tests. A small set of critical-path requests executed against the newly deployed service. Not the full test suite — just the 5-10 requests that cover the most important user journeys. For an e-commerce service: create a cart, add an item, fetch pricing. If any fail, auto-rollback.
3. Version verification. The deployed service reports its version (git SHA, build number) via an endpoint or log line. The deployment pipeline verifies that the reported version matches the expected deployment. This is the Knight Capital check — make sure every instance is running the code you think it is running.

• During database migration: read from the old database (serves the user), also read from the new database (compare results). When the new database returns identical results for 99.9%+ of reads over a week, cut over with confidence.
• During service extraction from a monolith: the monolith handles the request, the new service also handles it in shadow mode, compare outputs.
• During algorithm changes: the old recommendation algorithm serves users, the new algorithm also runs, compare quality metrics without affecting the user experience.

// Shadow read pattern — new system runs in background, results compared
async function getUser(userId) {
  // Primary path — serves the user
  const result = await oldUserService.getUser(userId);

  // Shadow path — runs async, never affects response
  shadowRead(async () => {
    const shadowResult = await newUserService.getUser(userId);
    if (!deepEqual(result, shadowResult)) {
      logger.warn("Shadow read mismatch", {
        userId,
        field: findDiff(result, shadowResult),
        primary: result,
        shadow: shadowResult,
      });
      metrics.increment("shadow_read.mismatch", { service: "user" });
    }
  });

  return result; // always return the primary result
}

​

Advanced Deploy Verification — Beyond Health Checks

​

Curated Resources — Testing

​

Testing Anti-Patterns to Avoid

​

Anti-Pattern 1: Testing Implementation Details

// BAD: testing implementation details
test("placeOrder calls validateInventory then processPayment", () => {
  const spy1 = jest.spyOn(orderService, "validateInventory");
  const spy2 = jest.spyOn(orderService, "processPayment");
  orderService.placeOrder(order);
  expect(spy1).toHaveBeenCalledBefore(spy2); // who cares about the order?
});

// GOOD: testing behavior
test("placeOrder creates an order and charges the customer", () => {
  const result = orderService.placeOrder(order);
  expect(result.status).toBe("confirmed");
  expect(result.chargedAmount).toBe(49.99);
});

​

Anti-Pattern 2: Testing Private Methods Directly

​

Anti-Pattern 3: 100% Coverage as a Goal

// Written solely to hit coverage — asserts nothing useful
test("constructor creates instance", () => {
  const service = new OrderService();
  expect(service).toBeDefined(); // congratulations, you tested the "new" keyword
});

​

Anti-Pattern 4: Slow and Flaky Tests That Get Ignored

​

Anti-Pattern 5: Test Suites Nobody Runs Locally

​

Part XIII — Logging, Audit Logs, and Data Trails

​

Real-World Story: GitLab’s Radical Transparency on Incidents and Logging

​

Chapter 20: Audit and Compliance Logging

​

20.1 Operational Logging vs Audit Logging

​

Structured Logging vs Unstructured Logging

2025-03-15 14:23:01 INFO OrderService - User user-456 placed order ord-789 for $120.50 from IP 192.168.1.42

{
  "timestamp": "2025-03-15T14:23:01.123Z",
  "level": "info",
  "service": "order-service",
  "message": "Order placed",
  "traceId": "abc-123-def-456",
  "spanId": "span-789",
  "userId": "user-456",
  "orderId": "ord-789",
  "amount": 120.50,
  "currency": "USD",
  "ip": "192.168.1.42",
  "duration_ms": 234
}

• Node.js: pino (fast, JSON-native), winston (flexible, multiple transports)
• Python: structlog, python-json-logger
• Java: Logback + Logstash encoder, Log4j2 JSON layout
• Go: zerolog, zap (both produce JSON by default)

​

Recommended Structured Log Fields

• request_id — unique ID for the HTTP request (distinct from trace_id in non-distributed contexts)
• span_id — the span within a trace (for distributed tracing)
• http_method, http_path, http_status — for request/response logging
• ip — client IP address (for security and audit)
• environment — "production", "staging", "development"
• version — application version or git SHA for identifying which build emitted the log

{
  "timestamp": "2025-03-15T14:23:01.123Z",
  "level": "error",
  "service": "payment-service",
  "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "span_id": "span-001",
  "request_id": "req-xyz-789",
  "user_id": "user-456",
  "action": "payment.charge",
  "duration_ms": 3042,
  "http_method": "POST",
  "http_path": "/api/v1/payments",
  "http_status": 502,
  "error": {
    "code": "GATEWAY_TIMEOUT",
    "message": "Stripe API did not respond within 3000ms",
    "retry_count": 2
  },
  "order_id": "ord-789",
  "amount": 120.50,
  "currency": "USD",
  "ip": "203.0.113.42",
  "environment": "production",
  "version": "v2.14.3"
}

​

Log Levels Guide — When to Use Each Level

{ "level": "trace", "message": "Entering calculateDiscount", "userId": "user-456",
  "cartTotal": 149.99, "couponCode": "SAVE20", "memberTier": "gold" }
{ "level": "trace", "message": "Coupon SAVE20 validated, discount: 20%",
  "discountAmount": 30.00 }

{ "level": "debug", "message": "Cache miss for product catalog",
  "cacheKey": "catalog:electronics:page-3", "ttl_seconds": 300,
  "fallback": "database-query" }
{ "level": "debug", "message": "Retry attempt for payment gateway",
  "attempt": 2, "maxRetries": 3, "backoffMs": 1500,
  "reason": "ETIMEDOUT" }

{ "level": "info", "message": "Order placed", "orderId": "ord-789",
  "userId": "user-456", "amount": 120.50, "items": 3,
  "paymentMethod": "card", "duration_ms": 234 }
{ "level": "info", "message": "User registered", "userId": "user-999",
  "registrationMethod": "google-oauth", "referralSource": "campaign-spring25" }
{ "level": "info", "message": "Scheduled job completed", "job": "invoice-generation",
  "processedCount": 1847, "duration_ms": 45200 }

{ "level": "warn", "message": "Database connection pool near capacity",
  "activeConnections": 47, "maxConnections": 50,
  "action": "Consider increasing pool size or investigating slow queries" }
{ "level": "warn", "message": "Response time exceeded SLO threshold",
  "endpoint": "/api/v1/search", "duration_ms": 4200,
  "sloThreshold_ms": 3000, "percentile": "p99" }
{ "level": "warn", "message": "Deprecated API version called",
  "apiVersion": "v1", "sunsetDate": "2025-06-01",
  "clientId": "partner-acme", "migration_guide": "/docs/api/v2-migration" }

{ "level": "error", "message": "Payment processing failed",
  "orderId": "ord-789", "userId": "user-456", "amount": 120.50,
  "gateway": "stripe", "errorCode": "card_declined",
  "errorMessage": "Insufficient funds",
  "userImpact": "Order not placed, user shown retry option",
  "trace_id": "abc-123" }
{ "level": "error", "message": "Failed to send welcome email",
  "userId": "user-999", "emailProvider": "sendgrid",
  "errorCode": "ETIMEDOUT", "retryScheduled": true,
  "retryAt": "2025-03-15T14:25:00Z" }

{ "level": "fatal", "message": "Database connection failed on startup",
  "host": "primary-db.internal", "port": 5432,
  "error": "Connection refused",
  "impact": "Service cannot start, no requests will be served",
  "action": "Check database availability and network connectivity" }
{ "level": "fatal", "message": "Out of memory — process terminating",
  "heapUsedMB": 3892, "heapMaxMB": 4096,
  "lastRequest": "/api/v1/reports/generate-annual" }

Did the operation succeed?
├── YES → Was anything unusual?
│         ├── NO → INFO (record the business event)
│         └── YES → WARN (record what was unusual and what action to take)
└── NO → Is the whole system affected?
          ├── NO → ERROR (this operation failed, log context for diagnosis)
          └── YES → FATAL (the system cannot function, page immediately)

​

20.2 Audit Trail Design

​

Compliance Requirements for Audit Logs

• Append-only tables with REVOKE DELETE, UPDATE on the audit schema
• Write-once storage (AWS S3 Object Lock, Azure Immutable Blob Storage, GCP Bucket Lock)
• Blockchain-anchored hashes for tamper-evidence in high-assurance environments

• All authentication events (login, logout, failed login, password reset)
• All authorization failures (access denied)
• All data mutations on sensitive resources (create, update, delete)
• All admin/elevated-privilege actions
• All changes to access control (role changes, permission grants)
• All data exports and bulk downloads
• System configuration changes
• Direct database access by operators

​

Interview Questions — Audit Logging

​

20.3 Audit Logging Under Privacy Constraints

{
  "timestamp": "2026-04-10T14:23:01.123Z",
  "actor_id": "usr_a8f3c2b1",
  "actor_type": "employee",
  "action": "patient_record.view",
  "target_id": "patient_rec_xyz789",
  "target_fields_accessed": ["diagnosis", "medication_list", "lab_results"],
  "source": "clinical_dashboard",
  "ip_hash": "sha256:a1b2c3d4...",
  "session_id": "sess_m9n8o7",
  "result": "allowed",
  "access_justification": "treating_physician"
}

​

20.4 Data Trails and Change History

​

Curated Resources — Logging

​

Part XIV — Versioning and Change Management

​

Real-World Story: How Spotify Handles Schema Versioning Across Hundreds of Microservices

​

Chapter 21: Versioning

​

21.1 API Versioning

​

Interview Questions — API Versioning

​

21.2 Database Schema Versioning

​

The Expand-Contract Pattern for Schema Migrations — Detailed Walkthrough

Phase          │ Database State           │ Application Behavior       │ Risk Level
───────────────┼──────────────────────────┼────────────────────────────┼───────────
               │                          │                            │
Day 1          │ [username] exists         │ Reads/writes username      │ NONE
EXPAND         │ [username] + [display_    │ Reads/writes username      │ (safe to
               │  name] both exist         │                            │  roll back)
               │                          │                            │
Day 2          │ Both columns exist        │ Writes to BOTH columns     │ LOW
DUAL-WRITE     │ Sync trigger keeps them  │ Reads from username        │ (either
               │ consistent               │                            │  column is
               │                          │                            │  valid)
               │                          │                            │
Day 3          │ Both columns exist        │ Writes to BOTH columns     │ LOW
BACKFILL       │ Batch copy old → new     │ Reads from username        │ (data
               │ for existing rows         │                            │  converging)
               │                          │                            │
Day 4-5        │ Both columns exist,       │ Writes to BOTH columns     │ LOW
SWITCH READS   │ all data synced          │ Reads from display_name    │ (new column
               │                          │                            │  is source
               │                          │                            │  of truth)
               │                          │                            │
Week 2-4       │ Both columns exist        │ Writes ONLY display_name  │ MEDIUM
MIGRATE        │                          │ All consumers updated      │ (verify no
CONSUMERS      │                          │                            │  stragglers)
               │                          │                            │
Week 5         │ Both columns exist        │ Monitor old column for     │ LOW
VERIFY         │                          │ any remaining access       │ (final
               │                          │                            │  safety
               │                          │                            │  check)
               │                          │                            │
Week 6         │ [display_name] only      │ Reads/writes display_name  │ NONE
CONTRACT       │ Old column dropped       │                            │ (migration
               │                          │                            │  complete)

-- Migration 001: add new column (non-breaking)
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

# Application code writes to both columns
user.username = new_value
user.display_name = new_value  # write to new column too
db.save(user)

-- Migration 002: backfill (run as a batch job, not in a transaction lock)
UPDATE users SET display_name = username WHERE display_name IS NULL;
-- For large tables, do this in batches of 1000-10000 rows to avoid locking

# Application now reads from display_name
name = user.display_name  # new column is the source of truth

user.display_name = new_value
# no longer writing to user.username

-- Migration 003: remove old column (only after all app instances are on Step 5)
ALTER TABLE users DROP COLUMN username;

​

21.3 Application Versioning

​

21.4 Event Schema Versioning

​

21.5 Ownership of Schema and Event Evolution

1. Schema registry as the source of truth. Every event schema is registered, versioned, and validated. The registry enforces compatibility rules automatically.
2. Schema changelog alongside code changes. Every PR that modifies an event schema includes a human-readable changelog entry: what changed, why, and which consumers are affected.
3. Deprecation-before-removal policy. Fields are never removed in the same release they are deprecated. Deprecate (stop populating, mark in documentation), wait for a full deprecation cycle (e.g., 3 months), verify zero consumers rely on the field, then remove.
4. Consumer impact analysis in CI. Before a schema change is merged, CI checks all registered consumers and reports which ones would be affected. This is the automated version of “did you tell everyone?”

​

The Schema Ownership Decision Matrix

1. Co-locate documentation with the schema definition. If the schema is defined in Avro/Protobuf, documentation comments live in the schema file itself. If it is defined in code, use JSDoc/Javadoc annotations. Documentation that requires visiting a separate wiki page has a half-life of 3 months before it is stale.
2. Generate a searchable schema catalog from the source of truth. Tools: Backstage entity catalog, Confluent Schema Registry’s REST API + a simple frontend, or a generated static site from your schema files. Engineers should be able to search “what events does the order service produce?” and get an answer in seconds.
3. Include example payloads in the documentation. A schema definition tells you the structure. An example payload tells you what it looks like. For every event version, include a representative JSON example with realistic (not placeholder) data.

​

21.6 Rollback-Safe Migrations

• Adding a nullable column (old app ignores it, new app uses it)
• Adding an index (old app does not care about indexes)
• Adding a new table (old app does not query it)
• Expanding a column width (VARCHAR(50) to VARCHAR(255) — old app writes shorter strings, still valid)

• Dropping a column (old app will crash trying to read it)
• Renaming a column (old app references the old name)
• Adding a NOT NULL constraint without a default (old app inserts without the column)
• Changing a column type (old app writes the old type)

NOT rollback-safe: DROP COLUMN username
                   ↓
Rollback-safe:     Step 1: Deploy app that stops reading/writing username
                   Step 2: Wait for all old instances to drain (rolling deploy)
                   Step 3: DROP COLUMN username
                   Step 4: If step 3 causes issues, the app already doesn't use the column

​

The Expand-Contract Migration Playbook — Step by Step

• Add the new column/table/field alongside the old one.
• Deploy application code that writes to both old and new (dual-write).
• Reads continue from the old column. The new column is invisible to users.
• Rollback safety: remove the dual-write code, the old column still works.

• Backfill the new column from the old column’s data. Run this as a background job, not as part of the migration script. For tables with millions of rows, batch the backfill (1000 rows at a time with WHERE new_col IS NULL LIMIT 1000) to avoid locking.
• Verify: count rows where old and new columns disagree. This should be zero after backfill.
• Rollback safety: the old column is still the source of truth.

• Deploy application code that reads from the new column. Keep writing to both.
• Monitor: compare old and new values for a sample of reads. Any divergence indicates a backfill gap or a race condition in the dual-write.
• Rollback safety: revert to reading from the old column.

• Deploy application code that stops writing to the old column.
• After all old-version instances are drained (verify with SELECT COUNT(*) WHERE old_col != new_col), drop the old column.
• Rollback safety: this is the only non-rollback-safe step. Only execute when you are confident the new column is correct.

• Dual-write race conditions. If two concurrent requests update the same row, one might write to old before new and the other might write to new before old, leaving the columns inconsistent. Fix: use a database trigger or a single UPDATE statement that sets both columns atomically.
• Backfill on large tables takes days. A table with 500M rows takes time to backfill. During that time, new writes are dual-written but old rows are not yet backfilled. Your application code in Phase 3 must handle reading NULL from the new column for unbackfilled rows (fall back to the old column).
• ORM cache invalidation. If your ORM caches the table schema (ActiveRecord, Django ORM), adding a column requires a process restart or cache refresh. Without this, the ORM does not know the new column exists and will not include it in INSERT statements. Test this in staging before production.
• Replication lag during cutover. If you switch reads to the new column and a read replica is 5 seconds behind, the replica might not have the latest dual-writes. For read-heavy services using replicas, verify replication lag is within tolerance before cutover.

​

21.7 Tolerating Partial Adoption and Long-Tail Compatibility

• Internal services: 15 consuming teams, 4 will migrate in week 1, 6 in month 1, 3 will need reminders, 2 will need help.
• Public API: 2,000 integrations, 800 will upgrade within a year, 600 will upgrade within 2 years, 600 will never upgrade.
• Shared library: 50 internal consumers, version adoption follows a power law — most upgrade quickly, but a long tail stays on old versions indefinitely.

​

Measuring Adoption and Sunsetting Safely

1. Version header in every request. For APIs: require or encourage clients to send a X-Client-Version or User-Agent header with their library/integration version. Log it. Dashboard it. When you see 0 requests from v1 for 14 consecutive days, v1 is safe to remove.
2. Schema version telemetry for events. Every event includes a schema_version field. The consuming services log which version they processed. When no consumer has processed a v1 event in 30 days, v1 can be deprecated.
3. Library version dependency scanning. For internal shared libraries, use your artifact repository (Artifactory, npm registry, PyPI) to query which services depend on which versions. Combine with deployment data to identify which running services use old versions (a dependency in package.json does not mean it is deployed).

1. Announce deprecation. Communicate the timeline and the reason to all known consumers. For internal APIs: Slack channel + email to tech leads. For public APIs: changelog entry + email + in-response deprecation header (Deprecation: true, Sunset: 2026-06-01).
2. Monitor adoption daily. Track the percentage of traffic on the deprecated version. It should trend toward zero. If it plateaus, reach out to the remaining consumers.
3. Provide migration tooling. Do not just tell consumers “upgrade.” Give them: a migration guide, a diff of the old vs new schema, a code example showing the changes, and ideally a codemod or script that automates the migration.
4. Grace period with degraded support. After the sunset date, stop adding features to the old version but continue serving it for a grace period (30-90 days). Return a Warning header on every response: Warning: 299 - "API v1 is deprecated and will be removed on 2026-09-01".
5. Hard removal. After the grace period, return 410 Gone with a response body pointing to the migration guide. Do not return 404 — 410 explicitly communicates “this existed and was intentionally removed.”

​

Curated Resources — Versioning and Change Management

​

The Production Readiness Checklist

​

Before Merge

• Unit tests pass — all business logic tested with deterministic inputs, including edge cases (nulls, empty strings, boundary values, leap years, unicode).
• Integration tests pass — database queries verified against real infrastructure (Testcontainers). External API interactions contract-tested or stub-verified.
• Contract tests pass — if this change touches an API response, event schema, or shared data format, consumer contracts verified via Pact or equivalent.
• Schema migration is rollback-safe — the previous application version can run against the new database state without crashing or producing incorrect results.
• Feature flag tests cover both states — if the change is behind a flag, both flag-on and flag-off are tested.
• No PII in logs — new log statements reviewed for accidental PII. Structured logging with typed events, not raw object dumps.
• Lint and type checks pass — static analysis catches obvious issues before human reviewers spend time.
• Security review (if applicable) — changes to authentication, authorization, encryption, or data handling reviewed by a security-aware engineer.

​

Before Deploy

• Build artifact matches the tested commit — the exact same artifact that passed CI is deployed. No rebuilds, no “it compiled differently on the deploy server.”
• Configuration parity verified — environment variables, secrets, feature flag states, and connection strings match expectations. Drift between staging and production is flagged.
• Database migration applied successfully — migration ran without errors, no lock contention detected, replication lag within threshold.
• Health check passes on new instances — the /health endpoint confirms the service can reach its database, cache, and critical downstream dependencies.
• Version reported matches expected — the deployed service reports the correct git SHA or build number.
• Canary deployment initiated — new version receives a small percentage of traffic (5-10%). Error rates, latency percentiles, and key business metrics are compared against the control group.

​

After Deploy

• Error rate stable or improved — no spike in 5xx errors, timeouts, or exception rates compared to pre-deploy baseline. Checked per-service and per-endpoint.
• Latency within SLO — p50, p95, and p99 latency for critical endpoints within defined thresholds. Bimodal distributions investigated.
• Business metrics healthy — conversion rate, order volume, signup rate, or other key business metrics are not degraded. A/B comparison if canary was used.
• Smoke tests pass against production — a small set of synthetic requests to critical paths (login, checkout, API key endpoints) executed against the live deployment.
• Logs flowing and queryable — structured logs from the new version are visible in the log aggregator with correct trace IDs, service names, and version tags.
• Rollback tested or verified ready — the team has confirmed that a rollback can be executed within the target window (typically <5 minutes) and that the rollback path is safe given the current database state.
• Alert channels operational — PagerDuty/Opsgenie alerts are configured for the new service version. On-call engineer is aware of the deployment.

​

Interview Deep-Dive Questions

​

Q1. You inherit a codebase with 4,000 tests, but the team tells you “nobody trusts the suite.” How do you diagnose what is wrong and build a recovery plan?

​

Follow-up: How do you handle a test that is flaky only in CI but never reproduces locally?

​

Follow-up: What metrics would you put on an engineering dashboard to track test suite health over time?

​

Q2. Explain the difference between mocks, stubs, fakes, and spies. When is each one the right choice, and when does using one become a code smell?

​

Follow-up: A teammate argues that Testcontainers has made fakes obsolete — “just use a real database in every test.” How do you respond?

​

Going Deeper: How would you test a complex SQL query that uses window functions and CTEs — is a fake even possible there?

​

Q3. Walk me through how you would design a structured logging strategy for a system of 20 microservices processing 50,000 requests per second.

​

Follow-up: How do you handle the tension between “log everything for debugging” and “logging costs us $40,000/month”?

​

Follow-up: An incident happens and the on-call engineer says “I can’t find anything useful in the logs.” What went wrong and how do you prevent it?

​

Q4. Your team uses feature flags extensively. A developer proposes adding a flag for a database migration. Is that a good idea or a terrible one?

​

Follow-up: How do you prevent feature flag accumulation from becoming its own form of technical debt?

​

Q5. Explain the expand-and-contract pattern for a schema migration. Now tell me what goes wrong with it at scale that the textbook explanation never mentions.

​

Follow-up: How do you backfill 50 million rows without locking the table or causing replication lag?

-- Pseudocode for a batched backfill
DO $$
DECLARE
  batch_size INT := 5000;
  rows_updated INT;
BEGIN
  LOOP
    UPDATE users
    SET display_name = username
    WHERE id IN (
      SELECT id FROM users
      WHERE display_name IS NULL
      LIMIT batch_size
      FOR UPDATE SKIP LOCKED  -- avoid contention with concurrent writes
    );
    GET DIAGNOSTICS rows_updated = ROW_COUNT;
    EXIT WHEN rows_updated = 0;
    PERFORM pg_sleep(0.2);  -- 200ms pause between batches
  END LOOP;
END $$;

​

Q6. What is the difference between operational logs and audit logs, and why does the distinction matter architecturally?

​

Follow-up: How do you handle the GDPR “right to be forgotten” when your audit logs are supposed to be immutable?

​

Q7. You are designing the contract testing strategy for a system where Service A publishes events to Kafka and Services B, C, and D consume them. Walk me through how you set this up.

// Consumer B's contract test
describe("OrderPlaced event contract", () => {
  it("processes billing for an OrderPlaced event", () => {
    const event = {
      type: "OrderPlaced",
      orderId: "ord-123",
      customerId: "cust-456",
      totalAmount: 99.99
      // Note: Consumer B does not declare 'items' -- it does not use that field
    };
    const result = billingHandler.process(event);
    expect(result.invoiceCreated).toBe(true);
  });
});

​

Follow-up: Service A needs to make a genuinely breaking change to the event schema. How do you manage this across three consumers?

​

Q8. Tell me about a time when poor logging made an incident significantly harder to resolve. What would you change?

​

Follow-up: How do you get a team to adopt structured logging discipline when they are used to console.log("something happened")?

​

Q9. A critical bug was caught by a contract test in CI, not by unit tests or integration tests. What does that tell you about the test strategy?

​

Follow-up: How do you decide which service boundaries need contract tests and which can get away without them?

​

Q10. If you were starting a new microservices project from scratch, what testing infrastructure would you set up before writing a single line of business logic?

​

Follow-up: How do you balance this infrastructure investment against the pressure to ship the MVP quickly?

​

Going Deeper: At what point does the testing infrastructure itself become a maintenance burden, and how do you manage that?

​

Q11. Explain event schema versioning. Why can you not just update the schema and redeploy all consumers?

​

Follow-up: How does the “tolerant reader” pattern work in practice, and what are its limits?

// Tolerant reader implementation
function processOrderEvent(event) {
  const orderId = event.orderId; // required -- fail if missing
  const amount = event.totalAmount ?? event.amount ?? 0; // handle renamed field
  const currency = event.currency ?? "USD"; // default for missing field
  const shippingAddress = event.shippingAddress; // new field -- may be undefined, and that is OK
  // Process with whatever data we have
}

​

Q12. You are reviewing a PR that adds 500 lines of new unit tests but no integration tests. The code being tested makes three database queries and calls one external API. What feedback do you give?

​

Follow-up: The developer pushes back and says “Integration tests are slow and we have a deadline. These mocks are sufficient.” How do you handle that?

​

Going Deeper: How do you establish a team culture where integration tests are seen as a normal part of the PR, not an optional extra?

// This is what "easy" looks like
const { withTestDatabase } = require("@company/test-utils");

test("getOrderWithItems returns correct total", withTestDatabase(async (db) => {
  await seedOrder(db, { id: "ord-1", items: [{ sku: "A", qty: 2, price: 10 }] });
  const order = await getOrderWithItems(db, "ord-1");
  expect(order.total).toBe(20);
}));

​

Advanced Interview Scenarios

​

Q13. Your load test shows p50 latency of 12ms and p99 of 1,800ms for the same endpoint. The team says “average is fine, ship it.” Why is that the wrong call, and how do you investigate?

​

Follow-up: The investigation reveals that slow requests correlate with database read-replica lag. How do you fix this without just throwing hardware at it?

​

Follow-up: How do you performance-test this fix without affecting production users?

​

Q14. You run SELECT COUNT(*) FROM audit_logs and get 2.3 billion rows. Queries for compliance reports now take 45 minutes. An engineer proposes “just add an index.” Why is that probably not the right answer, and what do you do instead?

​

Follow-up: The compliance team needs the full 7-year retention queryable within 24 hours of a request. How do you architect that?

​

Follow-up: How do you prove the audit logs have not been tampered with over those 7 years?

​

Q15. A junior developer writes a test that mocks the system clock and freezes it to January 15. The test passes for 11 months, then breaks every January. The obvious fix is “pick a non-edge-case date.” Why is that fix wrong?

​

Follow-up: How do you prevent time-dependent bugs systematically across a team of 30 engineers?

​

Q16. Your team is migrating from a monolith to microservices. The monolith has 3,000 integration tests that hit the database directly. How do you handle the test suite during the migration — and what is the counter-intuitive mistake most teams make?

​

Follow-up: The monolith test suite takes 40 minutes. With the new microservice tests added on top, CI now takes 70 minutes. Engineering leadership says “this is unacceptable.” How do you respond?

​

Q17. Your distributed tracing shows a request spent 200ms in Service A, 150ms in Service B, and 100ms in Service C. But the user experienced 900ms total latency. Where did the other 450ms go, and how do you find it?

​

Follow-up: You find that 300ms of the dark time is TLS handshake overhead. The service-to-service calls use mTLS. How do you reduce this without weakening security?

​

Q18. Your company publishes a public REST API used by 2,000 third-party integrations. Product wants to change the created_at field from a Unix timestamp (integer) to an ISO 8601 string. “It is just a format change.” How do you handle this?

{
  "id": "res_abc123",
  "created_at": 1744307400,
  "created_at_iso": "2026-04-10T14:30:00Z"
}

​

Follow-up: A partner calls and says your API change (the additive one — adding created_at_iso) broke their integration because their parser is strict and rejects unknown fields. Whose fault is it, and what do you do?

​

Q19. An on-call engineer pages you at 3 AM: “errors spiked to 15% but I cannot find the root cause in the logs.” You have 10 minutes before this becomes a customer-facing SEV1. Walk me through your exact triage process.

​

Follow-up: You roll back the deployment and errors drop to normal. The developer says “it worked in staging.” What is your investigation framework for why staging did not catch it?

​

Q20. Your team has 85% code coverage and the CTO wants to mandate 95% across all services. As the tech lead, you disagree. How do you make your case, and what metric do you propose instead?

// This test "covers" processPayment but verifies nothing
test("processPayment runs", () => {
  processPayment({ amount: 50, currency: "USD" });
  // no assertion — coverage says this line is "covered"
});

​

Follow-up: The CTO says “I understand your technical argument, but we need a simple number for the board. Give me something I can put on a slide.” What do you do?

​

Follow-up: How would you roll out mutation testing on a team that has never used it, without it feeling like a punishment?

​

Q21. You discover that your structured logging is accidentally writing PII (email addresses, IP addresses) into your log aggregator. You have 90 days of contaminated logs in Datadog. What is your incident response plan?

​

Follow-up: How do you design a logging system from scratch that makes PII leakage structurally impossible, not just discouraged?

// Only these fields can be logged — no raw user objects, no request bodies
interface OrderPlacedEvent {
  orderId: string;
  userId: string;       // pseudonymous ID, not email
  amount: number;
  currency: string;
  itemCount: number;
  duration_ms: number;
}

// The logger only accepts known event types
logger.event<OrderPlacedEvent>("order.placed", {
  orderId: order.id,
  userId: order.userId,  // "usr_abc123", not "jane@example.com"
  amount: order.total,
  currency: order.currency,
  itemCount: order.items.length,
  duration_ms: elapsed,
});