Car breaks down
  ↓
Mechanic manually fixes it
  ↓
Car breaks down again next month
  ↓
Mechanic manually fixes it again
  ↓
Repeat forever... ❌

Problems:
- Reactive (fixing problems after they happen)
- Manual work (mechanic required every time)
- No improvement (same problems repeat)
- Doesn't scale (need more mechanics as fleet grows)

Car sensor detects problem before failure
  ↓
Automated system orders replacement part
  ↓
Technician installs during scheduled maintenance
  ↓
System analyzes failure pattern
  ↓
Software patch prevents issue in all cars ✅

Benefits:
- Proactive (preventing problems before they happen)
- Automated (systems fix themselves)
- Continuous improvement (learn from every incident)
- Scales (same team manages 10x more systems)

Pizza Delivery Service:
- SLI #1: Delivery time (measured with GPS tracker)
  - Order at 7:00 PM → Delivered at 7:25 PM = 25 minutes

- SLI #2: Pizza temperature (measured with thermometer)
  - Temperature at delivery: 145°F

- SLI #3: Order accuracy (measured by customer)
  - Correct toppings: Yes / No

**SLI = The actual measurements**

Pizza Company's Goals (not told to customers):
- SLO #1: 95% of pizzas delivered in < 30 minutes
- SLO #2: 99% of pizzas arrive above 140°F
- SLO #3: 99.9% of orders are correct

**SLO = What you promise yourself to achieve**

Pizza Company's Guarantee (public promise):
- SLA: "Pizza in 30 minutes or it's free!"

**SLA = What you promise customers (with consequences)**

SLI (What you measure) → SLO (Your goal) → SLA (Customer promise)

Real Numbers:
- SLI: Last 100 pizzas averaged 28 minutes
- SLO: We want 95% delivered in < 30 min
- SLA: We promise 30 min or it's free

Buffer: SLO (95%) is tighter than SLA (100%)
→ This gives you room for occasional failures ✅

What we measure:
- API response time: 250ms (measured with Application Insights)
- Error rate: 0.05% (measured with logs)
- Uptime: 99.95% (measured with health checks)

These are FACTS (actual measurements)

Our internal targets:
- p95 latency < 300ms (95% of requests faster than 300ms)
- Error rate < 0.1% (99.9% success)
- Uptime > 99.9% (43 minutes downtime/month OK)

These guide our engineering decisions

Our public guarantee:
- 99.5% uptime or we give you credits
- (We promise customers 99.5%, but aim for 99.9% internally)

Buffer: 99.9% (SLO) vs 99.5% (SLA) = 0.4% buffer ✅

Month 1:
- Actual uptime: 99.8% ✅ (above SLO 99.9%? No, but above SLA 99.5%? Yes!)
- No customer refunds
- Internal: "We missed our SLO, let's improve"

Month 2:
- Actual uptime: 99.4% ❌ (below SLA 99.5%)
- Customer refunds triggered
- Internal: "Emergency! Stop feature releases!"

The buffer prevents small problems from costing money

SLO: 99.9% uptime

100% - 99.9% = 0.1% allowed downtime

0.1% of 30 days (1 month):
30 days = 43,200 minutes
0.1% of 43,200 = 43.2 minutes

Error Budget: 43.2 minutes/month ✅

You have 43.2 minutes of downtime to "spend" each month

Week 1: 10 minutes downtime
  Remaining: 33.2 minutes (77%)

Week 2: 20 minutes downtime
  Total used: 30 minutes
  Remaining: 13.2 minutes (31%)

Week 3: 15 minutes downtime
  Total used: 45 minutes
  Remaining: -1.8 minutes ❌

STATUS: OVER BUDGET! Stop deploying! ❌

Error Budget: 43.2 minutes/month
Used so far: 5 minutes (12%)
Remaining: 38.2 minutes (88%)

Decision: ✅ Ship new feature Friday!
Why? We have plenty of budget left
If it breaks, we can afford 30 min of downtime

Error Budget: 43.2 minutes/month
Used so far: 40 minutes (93%)
Remaining: 3.2 minutes (7%)

Decision: ❌ STOP all deployments!
Why? We're almost out of budget
Focus on stability, not new features

Netflix SLO: 99.99% uptime
Error Budget: 4.3 minutes/month

Policy:
- Budget > 50% → Deploy multiple times per day ✅
- Budget 25-50% → Deploy once per day with extra testing ⚠️
- Budget < 25% → FREEZE deployments, fix reliability 🚨

Result:
- Balances innovation (new features) with stability (uptime)
- Engineers don't argue about "should we ship?" → Check error budget!
- Data-driven decisions, not politics

Engineer: "Let's aim for 99.999% uptime!" (26 seconds downtime/month)

Cost to achieve 99.999%:
- Multi-region active-active: $50,000/month
- Chaos engineering tools: $5,000/month
- SRE team (5 people): $75,000/month
Total: $130,000/month

Revenue: $10,000/month ❌
Lost money: $120,000/month ❌

Internal tool used by 10 employees:
- SLO: 99% (7 hours downtime/month) ✅
- Cost: $500/month
- Why: Employees can wait, not generating revenue

E-commerce site with $1M/month revenue:
- SLO: 99.99% (4 minutes downtime/month) ✅
- Cost: $10,000/month
- Why: $1M ÷ 43,200 min/month = $23/minute lost revenue
- ROI: $10k cost prevents $100k+ losses ✅

Month 1: 40 minutes downtime (over budget)
→ Team: "Let's ship anyway!" ❌
→ Month 2: 60 minutes downtime (way over budget)
→ CEO: "Why is our site always down?!" ❌

Error Budget Policy:
├─ Budget > 50% → Normal operations (deploy freely)
├─ Budget 25-50% → Caution mode (extra testing required)
└─ Budget < 25% → FREEZE (only emergency fixes)

Month 1: 40 min downtime → Over budget
→ FREEZE enforced ✅
→ Team fixes reliability issues
→ Month 2: Only 5 min downtime
→ Budget restored, resume normal operations ✅

Bad SLO:
- "CPU < 80%" ❌
- "Database connections < 100" ❌
- "Server is running" ❌

Problem: Server can be "running" but users see errors!

Good SLO:
- "95% of page loads < 2 seconds" ✅ (user-facing)
- "99.9% of API calls succeed" ✅ (user-facing)
- "99% of search results in < 500ms" ✅ (user-facing)

Why better? Measures what users actually experience

Team defines 50 SLOs:
- API latency p50, p75, p90, p95, p99, p99.9
- Error rate by endpoint (20 endpoints)
- Database query time by query type (30 types)

Result: Nobody knows which metrics matter ❌
Meetings spent arguing about SLO violations ❌

E-Commerce Site (5 SLOs):
1. Availability: 99.9% (most important)
2. p95 page load: < 2 seconds
3. p95 API latency: < 300ms
4. Payment error rate: < 0.01%
5. Search results: < 500ms (p99)

Rule: If it's not in top 5, don't make it an SLO

❌ "Server is up" (doesn't reflect user experience)
❌ "Average latency" (hides outliers)
❌ "Internal queue depth" (users don't care)

✅ "95% of API requests complete in < 300ms"
✅ "99.9% of requests return 2xx or 3xx"
✅ "Page load completes in < 2 seconds (p95)"
✅ "Search results returned in < 500ms (p99)"

Month: January
SLO: 99.9% availability (error budget = 0.1% = 43.2 minutes)

Week 1: 5 minutes downtime
  Remaining budget: 38.2 minutes (88%)
  ✅ Status: GREEN - ship new features

Week 2: 15 minutes downtime (cumulative: 20 min)
  Remaining budget: 23.2 minutes (54%)
  ✅ Status: GREEN - continue

Week 3: 20 minutes downtime (cumulative: 40 min)
  Remaining budget: 3.2 minutes (7%)
  🚨 Status: RED - STOP feature releases
  Actions:
    - Cancel Friday deployment
    - Focus on reliability fixes
    - Root cause analysis of incidents
    - Increase monitoring

Week 4: 0 minutes downtime (cumulative: 40 min)
  Remaining budget: 3.2 minutes (7%)
  ⚠️  Status: YELLOW - Careful releases only
  Actions:
    - Small, low-risk changes only
    - Extended bake time
    - Manual approval required

// Error budget burn rate (Azure Monitor)
let slo = 99.9; // 99.9% availability SLO
let errorBudget = 100 - slo; // 0.1%
let timeWindow = 30d; // Monthly budget

requests
| where timestamp > ago(timeWindow)
| summarize
    total = count(),
    failed = countif(success == false)
| extend
    actualAvailability = ((total - failed) * 100.0) / total,
    budgetUsed = 100 - actualAvailability,
    budgetRemaining = errorBudget - (100 - actualAvailability),
    burnRate = (100 - actualAvailability) / errorBudget // 1.0 = burning at expected rate
| project
    SLO = slo,
    ActualAvailability = round(actualAvailability, 2),
    ErrorBudget = errorBudget,
    BudgetUsed = round(budgetUsed, 4),
    BudgetRemaining = round(budgetRemaining, 4),
    BurnRate = round(burnRate, 2),
    Status = case(
        burnRate < 0.5, "GREEN - Under budget",
        burnRate < 1.0, "YELLOW - On track",
        burnRate < 2.0, "ORANGE - Over budget",
        "RED - Critical"
    )

# Enable Chaos Studio on AKS cluster
az aks update \
  --resource-group rg-prod \
  --name aks-prod \
  --enable-chaos

# Create chaos experiment
az chaos experiment create \
  --name "pod-failure-experiment" \
  --resource-group rg-chaos \
  --location eastus \
  --identity SystemAssigned \
  --steps '[
    {
      "name": "Kill Random Pods",
      "branches": [
        {
          "name": "Kill 30% of pods",
          "actions": [
            {
              "type": "continuous",
              "name": "urn:csci:microsoft:azureKubernetesServiceChaosMesh:podChaos/2.1",
              "duration": "PT5M",
              "parameters": [
                {
                  "key": "jsonSpec",
                  "value": "{\"action\":\"pod-kill\",\"mode\":\"fixed-percent\",\"value\":\"30\"}"
                }
              ],
              "selectorId": "aks-cluster-target"
            }
          ]
        }
      ]
    }
  ]'

# Run experiment
az chaos experiment start \
  --name "pod-failure-experiment" \
  --resource-group rg-chaos

Timeline:
09:00 - Start load test (5,000 RPS)
09:15 - Kill 50% of web pods
09:20 - Inject 1s latency to payment service
09:25 - Trigger database failover
09:30 - Simulate CDN failure (disable Azure Front Door)
09:35 - End test

Metrics to Track:
- Error rate (target: < 1%)
- p95 latency (target: < 2s)
- Successful checkouts (target: > 99%)
- Revenue lost (target: < $1,000)

Team Roles:
- Incident Commander
- Operations (executes fixes)
- Communications (updates stakeholders)
- Observers (document learnings)

Post-GameDay:
- Blameless postmortem
- Action items with owners
- Update runbooks

# Incident Postmortem: Payment Service Outage

**Date**: 2026-01-20
**Duration**: 47 minutes
**Severity**: SEV1
**Impact**: $125K revenue lost, 15K failed transactions

## Timeline

09:42 - Alert fired: Payment API error rate 50%
09:43 - On-call engineer acknowledged
09:45 - Incident Commander assigned
09:47 - Root cause identified: Database connection pool exhausted
09:50 - Mitigation: Restarted API pods
09:55 - Error rate dropped to 5%
10:00 - Mitigation: Increased connection pool size
10:15 - Error rate back to normal (< 0.1%)
10:29 - Incident resolved

## Root Cause

Database connection pool configured for 100 connections.
Traffic spike (Black Friday sale) increased to 5,000 RPS.
Each request held connection for 500ms average.
Required connections: 5,000 * 0.5 = 2,500.

Result: Connection pool exhausted → requests failed.

## What Went Well

✅ Alert fired within 30 seconds
✅ Clear escalation path
✅ Team mobilized quickly
✅ Mitigation identified in < 5 minutes

## What Went Wrong

❌ No capacity planning for Black Friday
❌ Connection pool size not monitored
❌ No autoscaling for connection pool
❌ Runbook didn't cover this scenario

## Action Items

1. [P0] Increase connection pool to 5,000 (@alice, Due: Jan 21)
2. [P0] Add connection pool saturation alert (@bob, Due: Jan 21)
3. [P1] Implement autoscaling for connection pool (@charlie, Due: Jan 25)
4. [P1] Update runbook with connection pool troubleshooting (@david, Due: Jan 27)
5. [P2] Load test before major events (@team, Due: Feb 1)

## Lessons Learned

- Monitor resource saturation, not just errors
- Load test before high-traffic events
- Connection pools are a finite resource

Team of 6 engineers
Rotation: 1 week on-call, 5 weeks off

Primary On-Call:
- Responds to all alerts
- Pages: 5-10 per week expected
- Compensation: $500/week stipend

Secondary On-Call:
- Backup if primary unreachable
- Takes over for escalations
- Compensation: $250/week stipend

Schedule:
Week 1: Alice (primary), Bob (secondary)
Week 2: Charlie (primary), David (secondary)
Week 3: Eve (primary), Frank (secondary)
...

1. Primary on-call (5 min)
   ↓
2. Secondary on-call (10 min)
   ↓
3. Engineering Manager (15 min)
   ↓
4. VP Engineering (20 min)

## On-Call Handoff: Alice → Charlie

**Open Incidents**:
- INC-1234: Database latency spikes (SEV3, investigating)

**Ongoing Issues**:
- Redis cache hit rate dropped to 60% (normal: 90%)
- Monitoring this, no action needed yet

**Upcoming Maintenance**:
- SQL failover test scheduled for Wednesday 2am

**Tips**:
- If you see "connection timeout" errors, restart worker pods
- Database runbook: https://wiki.company.com/db-runbook
- Slack #oncall-help if you need guidance

# Create Azure Load Testing resource
az load create \
  --name myloadtest \
  --resource-group rg-prod \
  --location eastus

# Upload JMeter test plan
az load test create \
  --load-test-resource myloadtest \
  --test-id checkout-load-test \
  --display-name "Checkout API Load Test" \
  --description "Test 1000 concurrent users" \
  --test-plan checkout-test.jmx \
  --engine-instances 10

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Checkout Load Test">
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments">
        <collectionProp name="Arguments.arguments">
          <elementProp name="TARGET_URL" elementType="Argument">
            <stringProp name="Argument.name">TARGET_URL</stringProp>
            <stringProp name="Argument.value">${__P(TARGET_URL,https://api.example.com)}</stringProp>
          </elementProp>
        </collectionProp>
      </elementProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Users">
        <intProp name="ThreadGroup.num_threads">1000</intProp>
        <intProp name="ThreadGroup.ramp_time">60</intProp>
        <longProp name="ThreadGroup.duration">600</longProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="POST /checkout">
          <stringProp name="HTTPSampler.domain">${TARGET_URL}</stringProp>
          <stringProp name="HTTPSampler.path">/api/checkout</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.auto_redirects">true</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
        </HTTPSamplerProxy>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // Ramp up to 100 users
    { duration: '5m', target: 100 },   // Stay at 100 users
    { duration: '2m', target: 200 },   // Spike to 200 users
    { duration: '5m', target: 200 },   // Stay at 200 users
    { duration: '2m', target: 0 },     // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95% of requests < 500ms
    http_req_failed: ['rate<0.01'],     // Error rate < 1%
  },
};

export default function () {
  const payload = JSON.stringify({
    userId: `user${__VU}`,
    cartId: `cart${__VU}`,
    items: [{ productId: 'prod123', quantity: 2 }],
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${__ENV.API_TOKEN}`,
    },
  };

  const res = http.post('https://api.example.com/checkout', payload, params);

  check(res, {
    'status is 200': (r) => r.status === 200,
    'checkout successful': (r) => r.json('success') === true,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);  // Think time: 1 second between requests
}

# Local test
k6 run load-test.js

# Cloud test (distributed load)
k6 cloud load-test.js

# With environment variables
k6 run -e API_TOKEN=$TOKEN load-test.js

# Output to InfluxDB + Grafana
k6 run --out influxdb=http://localhost:8086/k6 load-test.js

// black-friday.js
export const options = {
  scenarios: {
    // Normal traffic (1000 RPS)
    normal_traffic: {
      executor: 'constant-arrival-rate',
      rate: 1000,
      timeUnit: '1s',
      duration: '1h',
      preAllocatedVUs: 500,
      maxVUs: 2000,
    },
    // Black Friday spike (10x traffic for 30 min)
    black_friday_spike: {
      executor: 'constant-arrival-rate',
      rate: 10000,
      timeUnit: '1s',
      duration: '30m',
      startTime: '1h',  // Start after normal traffic
      preAllocatedVUs: 5000,
      maxVUs: 20000,
    },
  },
  thresholds: {
    'http_req_duration{scenario:normal_traffic}': ['p(99)<1000'],
    'http_req_duration{scenario:black_friday_spike}': ['p(99)<2000'],
    'http_req_failed': ['rate<0.05'],  // 5% error budget during spike
  },
};

// soak-test.js
export const options = {
  stages: [
    { duration: '5m', target: 100 },    // Ramp up
    { duration: '24h', target: 100 },   // Run for 24 hours
    { duration: '5m', target: 0 },      // Ramp down
  ],
};

export default function () {
  http.get('https://api.example.com/products');
  sleep(3);  // 3 seconds think time
}

// Program.cs
services.AddApplicationInsightsTelemetry(options =>
{
    options.EnableAdaptiveSampling = false;  // Disable during load tests
    options.DeveloperMode = false;
});

// LoadTestController.cs
[HttpGet("health")]
public IActionResult Health()
{
    var telemetry = new TelemetryClient();

    // Track custom metric during load test
    telemetry.TrackMetric("DatabaseConnectionPoolSize",
        GetConnectionPoolSize());

    telemetry.TrackMetric("MemoryUsageMB",
        GC.GetTotalMemory(false) / 1024 / 1024);

    return Ok(new { status = "healthy" });
}

// Performance during load test
requests
| where timestamp between(datetime(2026-01-21 10:00) .. datetime(2026-01-21 11:00))
| summarize
    P50 = percentile(duration, 50),
    P95 = percentile(duration, 95),
    P99 = percentile(duration, 99),
    RequestRate = count() / (max(timestamp) - min(timestamp)) / 1s,
    FailureRate = countif(success == false) * 100.0 / count()
    by bin(timestamp, 1m)
| project timestamp, P50, P95, P99, RequestRate, FailureRate
| render timechart

// Bad: Same user every time
const userId = 'user123';

// Good: Random users from pool
const users = ['user1', 'user2', 'user3', ...];  // 10,000 users
const userId = users[Math.floor(Math.random() * users.length)];

// Better: Unique user per VU
const userId = `user${__VU}`;

export default function () {
  // User lands on homepage
  http.get('https://example.com/');
  sleep(2);  // User reads content

  // User searches for product
  http.get('https://example.com/search?q=laptop');
  sleep(3);  // User browses results

  // User clicks product
  http.get('https://example.com/products/laptop-123');
  sleep(5);  // User reads reviews

  // User adds to cart
  http.post('https://example.com/cart/add', { productId: 'laptop-123' });
  sleep(1);
}

export const options = {
  batch: 20,  // Send 20 requests in parallel per VU
  batchPerHost: 6,  // Max 6 parallel connections per host (HTTP/1.1)
};

// Bad: Instant spike (thundering herd)
export const options = {
  vus: 1000,
  duration: '10m',
};

// Good: Gradual ramp-up
export const options = {
  stages: [
    { duration: '2m', target: 100 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 1000 },
    { duration: '10m', target: 1000 },
  ],
};

// Find slowest dependencies
dependencies
| where timestamp > ago(1h)
| summarize
    AvgDuration = avg(duration),
    P95Duration = percentile(duration, 95),
    Count = count()
    by target
| order by P95Duration desc
| take 10

✅ Observability
  ✅ Metrics instrumented (Golden Signals)
  ✅ Logs centralized (Application Insights / Log Analytics)
  ✅ Distributed tracing enabled
  ✅ Dashboards created
  ✅ Alerts configured with runbooks

✅ Reliability
  ✅ SLOs defined and measured
  ✅ Error budget policy in place
  ✅ Circuit breakers implemented
  ✅ Retries with exponential backoff
  ✅ Timeouts configured
  ✅ Health checks (liveness & readiness)

✅ Scalability
  ✅ Load tested at 2x peak traffic
  ✅ Autoscaling configured
  ✅ Database connection pooling
  ✅ Caching strategy
  ✅ Rate limiting for external APIs

✅ Security
  ✅ Secrets in Key Vault (not in code)
  ✅ Managed Identity for authentication
  ✅ Network isolation (Private Endpoints)
  ✅ WAF enabled
  ✅ Security scanning in CI/CD

✅ Operational
  ✅ Runbooks documented
  ✅ On-call rotation defined
  ✅ Disaster recovery tested
  ✅ Backup and restore verified
  ✅ Chaos engineering experiments run

✅ Cost
  ✅ Cost estimation completed
  ✅ Budgets and alerts set
  ✅ Autoscaling to reduce waste
  ✅ Reserved Instances for steady load