> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SRE & Production Excellence

> Master Site Reliability Engineering, SLIs/SLOs/SLAs, error budgets, chaos engineering, and production operations

# SRE & Production Excellence

Learn how Google's Site Reliability Engineering practices apply to Azure, from setting SLOs to running chaos experiments.

<img src="https://mintlify.s3.us-west-1.amazonaws.com/devweeekends/images/azure/17-sre-pyramid.svg" alt="SRE Pyramid" />

***

## What You'll Learn

By the end of this chapter, you'll understand:

* **What SRE is** (and how it's different from traditional operations)
* **How to measure reliability** (SLIs, SLOs, SLAs explained from scratch)
* **Error budgets** (and how they balance innovation vs stability)
* **Production best practices** (deployments, rollbacks, monitoring)
* **Chaos engineering** (intentionally breaking things to build resilience)
* **Real-world examples** with actual costs and trade-offs

***

## Introduction: What is Site Reliability Engineering (SRE)?

### Start Here if You're Completely New

**SRE = Treating operations like a software engineering problem**

Think of it like maintaining a car:

**Traditional Operations** (Old Way):

```
Car breaks down
  ↓
Mechanic manually fixes it
  ↓
Car breaks down again next month
  ↓
Mechanic manually fixes it again
  ↓
Repeat forever... ❌

Problems:
- Reactive (fixing problems after they happen)
- Manual work (mechanic required every time)
- No improvement (same problems repeat)
- Doesn't scale (need more mechanics as fleet grows)
```

**Site Reliability Engineering** (New Way):

```
Car sensor detects problem before failure
  ↓
Automated system orders replacement part
  ↓
Technician installs during scheduled maintenance
  ↓
System analyzes failure pattern
  ↓
Software patch prevents issue in all cars ✅

Benefits:
- Proactive (preventing problems before they happen)
- Automated (systems fix themselves)
- Continuous improvement (learn from every incident)
- Scales (same team manages 10x more systems)
```

**SRE in Software**:

* **Automation**: Replace manual tasks with code
* **Monitoring**: Detect problems before users notice
* **Reliability**: Measure and improve system uptime
* **Balance**: Ship features while maintaining stability

***

## Why SRE Matters: The Cost of Unreliability

### Real-World Failure Examples

**Amazon Prime Day 2018**

* **Problem**: Site crashed for 63 minutes
* **Cost**: \$99 million in lost revenue
* **Per-minute cost**: \$1.57 million/minute
* **Cause**: No error budget, no chaos testing
* **Prevention cost**: \~\$1 million (SRE team + tools)
* **ROI**: 99x return on investment ✅

**More Examples**:

| Company          | Incident | Downtime   | Cost      | SRE Practice Missing              |
| ---------------- | -------- | ---------- | --------- | --------------------------------- |
| **Facebook**     | Oct 2021 | 6 hours    | \$60-100M | Change management + error budgets |
| **GitHub**       | Oct 2018 | 24 hours   | Unknown   | Chaos engineering                 |
| **Google Cloud** | Jun 2019 | 4.5 hours  | Unknown   | Gradual rollouts                  |
| **Cloudflare**   | Jul 2019 | 30 minutes | Unknown   | Progressive deployments           |

**The Pattern**: Good SRE practices cost millions. Bad SRE practices cost tens of millions.

***

## Understanding SLI, SLO, SLA (From Absolute Zero)

These are the THREE most important acronyms in SRE. Let's break them down with a real-world analogy:

### The Pizza Delivery Analogy

**SLI (Service Level Indicator) = What You Measure**

```
Pizza Delivery Service:
- SLI #1: Delivery time (measured with GPS tracker)
  - Order at 7:00 PM → Delivered at 7:25 PM = 25 minutes

- SLI #2: Pizza temperature (measured with thermometer)
  - Temperature at delivery: 145°F

- SLI #3: Order accuracy (measured by customer)
  - Correct toppings: Yes / No

**SLI = The actual measurements**
```

**SLO (Service Level Objective) = Your Internal Goal**

```
Pizza Company's Goals (not told to customers):
- SLO #1: 95% of pizzas delivered in < 30 minutes
- SLO #2: 99% of pizzas arrive above 140°F
- SLO #3: 99.9% of orders are correct

**SLO = What you promise yourself to achieve**
```

**SLA (Service Level Agreement) = Your Customer Promise**

```
Pizza Company's Guarantee (public promise):
- SLA: "Pizza in 30 minutes or it's free!"

**SLA = What you promise customers (with consequences)**
```

**The Relationship**:

```
SLI (What you measure) → SLO (Your goal) → SLA (Customer promise)

Real Numbers:
- SLI: Last 100 pizzas averaged 28 minutes
- SLO: We want 95% delivered in < 30 min
- SLA: We promise 30 min or it's free

Buffer: SLO (95%) is tighter than SLA (100%)
→ This gives you room for occasional failures ✅
```

***

### Software Example: E-Commerce Website

**SLI (Service Level Indicator) = Measurements**

```
What we measure:
- API response time: 250ms (measured with Application Insights)
- Error rate: 0.05% (measured with logs)
- Uptime: 99.95% (measured with health checks)

These are FACTS (actual measurements)
```

**SLO (Service Level Objective) = Internal Goals**

```
Our internal targets:
- p95 latency < 300ms (95% of requests faster than 300ms)
- Error rate < 0.1% (99.9% success)
- Uptime > 99.9% (43 minutes downtime/month OK)

These guide our engineering decisions
```

**SLA (Service Level Agreement) = Customer Promise**

```
Our public guarantee:
- 99.5% uptime or we give you credits
- (We promise customers 99.5%, but aim for 99.9% internally)

Buffer: 99.9% (SLO) vs 99.5% (SLA) = 0.4% buffer ✅
```

**Why the Buffer?**

```
Month 1:
- Actual uptime: 99.8% ✅ (above SLO 99.9%? No, but above SLA 99.5%? Yes!)
- No customer refunds
- Internal: "We missed our SLO, let's improve"

Month 2:
- Actual uptime: 99.4% ❌ (below SLA 99.5%)
- Customer refunds triggered
- Internal: "Emergency! Stop feature releases!"

The buffer prevents small problems from costing money
```

***

## Error Budgets Explained (From Scratch)

**Error Budget = How much downtime you're allowed**

### The Simple Math

```
SLO: 99.9% uptime

100% - 99.9% = 0.1% allowed downtime

0.1% of 30 days (1 month):
30 days = 43,200 minutes
0.1% of 43,200 = 43.2 minutes

Error Budget: 43.2 minutes/month ✅
```

**What This Means**:

```
You have 43.2 minutes of downtime to "spend" each month

Week 1: 10 minutes downtime
  Remaining: 33.2 minutes (77%)

Week 2: 20 minutes downtime
  Total used: 30 minutes
  Remaining: 13.2 minutes (31%)

Week 3: 15 minutes downtime
  Total used: 45 minutes
  Remaining: -1.8 minutes ❌

STATUS: OVER BUDGET! Stop deploying! ❌
```

***

### Error Budgets as "Innovation Currency"

Think of error budget like a bank account:

**High Balance = Take Risks**

```
Error Budget: 43.2 minutes/month
Used so far: 5 minutes (12%)
Remaining: 38.2 minutes (88%)

Decision: ✅ Ship new feature Friday!
Why? We have plenty of budget left
If it breaks, we can afford 30 min of downtime
```

**Low Balance = Play it Safe**

```
Error Budget: 43.2 minutes/month
Used so far: 40 minutes (93%)
Remaining: 3.2 minutes (7%)

Decision: ❌ STOP all deployments!
Why? We're almost out of budget
Focus on stability, not new features
```

**Real-World Example: Netflix**

```
Netflix SLO: 99.99% uptime
Error Budget: 4.3 minutes/month

Policy:
- Budget > 50% → Deploy multiple times per day ✅
- Budget 25-50% → Deploy once per day with extra testing ⚠️
- Budget < 25% → FREEZE deployments, fix reliability 🚨

Result:
- Balances innovation (new features) with stability (uptime)
- Engineers don't argue about "should we ship?" → Check error budget!
- Data-driven decisions, not politics
```

***

## Common Mistakes in SRE (Learn from Others)

### Mistake #1: Setting SLOs Too High (Perfectionism)

**The Trap**:

```
Engineer: "Let's aim for 99.999% uptime!" (26 seconds downtime/month)

Cost to achieve 99.999%:
- Multi-region active-active: $50,000/month
- Chaos engineering tools: $5,000/month
- SRE team (5 people): $75,000/month
Total: $130,000/month

Revenue: $10,000/month ❌
Lost money: $120,000/month ❌
```

**The Fix**: Match SLO to business value

```
Internal tool used by 10 employees:
- SLO: 99% (7 hours downtime/month) ✅
- Cost: $500/month
- Why: Employees can wait, not generating revenue

E-commerce site with $1M/month revenue:
- SLO: 99.99% (4 minutes downtime/month) ✅
- Cost: $10,000/month
- Why: $1M ÷ 43,200 min/month = $23/minute lost revenue
- ROI: $10k cost prevents $100k+ losses ✅
```

***

### Mistake #2: No Error Budget Policy (Chaos Deploys)

**The Trap**:

```
Month 1: 40 minutes downtime (over budget)
→ Team: "Let's ship anyway!" ❌
→ Month 2: 60 minutes downtime (way over budget)
→ CEO: "Why is our site always down?!" ❌
```

**The Fix**: Enforce error budget policy

```
Error Budget Policy:
├─ Budget > 50% → Normal operations (deploy freely)
├─ Budget 25-50% → Caution mode (extra testing required)
└─ Budget < 25% → FREEZE (only emergency fixes)

Month 1: 40 min downtime → Over budget
→ FREEZE enforced ✅
→ Team fixes reliability issues
→ Month 2: Only 5 min downtime
→ Budget restored, resume normal operations ✅
```

***

### Mistake #3: SLO Based on System Metrics (Not User Experience)

**The Trap**:

```
Bad SLO:
- "CPU < 80%" ❌
- "Database connections < 100" ❌
- "Server is running" ❌

Problem: Server can be "running" but users see errors!
```

**The Fix**: SLOs based on user experience

```
Good SLO:
- "95% of page loads < 2 seconds" ✅ (user-facing)
- "99.9% of API calls succeed" ✅ (user-facing)
- "99% of search results in < 500ms" ✅ (user-facing)

Why better? Measures what users actually experience
```

***

### Mistake #4: Too Many SLOs (Analysis Paralysis)

**The Trap**:

```
Team defines 50 SLOs:
- API latency p50, p75, p90, p95, p99, p99.9
- Error rate by endpoint (20 endpoints)
- Database query time by query type (30 types)

Result: Nobody knows which metrics matter ❌
Meetings spent arguing about SLO violations ❌
```

**The Fix**: Start with 3-5 critical SLOs

```
E-Commerce Site (5 SLOs):
1. Availability: 99.9% (most important)
2. p95 page load: < 2 seconds
3. p95 API latency: < 300ms
4. Payment error rate: < 0.01%
5. Search results: < 500ms (p99)

Rule: If it's not in top 5, don't make it an SLO
```

***

> \[!TIP]
> **Jargon Alert: SLI vs SLO vs SLA**
> **SLI (Indicator)**: What you measure (e.g., latency = 250ms)
> **SLO (Objective)**: Your internal goal (e.g., p95 latency \< 300ms)
> **SLA (Agreement)**: Your customer promise (e.g., 99.9% uptime or money back)
> SLO should be tighter than SLA to give you a buffer!

> \[!WARNING]
> **Gotcha: 99.9% vs 99.99% Availability**
> The difference is NOT 0.09%—it's **10x more downtime**!
> 99.9% = 43 minutes/month downtime
> 99.99% = 4.3 minutes/month downtime
> Achieving 99.99% can cost 10x more. Choose based on business impact, not ego.

***

## 1. The SRE Mindset

**SRE** = What happens when you ask a software engineer to design operations.

### Core SRE Principles

<CardGroup cols={2}>
  <Card title="Embrace Risk" icon="dice">
    100% reliability is impossible and wasteful. Find the right balance between reliability and velocity.
  </Card>

  <Card title="Service Level Objectives" icon="bullseye">
    Define clear, measurable reliability targets. Make data-driven decisions.
  </Card>

  <Card title="Eliminate Toil" icon="robot">
    Automate repetitive manual work. Toil is the enemy of scaling.
  </Card>

  <Card title="Monitor Everything" icon="chart-line">
    You can't improve what you don't measure. Observability is mandatory.
  </Card>

  <Card title="Error Budgets" icon="wallet">
    Use remaining error budget to balance innovation and stability.
  </Card>

  <Card title="Blameless Postmortems" icon="magnifying-glass">
    Learn from failures without blaming individuals. Focus on systemic improvements.
  </Card>
</CardGroup>

***

## 2. Service Level Indicators (SLIs)

**SLIs** are carefully chosen metrics that represent the health of your service from the user's perspective.

### The Golden Signals (Google SRE)

<Tabs>
  <Tab title="Latency">
    **How long does a request take?**

    ```kusto theme={null}
    // Azure Application Insights - Latency SLI
    // Filter to successful requests only -- failed requests (5xx) often
    // have very short durations (fast error responses), which would
    // artificially lower your latency numbers and hide real problems.
    requests
    | where timestamp > ago(24h)
    | where success == true
    | summarize
        p50 = percentile(duration, 50),  // Median: "typical" user experience
        p95 = percentile(duration, 95),  // Slow users: 1 in 20 see this or worse
        p99 = percentile(duration, 99)   // Tail latency: 1 in 100 -- often reveals DB issues
        by bin(timestamp, 5m), name      // 5-minute buckets smooth out noise
    | render timechart
    ```

    **Good SLI**:

    * p95 latency \< 300ms (fast)
    * p99 latency \< 1000ms (acceptable)

    **Why percentiles?**

    * Average hides outliers (1ms + 10,000ms = 5,000ms average, useless!)
    * p95 = 95% of users have this experience or better
  </Tab>

  <Tab title="Traffic">
    **How many requests per second?**

    ```kusto theme={null}
    // Requests per second
    requests
    | where timestamp > ago(1h)
    | summarize count() by bin(timestamp, 1m)
    | extend rps = count_ / 60
    | render timechart
    ```

    **Good SLI**:

    * Current traffic: 5,000 RPS
    * Peak capacity: 10,000 RPS
    * Headroom: 100% (good!)
  </Tab>

  <Tab title="Errors">
    **What percentage of requests fail?**

    ```kusto theme={null}
    // Error rate SLI
    requests
    | where timestamp > ago(24h)
    | summarize
        total = count(),
        failed = countif(success == false)
        by bin(timestamp, 5m)
    | extend errorRate = (failed * 100.0) / total
    | render timechart
    ```

    **Good SLI**:

    * Error rate \< 0.1% (99.9% success)

    **Error classification**:

    * 4xx errors: Client errors (not counted against SLO)
    * 5xx errors: Server errors (counted!)
  </Tab>

  <Tab title="Saturation">
    **How full is your service?**

    ```kusto theme={null}
    // CPU saturation
    performanceCounters
    | where timestamp > ago(1h)
    | where name == "% Processor Time"
    | summarize avg(value) by bin(timestamp, 5m), cloud_RoleInstance
    | render timechart

    // Memory saturation
    performanceCounters
    | where name == "Available Bytes"
    | summarize avg(value / 1024 / 1024 / 1024) by bin(timestamp, 5m)
    | render timechart
    ```

    **Good SLI**:

    * CPU \< 70% (30% headroom)
    * Memory \< 80%
    * Disk \< 80%
  </Tab>
</Tabs>

### Choosing Good SLIs

**Bad SLI Examples**:

```
❌ "Server is up" (doesn't reflect user experience)
❌ "Average latency" (hides outliers)
❌ "Internal queue depth" (users don't care)
```

**Good SLI Examples**:

```
✅ "95% of API requests complete in < 300ms"
✅ "99.9% of requests return 2xx or 3xx"
✅ "Page load completes in < 2 seconds (p95)"
✅ "Search results returned in < 500ms (p99)"
```

***

## 3. Service Level Objectives (SLOs)

**SLOs** are internal reliability targets. They should be slightly tighter than your SLA.

### SLO Examples

<AccordionGroup>
  <Accordion title="E-Commerce Site" icon="cart-shopping">
    **SLIs & SLOs**:

    ```
    Latency:
    - SLI: p95 page load time
    - SLO: < 2 seconds
    - Why: Studies show 2s+ loads = 20% abandonment

    Availability:
    - SLI: Successful checkout rate
    - SLO: 99.9% (43 min downtime/month)
    - Why: Every minute down = $10K revenue lost

    Errors:
    - SLI: Payment processing error rate
    - SLO: < 0.01%
    - Why: Payment errors damage trust
    ```
  </Accordion>

  <Accordion title="API Service" icon="code">
    **SLIs & SLOs**:

    ```
    Latency:
    - SLI: p95 API response time
    - SLO: < 100ms
    - Why: Third-party integrations need fast responses

    Availability:
    - SLI: Successful API call rate
    - SLO: 99.95% (21 min downtime/month)
    - Why: Higher than web app (upstream dependency)

    Throughput:
    - SLI: Requests per second handled
    - SLO: 10,000 RPS sustained
    - Why: Contract guarantees 5,000 RPS average
    ```
  </Accordion>

  <Accordion title="Batch Processing System" icon="gears">
    **SLIs & SLOs**:

    ```
    Freshness:
    - SLI: Time from data arrival to processing completion
    - SLO: < 5 minutes (p95)
    - Why: Near real-time reporting requirement

    Correctness:
    - SLI: Percentage of jobs completing without errors
    - SLO: 99.99%
    - Why: Data quality is critical

    Throughput:
    - SLI: Events processed per hour
    - SLO: 100 million events/hour
    - Why: Peak load during business hours
    ```
  </Accordion>
</AccordionGroup>

***

## 4. Error Budgets

**Error Budget** = 100% - SLO

If your SLO is 99.9% availability, your error budget is 0.1% = 43.2 minutes/month.

### Error Budget Policy

<div className="flex justify-center my-4">
  <div className="w-full max-w-4xl">
    ```mermaid theme={null}
    graph TD
        A[Check Error Budget] --> B{Budget Remaining?}
        B -->|> 50%| C[✅ Ship Features<br/>Take Risks<br/>Accelerate]
        B -->|10-50%| D[⚠️  Caution Mode<br/>Increase Testing<br/>Slow Down]
        B -->|< 10%| E[🚨 Stop Releases<br/>Fix Reliability<br/>Pay Down Debt]

        C --> F[Monitor Closely]
        D --> F
        E --> G[Emergency Fixes Only]

        style C fill:#51CF66
        style D fill:#FFB900
        style E fill:#FF6B6B
    ```
  </div>
</div>

**Example Decision Making**:

```
Month: January
SLO: 99.9% availability (error budget = 0.1% = 43.2 minutes)

Week 1: 5 minutes downtime
  Remaining budget: 38.2 minutes (88%)
  ✅ Status: GREEN - ship new features

Week 2: 15 minutes downtime (cumulative: 20 min)
  Remaining budget: 23.2 minutes (54%)
  ✅ Status: GREEN - continue

Week 3: 20 minutes downtime (cumulative: 40 min)
  Remaining budget: 3.2 minutes (7%)
  🚨 Status: RED - STOP feature releases
  Actions:
    - Cancel Friday deployment
    - Focus on reliability fixes
    - Root cause analysis of incidents
    - Increase monitoring

Week 4: 0 minutes downtime (cumulative: 40 min)
  Remaining budget: 3.2 minutes (7%)
  ⚠️  Status: YELLOW - Careful releases only
  Actions:
    - Small, low-risk changes only
    - Extended bake time
    - Manual approval required
```

### Calculating Error Budget Burn Rate

```kusto theme={null}
// Error budget burn rate (Azure Monitor)
let slo = 99.9; // 99.9% availability SLO
let errorBudget = 100 - slo; // 0.1%
let timeWindow = 30d; // Monthly budget

requests
| where timestamp > ago(timeWindow)
| summarize
    total = count(),
    failed = countif(success == false)
| extend
    actualAvailability = ((total - failed) * 100.0) / total,
    budgetUsed = 100 - actualAvailability,
    budgetRemaining = errorBudget - (100 - actualAvailability),
    burnRate = (100 - actualAvailability) / errorBudget // 1.0 = burning at expected rate
| project
    SLO = slo,
    ActualAvailability = round(actualAvailability, 2),
    ErrorBudget = errorBudget,
    BudgetUsed = round(budgetUsed, 4),
    BudgetRemaining = round(budgetRemaining, 4),
    BurnRate = round(burnRate, 2),
    Status = case(
        burnRate < 0.5, "GREEN - Under budget",
        burnRate < 1.0, "YELLOW - On track",
        burnRate < 2.0, "ORANGE - Over budget",
        "RED - Critical"
    )
```

***

## 5. Toil Reduction

**Toil** = Repetitive, manual, automatable work that scales linearly with service growth.

### What is Toil?

<Tabs>
  <Tab title="✅ Toil Examples">
    ```
    ✅ Manually restarting failed services
    ✅ Provisioning new servers by hand
    ✅ Copy-pasting database queries for reports
    ✅ Manually scaling resources up/down
    ✅ SSH into servers to check logs
    ✅ Running the same kubectl commands daily
    ✅ Manually updating configuration files
    ✅ Responding to the same alerts with same fix
    ```

    **Characteristics**:

    * Manual
    * Repetitive
    * Automatable
    * Tactical (no long-term value)
    * Scales linearly with service size
  </Tab>

  <Tab title="❌ Not Toil">
    ```
    ❌ Writing automation scripts (engineering work)
    ❌ Planning capacity (strategic thinking)
    ❌ Incident response (novel problem-solving)
    ❌ Code reviews (knowledge sharing)
    ❌ Architecture design (creative work)
    ❌ Debugging new issues (requires expertise)
    ```

    **Characteristics**:

    * Requires human judgment
    * Novel problems
    * Strategic value
    * Knowledge creation
  </Tab>
</Tabs>

### Toil Elimination Strategy

<Steps>
  <Step title="Measure Toil">
    Track time spent on repetitive tasks

    ```
    Weekly time audit:
    - Manual deployments: 8 hours
    - Alert triage (false positives): 5 hours
    - Log searching: 4 hours
    - Manual scaling: 3 hours
    - Config updates: 2 hours

    Total toil: 22 hours/week (55% of time!)
    ```
  </Step>

  <Step title="Prioritize by ROI">
    Automate high-frequency, high-time tasks first

    ```
    Task                  Frequency    Time    Total/Week    ROI
    Manual deployments    10/week      0.5h    5h            High
    False alert triage    50/week      0.1h    5h            High
    Manual scaling        20/week      0.15h   3h            Medium
    Config updates        5/week       0.4h    2h            Low
    ```
  </Step>

  <Step title="Automate">
    Examples:

    ```bash theme={null}
    # Before: Manual deployment (30 min)
    ssh server
    sudo systemctl stop app
    git pull
    dotnet publish
    sudo systemctl start app

    # After: CI/CD pipeline (2 min, no human)
    git push
    # GitHub Actions deploys automatically

    # Before: Manual scaling (20 min)
    # Check metrics, decide, update config, restart

    # After: Autoscaling (0 min, automatic)
    az monitor autoscale create \
      --resource-group rg-prod \
      --resource webapp \
      --min-count 2 \
      --max-count 10 \
      --count 2

    # Before: Searching logs (15 min per incident)
    ssh server
    tail -f /var/log/app.log | grep ERROR

    # After: Centralized logging (30 seconds)
    # KQL query in Application Insights
    traces | where severityLevel == 3 | take 100
    ```
  </Step>

  <Step title="Measure Improvement">
    Track toil reduction over time

    ```
    Q1: 55% time spent on toil
    Q2: 40% (automated deployments)
    Q3: 25% (added autoscaling)
    Q4: 15% (improved alerting)

    Time reclaimed: 20 hours/week
    Used for: Feature development, reliability improvements
    ```
  </Step>
</Steps>

***

## 6. Chaos Engineering

**Chaos Engineering** = Intentionally breaking things in production to build confidence.

### Chaos Principles

1. **Build a hypothesis** (e.g., "If we kill 30% of pods, requests should still succeed")
2. **Define steady state** (e.g., "Error rate \< 1%, latency p95 \< 500ms")
3. **Introduce chaos** (e.g., kill pods, inject latency, fail database)
4. **Measure deviation** (Did error rate spike? Did latency increase?)
5. **Learn and improve** (Add retries? Implement circuit breaker?)

### Azure Chaos Studio

**Setup**:

```bash theme={null}
# Enable Chaos Studio on AKS cluster
az aks update \
  --resource-group rg-prod \
  --name aks-prod \
  --enable-chaos

# Create chaos experiment
az chaos experiment create \
  --name "pod-failure-experiment" \
  --resource-group rg-chaos \
  --location eastus \
  --identity SystemAssigned \
  --steps '[
    {
      "name": "Kill Random Pods",
      "branches": [
        {
          "name": "Kill 30% of pods",
          "actions": [
            {
              "type": "continuous",
              "name": "urn:csci:microsoft:azureKubernetesServiceChaosMesh:podChaos/2.1",
              "duration": "PT5M",
              "parameters": [
                {
                  "key": "jsonSpec",
                  "value": "{\"action\":\"pod-kill\",\"mode\":\"fixed-percent\",\"value\":\"30\"}"
                }
              ],
              "selectorId": "aks-cluster-target"
            }
          ]
        }
      ]
    }
  ]'

# Run experiment
az chaos experiment start \
  --name "pod-failure-experiment" \
  --resource-group rg-chaos
```

### Chaos Scenarios

<AccordionGroup>
  <Accordion title="1. Pod Failure (AKS)" icon="skull">
    **Hypothesis**: Application survives 30% pod failure

    **Test**:

    ```yaml theme={null}
    # Using Chaos Mesh
    apiVersion: chaos-mesh.org/v1alpha1
    kind: PodChaos
    metadata:
      name: pod-kill-experiment
    spec:
      action: pod-kill
      mode: fixed-percent
      value: "30"
      selector:
        namespaces:
          - production
        labelSelectors:
          app: web-api
      scheduler:
        cron: "@every 1h"
    ```

    **Expected**: No user-visible errors (replicas take over)
    **Actual**: ?
    **Learnings**: Need faster readiness probes? Increase replica count?
  </Accordion>

  <Accordion title="2. Network Latency" icon="network-wired">
    **Hypothesis**: Application handles 500ms network latency

    **Test**:

    ```yaml theme={null}
    apiVersion: chaos-mesh.org/v1alpha1
    kind: NetworkChaos
    metadata:
      name: network-delay
    spec:
      action: delay
      mode: one
      selector:
        namespaces:
          - production
        labelSelectors:
          app: payment-api
      delay:
        latency: "500ms"
        correlation: "100"
      duration: "5m"
    ```

    **Expected**: Requests timeout gracefully, circuit breaker opens
    **Actual**: ?
    **Learnings**: Add timeout? Implement retry with backoff?
  </Accordion>

  <Accordion title="3. CPU Stress" icon="microchip">
    **Hypothesis**: Autoscaling kicks in under CPU load

    **Test**:

    ```yaml theme={null}
    apiVersion: chaos-mesh.org/v1alpha1
    kind: StressChaos
    metadata:
      name: cpu-stress
    spec:
      mode: one
      selector:
        namespaces:
          - production
        labelSelectors:
          app: worker
      stressors:
        cpu:
          workers: 4
          load: 100
      duration: "10m"
    ```

    **Expected**: HPA scales pods from 3 → 10 within 2 minutes
    **Actual**: ?
    **Learnings**: Adjust HPA thresholds? Add more headroom?
  </Accordion>

  <Accordion title="4. Database Failover" icon="database">
    **Hypothesis**: App survives database failover

    **Test**:

    ```bash theme={null}
    # Trigger Azure SQL failover
    az sql db failover \
      --resource-group rg-prod \
      --server sql-prod \
      --database mydb

    # Expected failover time: 30-60 seconds
    ```

    **Expected**: Brief spike in errors (\< 1 minute), then recovery
    **Actual**: ?
    **Learnings**: Connection pool handles failover? Need retry logic?
  </Accordion>
</AccordionGroup>

### GameDay Exercise

**Scenario**: Black Friday Simulation

```
Timeline:
09:00 - Start load test (5,000 RPS)
09:15 - Kill 50% of web pods
09:20 - Inject 1s latency to payment service
09:25 - Trigger database failover
09:30 - Simulate CDN failure (disable Azure Front Door)
09:35 - End test

Metrics to Track:
- Error rate (target: &lt; 1%)
- p95 latency (target: &lt; 2s)
- Successful checkouts (target: > 99%)
- Revenue lost (target: &lt; $1,000)

Team Roles:
- Incident Commander
- Operations (executes fixes)
- Communications (updates stakeholders)
- Observers (document learnings)

Post-GameDay:
- Blameless postmortem
- Action items with owners
- Update runbooks
```

***

## 7. Incident Management

### Incident Response Lifecycle

<div className="flex justify-center my-4">
  <div className="w-full max-w-4xl">
    ```mermaid theme={null}
    graph LR
        A[🚨 Alert Fires] --> B[Acknowledge]
        B --> C{Severity?}
        C -->|SEV1: Critical| D[Page On-Call]
        C -->|SEV2: Major| E[Create Incident]
        C -->|SEV3: Minor| F[Create Ticket]

        D --> G[Incident Commander Assigned]
        G --> H[Assemble Team]
        H --> I[Diagnose]
        I --> J[Mitigate]
        J --> K{Resolved?}
        K -->|No| I
        K -->|Yes| L[Monitor]
        L --> M[Postmortem]

        style A fill:#FF6B6B
        style L fill:#51CF66
    ```
  </div>
</div>

### Severity Levels

| Severity | Impact                    | Response Time     | Example                                       |
| -------- | ------------------------- | ----------------- | --------------------------------------------- |
| **SEV1** | Critical - Revenue loss   | Page immediately  | Payment system down, database offline         |
| **SEV2** | Major - Degraded service  | 15 minutes        | API latency 10x higher, 10% error rate        |
| **SEV3** | Minor - Partial impact    | 1 hour            | Single pod failing, non-critical feature down |
| **SEV4** | Cosmetic - No user impact | Next business day | UI typo, log noise                            |

### Blameless Postmortem Template

```markdown theme={null}
# Incident Postmortem: Payment Service Outage

**Date**: 2026-01-20
**Duration**: 47 minutes
**Severity**: SEV1
**Impact**: $125K revenue lost, 15K failed transactions

## Timeline

09:42 - Alert fired: Payment API error rate 50%
09:43 - On-call engineer acknowledged
09:45 - Incident Commander assigned
09:47 - Root cause identified: Database connection pool exhausted
09:50 - Mitigation: Restarted API pods
09:55 - Error rate dropped to 5%
10:00 - Mitigation: Increased connection pool size
10:15 - Error rate back to normal (&lt; 0.1%)
10:29 - Incident resolved

## Root Cause

Database connection pool configured for 100 connections.
Traffic spike (Black Friday sale) increased to 5,000 RPS.
Each request held connection for 500ms average.
Required connections: 5,000 * 0.5 = 2,500.

Result: Connection pool exhausted → requests failed.

## What Went Well

✅ Alert fired within 30 seconds
✅ Clear escalation path
✅ Team mobilized quickly
✅ Mitigation identified in < 5 minutes

## What Went Wrong

❌ No capacity planning for Black Friday
❌ Connection pool size not monitored
❌ No autoscaling for connection pool
❌ Runbook didn't cover this scenario

## Action Items

1. [P0] Increase connection pool to 5,000 (@alice, Due: Jan 21)
2. [P0] Add connection pool saturation alert (@bob, Due: Jan 21)
3. [P1] Implement autoscaling for connection pool (@charlie, Due: Jan 25)
4. [P1] Update runbook with connection pool troubleshooting (@david, Due: Jan 27)
5. [P2] Load test before major events (@team, Due: Feb 1)

## Lessons Learned

- Monitor resource saturation, not just errors
- Load test before high-traffic events
- Connection pools are a finite resource
```

***

## 8. On-Call Best Practices

### On-Call Rotation

```
Team of 6 engineers
Rotation: 1 week on-call, 5 weeks off

Primary On-Call:
- Responds to all alerts
- Pages: 5-10 per week expected
- Compensation: $500/week stipend

Secondary On-Call:
- Backup if primary unreachable
- Takes over for escalations
- Compensation: $250/week stipend

Schedule:
Week 1: Alice (primary), Bob (secondary)
Week 2: Charlie (primary), David (secondary)
Week 3: Eve (primary), Frank (secondary)
...
```

### On-Call Expectations

**Response Times**:

* SEV1: 5 minutes
* SEV2: 15 minutes
* SEV3: 1 hour

**Escalation Path**:

```
1. Primary on-call (5 min)
   ↓
2. Secondary on-call (10 min)
   ↓
3. Engineering Manager (15 min)
   ↓
4. VP Engineering (20 min)
```

**Handoff**:

```markdown theme={null}
## On-Call Handoff: Alice → Charlie

**Open Incidents**:
- INC-1234: Database latency spikes (SEV3, investigating)

**Ongoing Issues**:
- Redis cache hit rate dropped to 60% (normal: 90%)
- Monitoring this, no action needed yet

**Upcoming Maintenance**:
- SQL failover test scheduled for Wednesday 2am

**Tips**:
- If you see "connection timeout" errors, restart worker pods
- Database runbook: https://wiki.company.com/db-runbook
- Slack #oncall-help if you need guidance
```

***

## 9. Load Testing & Performance Testing

Testing your system under load is critical for understanding scalability limits and preventing outages.

### Load Testing vs Performance Testing

| Type                    | Purpose                             | When                                 | Tools                          |
| ----------------------- | ----------------------------------- | ------------------------------------ | ------------------------------ |
| **Load Testing**        | Verify system handles expected load | Before launch, before traffic spikes | k6, JMeter, Azure Load Testing |
| **Stress Testing**      | Find breaking point                 | Capacity planning                    | k6, Locust                     |
| **Spike Testing**       | Handle sudden traffic bursts        | Black Friday prep                    | k6, Gatling                    |
| **Soak Testing**        | Detect memory leaks over time       | After deployments                    | k6 (24+ hours)                 |
| **Performance Testing** | Measure response times              | Every release                        | Azure Load Testing             |

<img src="https://mintcdn.com/devweeekends/sTu6A4whRFPJo0_g/images/azure/17-k6-load-stages.svg?fit=max&auto=format&n=sTu6A4whRFPJo0_g&q=85&s=d6a00492027d0fa7805df80c098c2c35" alt="k6 Load Test Stages" width="800" height="400" data-path="images/azure/17-k6-load-stages.svg" />

***

### Azure Load Testing

**Azure Load Testing** is a fully managed service for generating high-scale load.

#### Create Load Test

```bash theme={null}
# Create Azure Load Testing resource
az load create \
  --name myloadtest \
  --resource-group rg-prod \
  --location eastus

# Upload JMeter test plan
az load test create \
  --load-test-resource myloadtest \
  --test-id checkout-load-test \
  --display-name "Checkout API Load Test" \
  --description "Test 1000 concurrent users" \
  --test-plan checkout-test.jmx \
  --engine-instances 10
```

#### JMeter Test Plan Example

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Checkout Load Test">
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments">
        <collectionProp name="Arguments.arguments">
          <elementProp name="TARGET_URL" elementType="Argument">
            <stringProp name="Argument.name">TARGET_URL</stringProp>
            <stringProp name="Argument.value">${__P(TARGET_URL,https://api.example.com)}</stringProp>
          </elementProp>
        </collectionProp>
      </elementProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Users">
        <intProp name="ThreadGroup.num_threads">1000</intProp>
        <intProp name="ThreadGroup.ramp_time">60</intProp>
        <longProp name="ThreadGroup.duration">600</longProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="POST /checkout">
          <stringProp name="HTTPSampler.domain">${TARGET_URL}</stringProp>
          <stringProp name="HTTPSampler.path">/api/checkout</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.auto_redirects">true</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
        </HTTPSamplerProxy>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>
```

***

### k6 Load Testing

**k6** is a modern, developer-friendly load testing tool.

#### Basic Load Test

```javascript theme={null}
// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // Ramp up to 100 users
    { duration: '5m', target: 100 },   // Stay at 100 users
    { duration: '2m', target: 200 },   // Spike to 200 users
    { duration: '5m', target: 200 },   // Stay at 200 users
    { duration: '2m', target: 0 },     // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)&lt;500'],  // 95% of requests &lt; 500ms
    http_req_failed: ['rate&lt;0.01'],     // Error rate &lt; 1%
  },
};

export default function () {
  const payload = JSON.stringify({
    userId: `user${__VU}`,
    cartId: `cart${__VU}`,
    items: [{ productId: 'prod123', quantity: 2 }],
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${__ENV.API_TOKEN}`,
    },
  };

  const res = http.post('https://api.example.com/checkout', payload, params);

  check(res, {
    'status is 200': (r) => r.status === 200,
    'checkout successful': (r) => r.json('success') === true,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);  // Think time: 1 second between requests
}
```

#### Run k6 Test

```bash theme={null}
# Local test
k6 run load-test.js

# Cloud test (distributed load)
k6 cloud load-test.js

# With environment variables
k6 run -e API_TOKEN=$TOKEN load-test.js

# Output to InfluxDB + Grafana
k6 run --out influxdb=http://localhost:8086/k6 load-test.js
```

***

### Performance Testing Scenarios

#### Scenario 1: Black Friday Load Test

```javascript theme={null}
// black-friday.js
export const options = {
  scenarios: {
    // Normal traffic (1000 RPS)
    normal_traffic: {
      executor: 'constant-arrival-rate',
      rate: 1000,
      timeUnit: '1s',
      duration: '1h',
      preAllocatedVUs: 500,
      maxVUs: 2000,
    },
    // Black Friday spike (10x traffic for 30 min)
    black_friday_spike: {
      executor: 'constant-arrival-rate',
      rate: 10000,
      timeUnit: '1s',
      duration: '30m',
      startTime: '1h',  // Start after normal traffic
      preAllocatedVUs: 5000,
      maxVUs: 20000,
    },
  },
  thresholds: {
    'http_req_duration{scenario:normal_traffic}': ['p(99)&lt;1000'],
    'http_req_duration{scenario:black_friday_spike}': ['p(99)&lt;2000'],
    'http_req_failed': ['rate&lt;0.05'],  // 5% error budget during spike
  },
};
```

#### Scenario 2: Soak Test (Memory Leak Detection)

```javascript theme={null}
// soak-test.js
export const options = {
  stages: [
    { duration: '5m', target: 100 },    // Ramp up
    { duration: '24h', target: 100 },   // Run for 24 hours
    { duration: '5m', target: 0 },      // Ramp down
  ],
};

export default function () {
  http.get('https://api.example.com/products');
  sleep(3);  // 3 seconds think time
}
```

**What to Monitor**:

* Memory usage (should be flat, not increasing)
* Connection pool size (should stabilize)
* Database connections (no leaks)
* HTTP response times (should not degrade)

***

### Azure Application Insights Performance Testing

#### Custom Telemetry for Load Tests

```csharp theme={null}
// Program.cs
services.AddApplicationInsightsTelemetry(options =>
{
    options.EnableAdaptiveSampling = false;  // Disable during load tests
    options.DeveloperMode = false;
});

// LoadTestController.cs
[HttpGet("health")]
public IActionResult Health()
{
    var telemetry = new TelemetryClient();

    // Track custom metric during load test
    telemetry.TrackMetric("DatabaseConnectionPoolSize",
        GetConnectionPoolSize());

    telemetry.TrackMetric("MemoryUsageMB",
        GC.GetTotalMemory(false) / 1024 / 1024);

    return Ok(new { status = "healthy" });
}
```

#### KQL Query: Analyze Load Test Results

```kusto theme={null}
// Performance during load test
requests
| where timestamp between(datetime(2026-01-21 10:00) .. datetime(2026-01-21 11:00))
| summarize
    P50 = percentile(duration, 50),
    P95 = percentile(duration, 95),
    P99 = percentile(duration, 99),
    RequestRate = count() / (max(timestamp) - min(timestamp)) / 1s,
    FailureRate = countif(success == false) * 100.0 / count()
    by bin(timestamp, 1m)
| project timestamp, P50, P95, P99, RequestRate, FailureRate
| render timechart
```

***

### Performance Testing Best Practices

#### 1. Realistic Test Data

```javascript theme={null}
// Bad: Same user every time
const userId = 'user123';

// Good: Random users from pool
const users = ['user1', 'user2', 'user3', ...];  // 10,000 users
const userId = users[Math.floor(Math.random() * users.length)];

// Better: Unique user per VU
const userId = `user${__VU}`;
```

#### 2. Think Time (Realistic User Behavior)

```javascript theme={null}
export default function () {
  // User lands on homepage
  http.get('https://example.com/');
  sleep(2);  // User reads content

  // User searches for product
  http.get('https://example.com/search?q=laptop');
  sleep(3);  // User browses results

  // User clicks product
  http.get('https://example.com/products/laptop-123');
  sleep(5);  // User reads reviews

  // User adds to cart
  http.post('https://example.com/cart/add', { productId: 'laptop-123' });
  sleep(1);
}
```

#### 3. Connection Pooling

```javascript theme={null}
export const options = {
  batch: 20,  // Send 20 requests in parallel per VU
  batchPerHost: 6,  // Max 6 parallel connections per host (HTTP/1.1)
};
```

#### 4. Gradual Ramp-Up

```javascript theme={null}
// Bad: Instant spike (thundering herd)
export const options = {
  vus: 1000,
  duration: '10m',
};

// Good: Gradual ramp-up
export const options = {
  stages: [
    { duration: '2m', target: 100 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 1000 },
    { duration: '10m', target: 1000 },
  ],
};
```

***

### Interpreting Load Test Results

#### Metrics to Track

| Metric           | Good      | Warning       | Critical       |
| ---------------- | --------- | ------------- | -------------- |
| **P95 Latency**  | \<500ms   | 500-1000ms    | >1000ms        |
| **Error Rate**   | \<0.1%    | 0.1-1%        | >1%            |
| **Throughput**   | Meets SLO | 80% of SLO    | \<80% of SLO   |
| **CPU Usage**    | \<70%     | 70-85%        | >85%           |
| **Memory Usage** | Stable    | Slow increase | Rapid increase |

#### Bottleneck Identification

```kusto theme={null}
// Find slowest dependencies
dependencies
| where timestamp > ago(1h)
| summarize
    AvgDuration = avg(duration),
    P95Duration = percentile(duration, 95),
    Count = count()
    by target
| order by P95Duration desc
| take 10
```

**Common Bottlenecks**:

1. **Database queries** (most common) - Add indexes, optimize queries
2. **External API calls** - Add caching, circuit breakers
3. **CPU-intensive operations** - Move to background jobs
4. **Network latency** - Use CDN, co-locate services
5. **Memory allocation** - Reduce allocations, use object pooling

***

### Load Testing Checklist

Before running production load tests:

* [ ] **Test environment matches production** (same VM SKU, database tier, autoscaling rules)
* [ ] **Disable sampling** in Application Insights (100% telemetry during tests)
* [ ] **Notify team** (Slack #on-call: "Load test starting at 10 AM")
* [ ] **Monitor dashboards** open (Grafana, Application Insights Live Metrics)
* [ ] **Rollback plan ready** (know how to stop the test immediately)
* [ ] **Test data prepared** (realistic user IDs, product IDs, payment methods)
* [ ] **Success criteria defined** (P95 \<500ms, error rate \<1%, no memory leaks)
* [ ] **Baseline captured** (run test before changes, compare after)

> \[!WARNING]
> **Gotcha: Testing in Production**
> Load testing in production is risky but sometimes necessary. If you must:
>
> * Use **synthetic users** (test-user-1, test-user-2) to isolate test data
> * Route test traffic to a **separate backend pool** (blue-green)
> * Start with **5% of production load**, increase gradually
> * Have **instant rollback** ready (kill switch)
> * **Never** test payment processing or critical user actions in production

***

## 10. Production Readiness Checklist

Before launching a new service to production:

```
✅ Observability
  ✅ Metrics instrumented (Golden Signals)
  ✅ Logs centralized (Application Insights / Log Analytics)
  ✅ Distributed tracing enabled
  ✅ Dashboards created
  ✅ Alerts configured with runbooks

✅ Reliability
  ✅ SLOs defined and measured
  ✅ Error budget policy in place
  ✅ Circuit breakers implemented
  ✅ Retries with exponential backoff
  ✅ Timeouts configured
  ✅ Health checks (liveness & readiness)

✅ Scalability
  ✅ Load tested at 2x peak traffic
  ✅ Autoscaling configured
  ✅ Database connection pooling
  ✅ Caching strategy
  ✅ Rate limiting for external APIs

✅ Security
  ✅ Secrets in Key Vault (not in code)
  ✅ Managed Identity for authentication
  ✅ Network isolation (Private Endpoints)
  ✅ WAF enabled
  ✅ Security scanning in CI/CD

✅ Operational
  ✅ Runbooks documented
  ✅ On-call rotation defined
  ✅ Disaster recovery tested
  ✅ Backup and restore verified
  ✅ Chaos engineering experiments run

✅ Cost
  ✅ Cost estimation completed
  ✅ Budgets and alerts set
  ✅ Autoscaling to reduce waste
  ✅ Reserved Instances for steady load
```

***

## 10. Interview Questions

### Beginner Level

<AccordionGroup>
  <Accordion title="Q1: What is the difference between SLI, SLO, and SLA?">
    **Answer**:

    **SLI (Service Level Indicator)**:

    * What you measure
    * Example: "p95 latency = 250ms"

    **SLO (Service Level Objective)**:

    * Your internal target
    * Example: "p95 latency \< 300ms"
    * Guides engineering decisions

    **SLA (Service Level Agreement)**:

    * Customer promise with consequences
    * Example: "99.9% uptime or 10% refund"
    * Legal/financial agreement

    **Relationship**: SLI ≤ SLO ≤ SLA
  </Accordion>

  <Accordion title="Q2: What is an error budget?">
    **Answer**:

    Error budget = How much unreliability you can tolerate.

    **Calculation**:

    ```
    SLO: 99.9% availability
    Error budget: 100% - 99.9% = 0.1%

    Monthly: 30 days × 24 hours × 60 min = 43,200 min
    Error budget: 43,200 × 0.001 = 43.2 minutes downtime allowed
    ```

    **Use**:

    * Budget remaining → ship features fast
    * Budget exhausted → focus on reliability
  </Accordion>
</AccordionGroup>

### Intermediate Level

<AccordionGroup>
  <Accordion title="Q3: How do you measure and reduce toil?">
    **Answer**:

    **Measure**:

    1. Time audit: Track manual, repetitive tasks
    2. Calculate percentage of time spent on toil
    3. Goal: Keep toil \< 50% (ideally \< 30%)

    **Reduce**:

    1. Automate high-frequency tasks first
    2. Examples:
       * Manual deployments → CI/CD
       * Manual scaling → Autoscaling
       * Log searching → Centralized logging
    3. Measure improvement quarterly

    **Not Toil**:

    * Novel problem solving
    * Architecture design
    * Incident response (new incidents)
  </Accordion>

  <Accordion title="Q4: Design an on-call rotation for a team of 6">
    **Answer**:

    **Rotation**:

    * Primary on-call: 1 week
    * Secondary on-call: 1 week
    * Rotation: 6 weeks per person

    **Schedule**:

    ```
    Week 1: Alice (P), Bob (S)
    Week 2: Charlie (P), David (S)
    Week 3: Eve (P), Frank (S)
    Week 4: Bob (P), Alice (S)
    Week 5: David (P), Charlie (S)
    Week 6: Frank (P), Eve (S)
    ```

    **Compensation**:

    * Primary: \$500/week
    * Secondary: \$250/week

    **Handoff Process**:

    * Written handoff document
    * 15-minute sync call
    * Slack thread with context
  </Accordion>
</AccordionGroup>

### Advanced Level

<AccordionGroup>
  <Accordion title="Q5: Implement an error budget policy">
    **Answer**:

    ```
    Service: E-Commerce Checkout
    SLO: 99.95% availability (21.6 min downtime/month)

    Error Budget Policy:

    GREEN (> 80% budget remaining):
    - Ship features daily
    - Canary deployments (10% traffic)
    - Automated rollouts
    - Chaos experiments allowed

    YELLOW (30-80% budget remaining):
    - Ship features 2x per week
    - Extended canary period (24 hours)
    - Manual approval for risky changes
    - Pause chaos experiments

    ORANGE (10-30% budget remaining):
    - Feature freeze (critical fixes only)
    - Extended testing period
    - Manual deployments
    - Root cause analysis required

    RED (< 10% budget remaining):
    - FULL STOP on features
    - Emergency fixes only
    - Incident review with leadership
    - Mandatory postmortems
    - Focus: Pay down reliability debt

    Reset: Monthly (first day of month)

    Escalation:
    - Budget hits YELLOW → Notify team
    - Budget hits ORANGE → Notify manager
    - Budget hits RED → Notify VP Engineering
    ```
  </Accordion>

  <Accordion title="Q6: Design a chaos engineering program">
    **Answer**:

    ```text theme={null}
    **Phase 1: Foundation (Month 1-2)**
    - Set up Azure Chaos Studio
    - Define blast radius (start with dev/staging)
    - Create baseline metrics dashboard
    - Get stakeholder buy-in

    **Phase 2: Simple Experiments (Month 3-4)**
    - Pod failures (10% → 30% → 50%)
    - Network latency (100ms → 500ms → 1s)
    - CPU stress (50% → 80% → 100%)
    - Run in staging, measure impact

    **Phase 3: Production Testing (Month 5-6)**
    - Start with off-peak hours
    - Small blast radius (single AZ)
    - Gradual rollout to full production
    - Always have kill switch

    **Phase 4: GameDays (Month 7+)**
    - Quarterly exercises
    - Simulate real outages
    - Test incident response
    - Cross-team coordination

    **Metrics to Track**:
    - MTTR (Mean Time To Recovery)
    - Blast radius (how many users affected)
    - Learnings per experiment
    - Action items completed

    **Culture**:
    - Blameless
    - Learning-focused
    - Celebrate finding weaknesses
    - Share results company-wide
    ```
  </Accordion>
</AccordionGroup>

***

## 11. Key Takeaways

<CardGroup cols={2}>
  <Card title="SLOs Drive Decisions" icon="bullseye">
    Define clear SLOs. Use error budgets to balance velocity and reliability.
  </Card>

  <Card title="Eliminate Toil" icon="robot">
    Automate everything repetitive. Spend time on engineering, not firefighting.
  </Card>

  <Card title="Break Things on Purpose" icon="hammer">
    Chaos engineering builds confidence. Test in production before customers do.
  </Card>

  <Card title="Blameless Culture" icon="heart">
    Focus on systems, not people. Learn from failures without punishment.
  </Card>

  <Card title="Monitor What Matters" icon="chart-line">
    User experience > server metrics. Latency, errors, saturation.
  </Card>

  <Card title="Operational Excellence" icon="star">
    Production readiness checklist before launch. On-call with clear expectations.
  </Card>
</CardGroup>

***

## Next Steps

<Card title="Back to Course Overview" icon="arrow-left" href="/courses/azure-cloud-engineering/00-overview">
  Return to course overview or continue to the Capstone Project
</Card>

<Card title="Continue to Chapter 15" icon="arrow-right" href="/courses/azure-cloud-engineering/15-capstone-project">
  Apply everything you've learned in the enterprise e-commerce capstone
</Card>