> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring & Observability

> Master Azure Monitor, Application Insights, Log Analytics, and KQL queries

# Monitoring & Observability

## What You'll Learn

By the end of this chapter, you'll understand:

* **What monitoring and observability mean** - The difference between knowing your app is broken vs. knowing WHY it's broken
* **The Three Pillars** - Metrics, Logs, and Traces and when to use each one
* **Azure Monitor ecosystem** - Application Insights, Log Analytics, and how they work together
* **KQL (Kusto Query Language)** - How to query logs to find problems in production
* **Distributed tracing** - How to track a single user request across 10 different microservices
* **Cost optimization** - How to avoid \$10,000/month telemetry bills (yes, this happens!)

***

## Introduction: What is Monitoring & Observability?

### Start Here if You're Completely New

**Monitoring = Knowing your application is broken**
**Observability = Knowing WHY your application is broken**

Think of it like a car:

**Monitoring (Dashboard lights)**:

* Check Engine Light ✅ (Something is wrong!)
* Temperature Gauge: HIGH ⚠️ (Engine is overheating!)
* Fuel Gauge: EMPTY ⚠️ (You're out of gas!)

**What you know**: The car has a problem
**What you DON'T know**: Why the engine is overheating (radiator leak? broken fan? low coolant?)

**Observability (Diagnostic tools)**:

* **Metrics**: Temperature readings every second (shows temperature spiked at 3:15 PM)
* **Logs**: "Coolant level low" warning at 3:10 PM → "Fan belt broke" error at 3:14 PM
* **Traces**: Complete timeline from "fan belt snapped" → "fan stopped" → "engine overheated" → "check engine light"

**Result**: You know EXACTLY what failed, when, and why!

***

## Why Observability Matters (Real-World Example)

### The \$2 Million Bug (True Story)

**Scenario**: E-commerce site during Black Friday sale

**What Happened**:

* 3:00 PM: Sales suddenly drop 90%
* 3:05 PM: CEO calls: "FIX IT NOW!"
* 3:30 PM: Still debugging...
* 4:00 PM: Finally found the bug (payment API timeout)
* **Total downtime**: 1 hour
* **Lost revenue**: \$2,000,000

**Without Observability**:

```
Step 1: Check if servers are running (10 min)
Step 2: Check database connections (15 min)
Step 3: Restart application (10 min - didn't help)
Step 4: Check payment gateway logs (20 min - found it!)
Step 5: Fix payment timeout setting (5 min)
Total time: 60 minutes ❌
```

**With Observability**:

```
Step 1: Open Application Insights dashboard (30 sec)
Step 2: See payment API latency spiked from 200ms → 5000ms (1 min)
Step 3: View distributed trace showing payment gateway timeout (2 min)
Step 4: Fix payment timeout setting (5 min)
Total time: 8.5 minutes ✅
Saved: $1.8 million in revenue
```

**Cost of observability tools**: \~\$500/month
**ROI**: 3,600x return on investment

***

## The Three Pillars of Observability (Explained Simply)

### Real-World Analogy: Investigating a Crime

**Crime Scene**: Your application crashed

**1. Metrics = Security Camera (Numbers over time)**

What you see:

* 3:14:23 PM: 100 people in store
* 3:14:25 PM: 95 people in store
* 3:14:27 PM: 0 people in store ← Everyone left suddenly!

**What it tells you**: WHEN something happened (3:14:27 PM)
**What it DOESN'T tell you**: WHY everyone left

Example metrics:

* CPU usage: 25% → 75% → 100% ← CPU spiked!
* Request rate: 1000/sec → 500/sec → 0/sec ← Requests dropped!
* Error rate: 1% → 5% → 25% ← Errors increased!

**2. Logs = Witness Statements (Discrete events)**

What you see:

```
3:14:20 PM: "Customer #1234 entered checkout"
3:14:22 PM: "Payment gateway timeout after 5 seconds"
3:14:23 PM: "Error: Cannot process payment - gateway unavailable"
3:14:25 PM: "Customer #1234 abandoned cart"
```

**What it tells you**: Exactly WHAT happened (payment gateway timed out)
**What it DOESN'T tell you**: Which service caused the timeout?

**3. Traces = Detective's Timeline (Request journey)**

What you see:

```
Request ID: abc123 (Customer #1234's checkout)

Frontend (50ms)
  ↓
API Gateway (20ms)
  ↓
Order Service (100ms)
  ↓
Payment Service (5000ms) ← BOTTLENECK!
  ↓ Timeout (payment gateway never responded)

Total time: 5,170ms (should be ~200ms)
```

**What it tells you**: WHERE the problem occurred (Payment Service → payment gateway)

***

## How Azure Monitor Works (Behind the Scenes)

### The Complete Picture

**Your Application**:

```
Web App (Frontend)
  ↓ sends telemetry
Application Insights
  ↓ stores data in
Log Analytics Workspace
  ↓ you query using
KQL (Kusto Query Language)
  ↓ creates
Dashboards & Alerts
```

**Think of it like a security system**:

1. **Sensors** (Your application with Application Insights SDK)
   * Motion sensors = Telemetry in your code
   * Automatically detect: HTTP requests, database queries, exceptions

2. **Recording device** (Log Analytics Workspace)
   * DVR that stores all footage
   * Stores: Metrics, logs, traces for 30-90 days

3. **Monitoring screen** (Azure Portal dashboards)
   * View live feeds
   * See alerts when motion detected

4. **Alert system** (Azure Monitor Alerts)
   * Calls police when break-in detected
   * Sends email/SMS when error rate > 5%

***

## Cost of Observability (Real Numbers)

### Before You Start: Understand the Costs

**Azure Monitor Pricing** (as of 2024):

**1. Data Ingestion (Getting data IN)**:

* First 5 GB/month: FREE ✅
* After 5 GB: \$2.76/GB

**2. Data Retention (Keeping data)**:

* First 31 days: FREE ✅
* After 31 days: \$0.12/GB/month

**Real-World Example: E-commerce App**

**Scenario**:

* 1 million requests/day
* Each request generates \~5 KB telemetry
* Total data: 5,000,000 KB/day = \~5 GB/day = 150 GB/month

**Cost Calculation**:

```
Monthly ingestion: 150 GB
- First 5 GB free: $0
- Remaining 145 GB × $2.76 = $400.20/month
Data retention (90 days):
- First 31 days: $0
- Days 32-90: 150 GB × $0.12 × 2 months = $36/month
Total: ~$436/month
```

**Optimization (Enable sampling at 10%)**:

```
Monthly ingestion: 15 GB (90% reduction!)
- First 5 GB free: $0
- Remaining 10 GB × $2.76 = $27.60/month
Data retention: 15 GB × $0.12 × 2 = $3.60/month
Total: ~$31/month (93% savings!)
```

**Cost for Typical Apps**:

* **Small app** (10k requests/day): \~\$5-20/month
* **Medium app** (1M requests/day): \~\$30-400/month (with/without sampling)
* **Large app** (100M requests/day): \~\$1,000-10,000/month

> \[!WARNING]
> **Gotcha: Debug Logs Cost Money!**
>
> Developers often enable debug logging in production and forget to turn it off.
>
> **Example mistake**:
>
> ```csharp theme={null}
> logger.LogDebug($"Processing order {orderId} for customer {customerId}");
> // This runs 1 million times/day = 5 GB/day = $400/month!
> ```
>
> **Fix**: Only log errors/warnings in production, use debug logs in development only.

***

## Observability vs Monitoring (The Real Difference)

**Monitoring = Pre-defined dashboards**

* You must know what to monitor ahead of time
* "I'll track CPU, memory, request rate"
* Works great for known issues

**Observability = Ask any question**

* You can investigate unknown problems
* "Show me all requests from user X that failed between 3-4 PM"
* Essential for debugging new issues

### Real-World Comparison

**Problem**: "Checkout is slow for some users"

**Monitoring approach** (limited):

```
Dashboard shows:
- Average response time: 250ms ✅ (Looks fine!)
- CPU usage: 40% ✅ (Looks fine!)
- Error rate: 0.5% ✅ (Looks fine!)

Conclusion: Everything looks normal, but users still complaining!
```

**Observability approach** (powerful):

```
KQL Query:
requests
| where name contains "checkout"
| where duration > 5000  // > 5 seconds
| summarize count() by client_City

Results:
London: 2 slow requests
Tokyo: 1,543 slow requests ← FOUND IT!

Cause: Database replica in Asia is down!
```

**Observability** lets you ask questions you didn't think of when building dashboards.

***

<img src="https://mintcdn.com/devweeekends/sTu6A4whRFPJo0_g/images/azure/09-monitoring-stack.svg?fit=max&auto=format&n=sTu6A4whRFPJo0_g&q=85&s=a958c73a6936e01a299cf7323fc22965" alt="Azure Monitoring Stack" width="1080" height="1080" data-path="images/azure/09-monitoring-stack.svg" />

> \[!TIP]
> **Jargon Alert: Observability vs Monitoring**
> **Monitoring**: Tells you the system is dead. ("CPU is 100%")
> **Observability**: Tells you *why* the system is dead. ("The database query from line 42 is hanging.")

> \[!WARNING]
> **Gotcha: Log Retention Costs**
> Log Analytics charges you to *ingest* data and to *keep* it. Storing debug logs for 365 days is expensive. Set retention to 30 days for dev/test and use "Data Export" to move old logs to Blob Storage (Archive Tier) for long-term compliance.

***

## 1. The Three Pillars of Observability

<CardGroup cols={3}>
  <Card title="Metrics" icon="gauge">
    **What**: Time-series numerical data

    Examples:

    * CPU usage: 75%
    * Request rate: 1,000/sec
    * Error rate: 2.5%
    * Response time: 250ms

    **Use**: Real-time monitoring, alerting
  </Card>

  <Card title="Logs" icon="file-lines">
    **What**: Discrete event records

    Examples:

    * "User login failed"
    * "Payment processed: \$99.99"
    * "Database connection timeout"

    **Use**: Debugging, troubleshooting
  </Card>

  <Card title="Traces" icon="route">
    **What**: Request flow across services

    Examples:

    * Frontend → API → Database
    * Total time: 450ms
    * DB query took 300ms

    **Use**: Performance analysis, bottleneck identification
  </Card>
</CardGroup>

***

## 2. Azure Monitor Components

<div className="flex justify-center my-4">
  <div className="w-full max-w-4xl">
    ```mermaid theme={null}
    graph TB
        Apps[Applications] -->|Telemetry| AI[Application Insights]
        VMs[VMs] -->|Metrics/Logs| Monitor[Azure Monitor]
        K8s[AKS] -->|Container Insights| Monitor
        DB[Databases] -->|Metrics| Monitor

        AI --> LA[Log Analytics Workspace]
        Monitor --> LA

        LA --> Query[KQL Queries]
        Query --> Dashboard[Dashboards]
        Query --> Alert[Alerts]
        Query --> Workbook[Workbooks]

        Alert --> Email[Email/SMS]
        Alert --> Webhook[Webhooks]
        Alert --> Logic[Logic Apps]

        style AI fill:#FFB900
        style LA fill:#0078D4
    ```
  </div>
</div>

***

## 3. Application Insights Deep Dive

### Enable Application Insights

<Tabs>
  <Tab title="ASP.NET Core">
    ```csharp theme={null}
    // Program.cs
    var builder = WebApplication.CreateBuilder(args);

    // This single line instruments your entire app -- it automatically captures:
    // - All incoming HTTP requests (duration, status code, URL)
    // - All outgoing HTTP calls (dependencies like SQL, Redis, APIs)
    // - Unhandled exceptions (with full stack traces)
    // - Performance counters (CPU, memory, GC pressure)
    // No manual logging required for basic observability.
    //
    // Cost impact: This generates ~1-5 KB of telemetry per request.
    // At 100K requests/day, that is ~500 MB/day or ~15 GB/month.
    // First 5 GB/month is free, remainder costs ~$28/month.
    // Enable sampling (below) to reduce this by 90% while keeping
    // statistical accuracy for most debugging scenarios.
    builder.Services.AddApplicationInsightsTelemetry();

    var app = builder.Build();
    ```

    **Install NuGet**:

    ```bash theme={null}
    dotnet add package Microsoft.ApplicationInsights.AspNetCore
    ```

    **Configuration** (appsettings.json):

    ```json theme={null}
    {
      "ApplicationInsights": {
        // Use Connection String (not Instrumentation Key alone) -- Connection Strings
        // support regional ingestion endpoints, reducing latency and ensuring
        // data residency compliance (EU data stays in EU).
        // NEVER hardcode this in source control -- use Key Vault or environment variables.
        "ConnectionString": "InstrumentationKey=xxx;IngestionEndpoint=https://xxx"
      }
    }
    ```

    **Cost Tip**: By default, Application Insights captures 100% of telemetry. For high-traffic apps (1M+ requests/day), enable adaptive sampling to automatically reduce volume while preserving statistically accurate data:

    ```csharp theme={null}
    builder.Services.AddApplicationInsightsTelemetry(options =>
    {
        // Adaptive sampling keeps enough data for accurate metrics
        // while reducing ingestion costs by 80-95% for high-traffic apps.
        options.EnableAdaptiveSampling = true;
    });
    ```
  </Tab>

  <Tab title="Node.js">
    ```javascript theme={null}
    // app.js
    const appInsights = require('applicationinsights');
    appInsights.setup('YOUR_CONNECTION_STRING')
      .setAutoDependencyCorrelation(true)
      .setAutoCollectRequests(true)
      .setAutoCollectPerformance(true, true)
      .setAutoCollectExceptions(true)
      .start();

    // Track custom events
    const client = appInsights.defaultClient;
    client.trackEvent({ name: 'OrderPlaced', properties: { orderId: '123' } });
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    # app.py
    from applicationinsights import TelemetryClient
    from applicationinsights.flask.ext import AppInsights

    app = Flask(__name__)
    app.config['APPINSIGHTS_INSTRUMENTATIONKEY'] = 'YOUR_KEY'
    appinsights = AppInsights(app)

    # Track custom events
    tc = TelemetryClient('YOUR_KEY')
    tc.track_event('OrderPlaced', {'orderId': '123'})
    tc.flush()
    ```
  </Tab>
</Tabs>

### Custom Telemetry

<AccordionGroup>
  <Accordion title="Track Custom Events" icon="tag">
    ```csharp theme={null}
    // Track business events
    telemetryClient.TrackEvent("OrderPlaced",
        properties: new Dictionary<string, string> {
            { "OrderId", orderId },
            { "CustomerId", customerId }
        },
        metrics: new Dictionary<string, double> {
            { "Amount", amount }
        });

    // Query in KQL:
    customEvents
    | where name == "OrderPlaced"
    | summarize totalRevenue=sum(todouble(customMeasurements.Amount)) by bin(timestamp, 1h)
    | render timechart
    ```
  </Accordion>

  <Accordion title="Track Dependencies" icon="plug">
    ```csharp theme={null}
    // Track external dependencies (DB, APIs, etc.)
    using (var operation = telemetryClient.StartOperation<DependencyTelemetry>("SQL Query"))
    {
        operation.Telemetry.Type = "SQL";
        operation.Telemetry.Data = "SELECT * FROM Orders WHERE CustomerId = @id";

        try
        {
            var result = await database.QueryAsync(sql);
            operation.Telemetry.Success = true;
        }
        catch (Exception ex)
        {
            operation.Telemetry.Success = false;
            telemetryClient.TrackException(ex);
            throw;
        }
    }
    ```
  </Accordion>
</AccordionGroup>

***

## 4. KQL (Kusto Query Language) Mastery

### Essential Queries for Production

<Tabs>
  <Tab title="Performance Analysis">
    ```kusto theme={null}
    // Find slowest requests
    requests
    | where timestamp > ago(24h)
    | summarize
        count=count(),
        avg_duration=avg(duration),
        p50=percentile(duration, 50),
        p95=percentile(duration, 95),
        p99=percentile(duration, 99)
        by operation_Name
    | order by p95 desc

    // Find requests slower than SLA
    requests
    | where duration > 1000  // > 1 second
    | project timestamp, name, url, duration, resultCode
    | order by duration desc
    ```
  </Tab>

  <Tab title="Error Analysis">
    ```kusto theme={null}
    // Find top errors
    requests
    | where timestamp > ago(1h)
    | where success == false
    | summarize count() by resultCode, operation_Name
    | order by count_ desc

    // Error rate over time
    requests
    | where timestamp > ago(24h)
    | summarize
        total=count(),
        failed=countif(success == false)
        by bin(timestamp, 5m)
    | extend errorRate = (failed * 100.0) / total
    | render timechart

    // Exceptions with stack traces
    exceptions
    | where timestamp > ago(1h)
    | project timestamp, type, outerMessage, innermostMessage
    | order by timestamp desc
    ```
  </Tab>

  <Tab title="Dependency Failures">
    ```kusto theme={null}
    // Failed dependencies
    dependencies
    | where timestamp > ago(1h)
    | where success == false
    | summarize count() by name, type, resultCode
    | order by count_ desc

    // Database query performance
    dependencies
    | where type == "SQL"
    | where timestamp > ago(24h)
    | summarize
        count(),
        avg(duration),
        p95=percentile(duration, 95)
        by name
    | order by p95 desc
    ```
  </Tab>

  <Tab title="User Analytics">
    ```kusto theme={null}
    // Active users
    pageViews
    | where timestamp > ago(7d)
    | summarize dau=dcount(user_Id) by bin(timestamp, 1d)
    | render timechart

    // Most popular pages
    pageViews
    | where timestamp > ago(7d)
    | summarize count() by name
    | order by count_ desc
    | take 10
    ```
  </Tab>
</Tabs>

***

## 5. Distributed Tracing

<div className="flex justify-center my-4">
  <div className="w-full max-w-4xl">
    ```mermaid theme={null}
    sequenceDiagram
        participant User
        participant Frontend
        participant API
        participant Database

        User->>Frontend: GET /checkout
        Note over Frontend: TraceId: abc123<br/>SpanId: span-1

        Frontend->>API: POST /api/orders
        Note over API: TraceId: abc123<br/>ParentSpanId: span-1<br/>SpanId: span-2

        API->>Database: INSERT INTO Orders
        Note over Database: TraceId: abc123<br/>ParentSpanId: span-2<br/>SpanId: span-3

        Database-->>API: Success
        API-->>Frontend: 201 Created
        Frontend-->>User: Order confirmation
    ```
  </div>
</div>

**Application Insights automatically correlates** using `traceparent` header.

***

## 6. Alerting Strategy

<Tabs>
  <Tab title="Metric Alerts">
    ```bash theme={null}
    # CPU alert
    az monitor metrics alert create \
      --name high-cpu-alert \
      --resource-group rg-prod \
      --scopes /subscriptions/.../virtualMachines/vm-web-01 \
      --condition "avg Percentage CPU > 80" \
      --window-size 5m \
      --evaluation-frequency 1m \
      --action email-action-group
    ```
  </Tab>

  <Tab title="Log Query Alerts">
    ```kusto theme={null}
    // High error rate alert
    requests
    | where timestamp > ago(5m)
    | summarize
        total=count(),
        failed=countif(success == false)
    | extend errorRate = (failed * 100.0) / total
    | where errorRate > 5  // Alert if > 5% errors
    ```
  </Tab>

  <Tab title="Smart Detection">
    Application Insights automatically detects:

    * Abnormal rise in failure rate
    * Performance degradation
    * Memory leak detection
    * Security anomalies

    **No configuration required** - enabled by default!
  </Tab>
</Tabs>

***

## 7. Interview Questions

### Beginner Level

<AccordionGroup>
  <Accordion title="Q1: What's the difference between metrics and logs?">
    **Answer**:

    **Metrics**:

    * Numerical time-series data (CPU: 75%, requests: 1000/sec)
    * Cheap to store (aggregated)
    * Real-time monitoring
    * Limited context

    **Logs**:

    * Discrete event records (structured or unstructured)
    * Expensive to store (high volume)
    * Rich context and details
    * Used for debugging

    **Example**: Metrics tell you "error rate is 5%", logs tell you "User X's payment failed because of timeout"
  </Accordion>

  <Accordion title="Q2: What are the three pillars of observability?">
    **Answer**:

    1. **Metrics**: Numerical measurements over time (CPU, memory, request rate)
    2. **Logs**: Discrete event records (errors, audit trails)
    3. **Traces**: Request flow across distributed systems

    All three are needed for complete observability. Metrics show *what* is wrong, logs show *why*, and traces show *where* in the system.
  </Accordion>

  <Accordion title="Q3: What is Application Insights?">
    **Answer**:

    Application Insights is Azure's Application Performance Monitoring (APM) service.

    **Features**:

    * Automatic telemetry collection (requests, dependencies, exceptions)
    * Distributed tracing
    * Application Map (visualize dependencies)
    * Live Metrics Stream (real-time monitoring)
    * Smart Detection (anomaly detection)

    **Use case**: Monitor web applications, detect performance issues, track user behavior
  </Accordion>
</AccordionGroup>

### Intermediate Level

<AccordionGroup>
  <Accordion title="Q4: How would you troubleshoot a slow API endpoint?">
    **Answer**:

    **Step-by-step approach**:

    1. **Identify the slow endpoint**:

    ```kusto theme={null}
    requests
    | where timestamp > ago(1h)
    | summarize p95=percentile(duration, 95) by operation_Name
    | order by p95 desc
    ```

    2. **Find slow dependencies**:

    ```kusto theme={null}
    dependencies
    | where operation_Name == "/api/orders"
    | summarize p95=percentile(duration, 95) by name
    | order by p95 desc
    ```

    3. **View end-to-end transaction**:
       Use Application Map or search by `operation_Id` to see the entire request flow
  </Accordion>

  <Accordion title="Q5: Explain distributed tracing and correlation">
    **Answer**:

    **Distributed Tracing** tracks a single user request as it flows through multiple services.

    **How it works**:

    1. Generate `TraceId` (unique per request)
    2. Each service creates a `SpanId` (unique per operation)
    3. Pass `TraceId` and parent `SpanId` via HTTP headers (`traceparent`)
    4. All telemetry includes `TraceId` for correlation

    **Query all operations in a trace**:

    ```kusto theme={null}
    union requests, dependencies
    | where operation_Id == "abc123"
    | project timestamp, itemType, name, duration
    | order by timestamp asc
    ```
  </Accordion>
</AccordionGroup>

### Advanced Level

<AccordionGroup>
  <Accordion title="Q6: Design a monitoring strategy for microservices">
    **Answer**:

    **1. Instrumentation**:

    * Enable Application Insights on all services
    * Implement distributed tracing
    * Use structured logging (JSON)

    **2. Metrics**:

    * **Service-level**: Request rate, error rate, latency (p50, p95, p99)
    * **Infrastructure**: CPU, memory, disk, network
    * **Business**: Orders/min, revenue/hour, conversion rate

    **3. Dashboards**:

    * **Overview**: Health of all services (green/yellow/red)
    * **Service Detail**: Golden signals per service
    * **Business**: KPIs (revenue, active users, conversion)

    **4. Alerts**:

    * **Critical**: Service down, high error rate (> 5%)
    * **Warning**: Degraded performance, resource usage > 80%

    **5. On-Call Runbooks**:

    * Document troubleshooting steps for each alert
    * Include dashboard links, KQL queries
    * Escalation paths
  </Accordion>

  <Accordion title="Q7: How do you optimize telemetry costs?">
    **Answer**:

    **Optimization Strategies**:

    **1. Enable Sampling**:

    ```csharp theme={null}
    services.AddApplicationInsightsTelemetry(options =>
    {
        options.EnableAdaptiveSampling = true;
        options.SamplingPercentage = 10; // Reduce by 90%
    });
    ```

    **2. Filter Unnecessary Telemetry**:

    ```csharp theme={null}
    // Don't send health check requests
    public class FilterHealthCheckProcessor : ITelemetryProcessor
    {
        public void Process(ITelemetry item)
        {
            if (item is RequestTelemetry request &&
                request.Url.AbsolutePath == "/health")
            {
                return; // Skip
            }
            _next.Process(item);
        }
    }
    ```

    **3. Reduce Retention**:

    ```bash theme={null}
    # Set retention to 30 days (vs 90 default)
    az monitor app-insights component update \
      --app myapp \
      --retention-time 30
    ```

    **Expected Savings**: 70-90% reduction
  </Accordion>
</AccordionGroup>

***

## 8. Best Practices

<CardGroup cols={2}>
  <Card title="Structured Logging" icon="list">
    Use structured logs (JSON) for easier querying. Include context (user ID, correlation ID).
  </Card>

  <Card title="Correlation IDs" icon="link">
    Track requests across services with operation\_Id. Essential for distributed tracing.
  </Card>

  <Card title="Sample High-Volume Data" icon="filter">
    Enable adaptive sampling for high-traffic apps to control costs while preserving insights.
  </Card>

  <Card title="Monitor SLIs/SLOs" icon="bullseye">
    Define Service Level Indicators (latency, error rate) and Objectives (99.9% uptime).
  </Card>

  <Card title="Alert Runbooks" icon="book">
    Every alert needs a runbook: What it means, how to troubleshoot, escalation path.
  </Card>

  <Card title="Test Observability" icon="vial">
    Regularly test your monitoring: Can you detect and diagnose issues quickly?
  </Card>
</CardGroup>

***

## 9. Key Takeaways

<CardGroup cols={2}>
  <Card title="Three Pillars" icon="columns">
    Metrics, Logs, and Traces work together for complete observability.
  </Card>

  <Card title="Distributed Tracing" icon="route">
    Application Insights automatically traces requests across services using correlation IDs.
  </Card>

  <Card title="KQL is Essential" icon="code">
    Master KQL for querying logs, creating dashboards, and building alerts.
  </Card>

  <Card title="Golden Signals" icon="star">
    Monitor Latency, Traffic, Errors, and Saturation for system health.
  </Card>

  <Card title="Smart Alerts" icon="bell">
    Alert on symptoms (error rate), not causes (CPU). Reduce alert fatigue.
  </Card>

  <Card title="Cost Optimization" icon="dollar-sign">
    Use sampling, filter noise, and reduce retention to control telemetry costs.
  </Card>
</CardGroup>

***

## Interview Deep-Dive

<AccordionGroup>
  <Accordion title="Checkout conversion drops 40% but no alerts fire. CPU, memory, and error rates look normal. How do you diagnose this?">
    **Strong Candidate Answer:**

    * **Why monitoring missed it:** CPU, memory, and 500 error rates are infrastructure metrics. A conversion drop is a business metric -- the app is technically working but something is wrong with user experience.
    * **Diagnosis:** Open Application Map for dependency latency. Check Performance blade for /checkout P95 response time -- if it jumped from 800ms to 4 seconds, users abandon due to slowness (200 OK response, but slow). Use distributed tracing to find which dependency call is the bottleneck. Check Failures blade for dependency 429s (rate limiting) causing retries.
    * **The likely culprit:** A third-party dependency (payment gateway, fraud detection) responding slowly. Application Insights dependency tracking shows exact call, latency, and success rate.
    * **What was missing:** No alerts on business KPIs. Create custom metrics tracking checkout start vs order confirmation events. Alert when conversion drops 20% vs same hour last week.

    **Follow-up: How do you alert on business problems without alert fatigue?**

    Use anomaly-based alerting (Smart Detection) instead of static thresholds. Dynamic threshold alerts compare current values against 7-day rolling averages. Route infrastructure alerts to platform team, business metrics to product team. Implement suppression during maintenance windows.
  </Accordion>

  <Accordion title="Your Application Insights bill jumped from $500 to $8,000/month. How do you reduce it without losing critical observability?">
    **Strong Candidate Answer:**

    * **What happened:** Ingestion went from \~185 GB to \~2,900 GB/month at \$2.76/GB. Common causes: verbose DEBUG logging in production, trace telemetry for every request, dependency tracking logging full SQL queries, or snapshot debugging enabled.
    * **Immediate fix:** Enable adaptive sampling (1-in-5 requests, 80% cost reduction while preserving statistical accuracy). Set daily cap to 200 GB to prevent runaway costs.
    * **Medium-term:** Use "Usage and estimated costs" blade to find which telemetry type is largest. Filter out health check endpoints with TelemetryProcessors. Move verbose logs to separate workspace with 7-day retention.
    * **Never cut:** Distributed tracing and exception tracking. A 1-hour outage costs \$50K+ for most e-commerce sites -- far more than monthly telemetry costs.

    **Follow-up: Compare Application Insights pricing with Datadog and New Relic for 50 microservices.**

    At 500 GB/month: Application Insights \~$1,400 (commitment tier), Datadog ~$2,050 ($31/host + logs), New Relic ~$1,130 ($0.30/GB + $49/user for 20 engineers). New Relic cheapest per-GB, Datadog best dashboarding, Application Insights best Azure integration. Real differentiator is team skills: if engineers know KQL, Application Insights is fastest to value.
  </Accordion>

  <Accordion title="A user says checkout is slow but your API returns 200 OK in 300ms. Where is the problem?">
    **Strong Candidate Answer:**

    * **The disconnect:** Server measures 300ms. User experiences: DNS (50ms) + TCP (100ms) + TLS (100ms) + TTFB (300ms) + download (200ms) + JS rendering (2s) + third-party scripts (analytics, chat). Total perceived: 2.75 seconds.
    * **How distributed tracing reveals this:** Application Insights JavaScript SDK captures browser timing. End-to-end trace shows: Browser (2.75s) -> CDN (50ms) -> Front Door (20ms) -> API (300ms). 2 seconds are client-side rendering and third-party scripts.
    * **The fix:** Defer non-critical third-party scripts after checkout. Lazy-load below-fold content. Pre-connect to payment API domain. Reduces perceived time from 2.75s to under 1 second without backend changes.

    **Follow-up: How do you implement distributed tracing across polyglot microservices (C#, Node.js, Python)?**

    Use OpenTelemetry as the instrumentation standard. Each service exports traces to Application Insights via Azure Monitor OTel exporter. W3C Trace Context headers propagate automatically across HTTP calls regardless of language. Unified traces in one Application Insights instance across all services. OpenTelemetry is vendor-neutral, avoiding lock-in to Azure-specific SDKs.
  </Accordion>
</AccordionGroup>

***

## Next Steps

<Card title="Continue to Chapter 10" icon="arrow-right" href="/courses/azure-cloud-engineering/10-security-compliance">
  Master Azure security, compliance, and governance
</Card>