> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Linux Internals & Kernel Mastery

> Master Linux kernel internals for infrastructure, observability, and SRE roles at top tech companies

# Linux Internals & Kernel Mastery

A comprehensive, interview-focused curriculum designed for engineers targeting **infrastructure, observability, and platform engineering roles** at companies like Datadog, Grafana Labs, Honeycomb, Chronosphere, New Relic, Splunk, Cloudflare, and FAANG infrastructure teams.

<Info>
  **Course Duration**: 16-20 weeks (self-paced)\
  **Modules**: 25+ deep-dive modules covering the complete kernel\
  **Target Roles**: Infrastructure Engineer, Platform Engineer, SRE, Observability Engineer, Kernel Developer\
  **Prerequisites**: C programming, basic Linux command line, OS fundamentals\
  **Hands-on Focus**: 60% theory, 40% practical labs with real kernel code
</Info>

***

## Why This Course?

Infrastructure and observability companies need engineers who understand Linux at its deepest levels. They build tools that:

* Monitor millions of systems in real-time
* Trace application behavior with eBPF
* Profile production workloads without overhead
* Debug issues that span kernel and user space

<CardGroup cols={2}>
  <Card title="Interview Ready" icon="user-tie">
    Deep technical discussions expected at Datadog, Grafana, Cloudflare, and similar companies
  </Card>

  <Card title="Kernel-First Approach" icon="microchip">
    Understand the kernel source code, not just APIs — essential for building observability tools
  </Card>

  <Card title="eBPF & Tracing" icon="chart-line">
    Master the technology powering modern observability: eBPF, kprobes, tracepoints
  </Card>

  <Card title="Production Systems" icon="server">
    Debug real issues with perf, ftrace, bpftrace — skills demanded in infrastructure roles
  </Card>
</CardGroup>

***

## Target Companies & Roles

| Company Type        | Example Companies                                    | Key Focus Areas                         |
| ------------------- | ---------------------------------------------------- | --------------------------------------- |
| **Observability**   | Datadog, Grafana, Honeycomb, Chronosphere, New Relic | eBPF, tracing, metrics, kernel events   |
| **Infrastructure**  | Cloudflare, Fastly, Akamai, Fly.io                   | Networking stack, performance, syscalls |
| **Cloud Platforms** | AWS, GCP, Azure infrastructure teams                 | Virtualization, containers, scheduling  |
| **Database**        | CockroachDB, TiDB, ScyllaDB, ClickHouse              | I/O, memory management, storage         |
| **FAANG Infra**     | Meta Infra, Google SRE, Netflix Platform             | Full spectrum of kernel knowledge       |

***

## Course Architecture

```
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                     LINUX INTERNALS & KERNEL MASTERY                                  │
│                          25+ Modules • 16-20 Weeks                                    │
├──────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                       │
│  TRACK 1: KERNEL FOUNDATIONS           TRACK 2: PROCESS & MEMORY                     │
│  ──────────────────────────            ────────────────────────                       │
│  □ Kernel Architecture                 □ Process Subsystem Deep Dive                 │
│  □ System Call Interface               □ Memory Management Internals                 │
│  □ Kernel Data Structures              □ Virtual Memory & Page Tables                │
│  □ Interrupts & Exceptions (NEW)       □ Signals & IPC (NEW)                         │
│  □ Synchronization Primitives (NEW)                                                  │
│                                                                                       │
│  TRACK 3: I/O & STORAGE                TRACK 4: NETWORKING STACK                     │
│  ─────────────────────                 ────────────────────────                       │
│  □ VFS & File Systems                  □ Network Stack Architecture                  │
│  □ Block Layer & I/O Scheduling        □ Socket Implementation                       │
│  □ Page Cache & Writeback              □ Packet Flow & Netfilter                     │
│                                                                                       │
│  TRACK 5: OBSERVABILITY & TRACING      TRACK 6: CONTAINERS & ISOLATION               │
│  ───────────────────────────────       ──────────────────────────────                │
│  □ eBPF Deep Dive                      □ Namespaces & Cgroups                        │
│  □ Tracing Infrastructure              □ Container Runtimes                          │
│  □ Profiling & Flame Graphs            □ Security Modules (LSM/SELinux) (NEW)        │
│                                                                                       │
│  TRACK 7: ADVANCED KERNEL (NEW)        CAPSTONE: INTERVIEW PREPARATION               │
│  ─────────────────────────────         ───────────────────────────────               │
│  □ Power & Thermal Management          □ Real Interview Questions                    │
│  □ Time Subsystem & Timers             □ System Design with Kernel                   │
│  □ Kernel Debugging Techniques         □ Debugging Scenarios                         │
│  □ Device Driver Model                 □ Hands-on Projects                           │
│                                                                                       │
└──────────────────────────────────────────────────────────────────────────────────────┘
```

***

## Track 1: Kernel Foundations

Master the fundamental architecture before diving into subsystems.

<AccordionGroup>
  <Accordion title="Module 1: Kernel Architecture" icon="sitemap">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/kernel-architecture)

    Understanding how the Linux kernel is organized.

    * Monolithic vs microkernel design — why Linux chose monolithic
    * Kernel source tree organization and navigation
    * Boot process: BIOS/UEFI → bootloader → kernel initialization
    * Kernel address space layout (KASLR, kernel/user split)
    * Kernel threads vs user threads
    * Loadable kernel modules (LKMs) — architecture and lifecycle
    * Kernel configuration and compilation

    **Interview Focus**: Explain Linux architecture choices, module loading process
  </Accordion>

  <Accordion title="Module 2: System Call Interface" icon="terminal">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/syscalls)

    The gateway between user space and kernel.

    * System call mechanism: `syscall` instruction, MSRs
    * System call table implementation and dispatching
    * Context switch on syscall entry/exit
    * vDSO (virtual dynamic shared object) optimization
    * Syscall overhead measurement and optimization
    * Adding a new system call (lab exercise)
    * seccomp-BPF: system call filtering
    * Compatibility layers: 32-bit on 64-bit, personality

    **Interview Focus**: Trace a syscall from user space to kernel, explain vDSO
  </Accordion>

  <Accordion title="Module 3: Kernel Data Structures" icon="database">
    **Duration**: 8-10 hours | [Start Learning →](/courses/linux-internals/data-structures)

    Building blocks used throughout the kernel.

    * Linked lists: `list_head`, circular doubly-linked
    * Red-black trees: `rb_tree` usage and implementation
    * Radix trees and XArray for efficient lookups
    * Hash tables in the kernel
    * Per-CPU variables and cache optimization
    * RCU (Read-Copy-Update) data structures
    * Memory allocation: kmalloc, vmalloc, slab allocator

    **Interview Focus**: Implement kernel-style linked list, explain RCU
  </Accordion>

  <Accordion title="Module 4: Interrupts & Exceptions" icon="bolt">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/interrupts-exceptions)

    How the kernel responds to hardware and software events.

    * Interrupt Descriptor Table (IDT) and interrupt vectors
    * Hardware interrupts: IRQ handling and sharing
    * Top-half and bottom-half processing
    * Softirqs: high-priority deferred work
    * Tasklets: dynamically schedulable softirqs
    * Workqueues: process context deferred work
    * Threaded IRQs and IRQF\_ONESHOT
    * Interrupt affinity and balancing (irqbalance)
    * Exception handling: page faults, general protection faults
    * NMI (Non-Maskable Interrupts) and MCE

    **Interview Focus**: Explain top/bottom half split, softirq vs workqueue
  </Accordion>

  <Accordion title="Module 5: Synchronization Primitives" icon="lock">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/synchronization)

    Concurrency control in the Linux kernel.

    * Spinlocks: basic, reader-writer, raw spinlocks
    * Mutexes vs spinlocks: when to use each
    * Semaphores and completions
    * Read-Copy-Update (RCU) deep dive
    * Sequence locks (seqlocks)
    * Per-CPU variables and preemption control
    * Memory barriers and ordering
    * Atomic operations and compare-and-swap
    * Lockdep: the kernel lock validator
    * Common deadlock patterns and avoidance

    **Interview Focus**: Explain RCU mechanics, spinlock vs mutex choice
  </Accordion>
</AccordionGroup>

***

## Track 2: Process & Memory

The two most critical subsystems in the kernel.

<AccordionGroup>
  <Accordion title="Module 4: Process Subsystem Deep Dive" icon="microchip">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/process-subsystem)

    How Linux manages processes and threads.

    * `task_struct` — the process descriptor in detail
    * Process creation: `fork()`, `vfork()`, `clone()`, `clone3()`
    * Copy-on-write implementation
    * Process termination and zombie handling
    * Thread implementation: NPTL, `futex`
    * Scheduler architecture: CFS, deadline, real-time
    * Scheduler classes and policies
    * Load balancing and CPU migration
    * NUMA-aware scheduling
    * CPU affinity and isolation (`isolcpus`, `taskset`)

    **Interview Focus**: Explain CFS implementation, clone flags, CPU isolation
  </Accordion>

  <Accordion title="Module 5: Memory Management Internals" icon="memory">
    **Duration**: 16-18 hours | [Start Learning →](/courses/linux-internals/memory-management)

    Linux memory management is complex but fascinating.

    * Physical memory organization: zones, nodes, pages
    * Buddy allocator for page allocation
    * Slab allocator: SLUB, SLAB, SLOB
    * `kmalloc` vs `vmalloc` vs `kzalloc`
    * Page tables: 4-level (5-level) page tables
    * TLB management and shootdowns
    * Virtual memory areas (VMAs) and `mm_struct`
    * Memory mapping: `mmap()` implementation
    * Huge pages: transparent and explicit
    * Memory reclaim: kswapd, direct reclaim
    * OOM killer: scoring and behavior
    * Memory cgroups and limits

    **Interview Focus**: Explain buddy allocator, TLB shootdown, OOM killer
  </Accordion>

  <Accordion title="Module 6: Virtual Memory & Address Translation" icon="arrows-alt">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/virtual-memory)

    Deep dive into address translation.

    * x86-64 virtual address space layout
    * Page table walks and hardware support
    * Page fault handling: minor, major, invalid
    * Demand paging and lazy allocation
    * Copy-on-write mechanics
    * Shared memory implementation
    * Memory-mapped I/O (MMIO)
    * IOMMU and DMA mapping
    * Memory protection keys (MPK)

    **Interview Focus**: Trace a page fault, explain copy-on-write
  </Accordion>

  <Accordion title="Module 7: Signals & IPC" icon="envelope">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/signals-ipc)

    Inter-process communication mechanisms.

    * Signal delivery and handling
    * Signal masks and blocking
    * Real-time signals vs standard signals
    * sigaction vs signal semantics
    * Pipes and FIFOs internals
    * System V IPC: message queues, semaphores, shared memory
    * POSIX IPC mechanisms
    * Unix domain sockets for IPC
    * eventfd, signalfd, timerfd
    * D-Bus and modern IPC patterns

    **Interview Focus**: Explain signal delivery, compare IPC mechanisms
  </Accordion>
</AccordionGroup>

***

## Track 3: I/O & Storage

Understanding the storage stack from applications to disks.

<AccordionGroup>
  <Accordion title="Module 7: VFS & File Systems" icon="folder-tree">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/vfs-filesystems)

    The Virtual File System layer that unifies all file systems.

    * VFS architecture: superblock, inode, dentry, file
    * File system registration and mounting
    * Path lookup: namei and dcache
    * Inode operations and file operations
    * ext4 internals: extents, journaling, delayed allocation
    * XFS: B+ trees, allocation groups, delayed logging
    * Btrfs: copy-on-write, checksumming, snapshots
    * Overlay filesystems and union mounts
    * Pseudo filesystems: procfs, sysfs, debugfs

    **Interview Focus**: Explain VFS abstraction, compare ext4 vs XFS
  </Accordion>

  <Accordion title="Module 8: Block Layer & I/O Scheduling" icon="hard-drive">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/block-layer)

    How I/O requests flow through the kernel.

    * Block device architecture
    * Bio and request structures
    * Multi-queue block layer (blk-mq)
    * I/O schedulers: mq-deadline, BFQ, kyber, none
    * Request merging and plugging
    * Write barriers and ordering
    * NVMe driver architecture
    * io\_uring: modern async I/O
    * Direct I/O vs buffered I/O
    * I/O prioritization (ionice, cgroups)

    **Interview Focus**: Explain blk-mq, io\_uring benefits, NVMe design
  </Accordion>

  <Accordion title="Module 9: Page Cache & Writeback" icon="clock-rotate-left">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/page-cache)

    Caching and write-back mechanisms.

    * Page cache organization and lookup
    * Read-ahead: adaptive and explicit
    * Dirty page tracking
    * Writeback: background and periodic
    * flusher threads and dirty ratios
    * fsync, fdatasync, sync semantics
    * Direct I/O and O\_DIRECT
    * Memory pressure and cache eviction
    * Cgroup writeback throttling

    **Interview Focus**: Explain dirty ratio tuning, fsync guarantees
  </Accordion>
</AccordionGroup>

***

## Track 4: Networking Stack

Critical for infrastructure and observability roles.

<AccordionGroup>
  <Accordion title="Module 10: Network Stack Architecture" icon="network-wired">
    **Duration**: 16-18 hours | [Start Learning →](/courses/linux-internals/network-stack)

    Linux networking from NIC to application.

    * Network stack layers in Linux
    * sk\_buff: the network buffer structure
    * Packet reception: NAPI, interrupt coalescing
    * Packet transmission: qdisc, TSO, GSO
    * Protocol handlers: L2, L3, L4
    * Routing subsystem: FIB, routing cache
    * Neighbor subsystem: ARP/NDP
    * Traffic control (tc) and queueing disciplines
    * XDP (eXpress Data Path) introduction
    * AF\_XDP for zero-copy networking

    **Interview Focus**: Trace packet flow, explain NAPI, XDP use cases
  </Accordion>

  <Accordion title="Module 11: Socket Implementation" icon="plug">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/sockets)

    How sockets work under the hood.

    * Socket architecture and socket types
    * TCP implementation: connection management
    * TCP congestion control algorithms (Cubic, BBR)
    * TCP buffer management
    * UDP implementation
    * Unix domain sockets
    * Socket options and tuning
    * SO\_REUSEPORT and load balancing
    * Kernel bypass: DPDK concepts

    **Interview Focus**: Explain TCP congestion control, socket tuning
  </Accordion>

  <Accordion title="Module 12: Packet Flow & Netfilter" icon="shield-halved">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/netfilter)

    Packet filtering and network security.

    * Netfilter hooks and chains
    * Connection tracking (conntrack)
    * iptables architecture and rule processing
    * nftables: the modern replacement
    * NAT implementation
    * eBPF for networking: TC and XDP programs
    * Network namespaces and veth pairs
    * Bridge, macvlan, ipvlan devices
    * Container networking under the hood

    **Interview Focus**: Explain conntrack, container networking
  </Accordion>
</AccordionGroup>

***

## Track 5: Observability & Tracing

The core skills for observability engineering roles.

<AccordionGroup>
  <Accordion title="Module 13: eBPF Deep Dive" icon="magnifying-glass-chart">
    **Duration**: 18-20 hours | [Start Learning →](/courses/linux-internals/ebpf)

    The technology revolutionizing observability.

    * eBPF architecture and virtual machine
    * BPF program types and attach points
    * BPF maps: hash, array, ringbuf, perf buffer
    * BPF verifier: safety guarantees
    * Helper functions and kernel integration
    * libbpf and BPF CO-RE (Compile Once, Run Everywhere)
    * Writing eBPF programs in C
    * bpftrace: high-level tracing language
    * BCC (BPF Compiler Collection) tools
    * eBPF for networking: XDP, TC
    * eBPF for security: LSM hooks
    * Production eBPF: overhead and safety

    **Interview Focus**: Write eBPF program, explain verifier, map types
  </Accordion>

  <Accordion title="Module 14: Tracing Infrastructure" icon="route">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/tracing)

    Kernel tracing mechanisms and tools.

    * Static tracepoints: how they work
    * Dynamic tracing: kprobes and uprobes
    * Function tracing: ftrace framework
    * Event tracing: trace events and filters
    * perf events subsystem
    * perf\_event\_open() interface
    * Hardware performance counters (PMU)
    * LTTng and kernel ring buffers
    * User-space tracing: USDT probes
    * Distributed tracing integration

    **Interview Focus**: Instrument kernel function, explain tracepoints vs kprobes
  </Accordion>

  <Accordion title="Module 15: Profiling & Flame Graphs" icon="fire">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/profiling)

    Performance analysis for production systems.

    * CPU profiling: sampling vs tracing
    * perf record/report/annotate
    * Flame graphs: creation and interpretation
    * Off-CPU analysis: blocking time
    * Memory profiling and leak detection
    * Cache miss analysis
    * Lock contention profiling
    * Latency analysis and histograms
    * Continuous profiling in production
    * Profiling containers and Kubernetes pods

    **Interview Focus**: Analyze flame graph, explain off-CPU profiling
  </Accordion>
</AccordionGroup>

***

## Track 6: Containers & Isolation

Understanding container technology at the kernel level.

<AccordionGroup>
  <Accordion title="Module 16: Namespaces Deep Dive" icon="boxes-stacked">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/namespaces)

    The foundation of container isolation.

    * Namespace types: mnt, uts, ipc, net, pid, user, cgroup, time
    * Creating namespaces: `clone()`, `unshare()`, `setns()`
    * PID namespace: init process, orphan reaping
    * Mount namespace: propagation, pivot\_root
    * Network namespace: veth, bridges, routing
    * User namespace: UID/GID mapping, capabilities
    * Namespace interaction with cgroups
    * Rootless containers

    **Interview Focus**: Explain namespace types, rootless container challenges
  </Accordion>

  <Accordion title="Module 17: Cgroups v1 & v2" icon="layer-group">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/cgroups)

    Resource control and accounting.

    * Cgroups v1 architecture and controllers
    * Cgroups v2 unified hierarchy
    * CPU controller: shares, quota, period
    * Memory controller: limits, OOM handling
    * I/O controller: BFQ, throttling
    * PID controller: process limits
    * Cgroup namespaces
    * systemd and cgroups integration
    * Kubernetes resource management
    * Container resource limits in practice

    **Interview Focus**: Configure resource limits, explain v1 vs v2 differences
  </Accordion>

  <Accordion title="Module 18: Container Runtimes & Security" icon="shield">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/container-runtime)

    How container runtimes work.

    * OCI runtime specification
    * runc internals: container creation flow
    * containerd architecture
    * Docker daemon and containerd relationship
    * seccomp-BPF in containers
    * Capabilities and privilege dropping
    * AppArmor and SELinux profiles
    * rootfs and overlay filesystems
    * gVisor and Kata Containers (isolation alternatives)
    * Container escape vulnerabilities

    **Interview Focus**: Explain container creation, security boundaries
  </Accordion>

  <Accordion title="Module 19: Security Modules (LSM)" icon="shield-halved">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/security-modules)

    Linux Security Modules framework and implementations.

    * LSM framework architecture and hooks
    * SELinux: type enforcement, MLS, RBAC
    * AppArmor: path-based MAC
    * Capabilities system deep dive
    * seccomp-BPF: syscall filtering
    * Integrity Measurement Architecture (IMA)
    * Extended Verification Module (EVM)
    * Landlock: unprivileged sandboxing
    * Security namespaces and user namespaces
    * Container security hardening patterns

    **Interview Focus**: Explain LSM architecture, seccomp-BPF filter design
  </Accordion>
</AccordionGroup>

***

## Track 7: Advanced Kernel Topics

Deep dives into specialized kernel subsystems.

<AccordionGroup>
  <Accordion title="Module 20: Power & Thermal Management" icon="temperature-half">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/power-management)

    Understanding how Linux manages power and thermal.

    * CPUFreq governors and scaling drivers
    * CPUIdle: idle states and menu/teo governors
    * P-states and C-states on x86
    * Thermal zones and cooling devices
    * Intel RAPL for power measurement
    * Runtime PM for device power management
    * System suspend and hibernate
    * Wake-on-LAN and wake sources
    * Power profiling with PowerTOP
    * Energy-aware scheduling (EAS)

    **Interview Focus**: Explain cpufreq/cpuidle interaction, power optimization
  </Accordion>

  <Accordion title="Module 21: Time Subsystem & Timers" icon="clock">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/time-timers)

    Timekeeping and timer infrastructure.

    * Clocksources and clock event devices
    * High-resolution timers (hrtimers)
    * Timer wheel for low-resolution timers
    * POSIX timers and interval timers
    * timerfd and polling on time
    * NOHZ (tickless) operation
    * Dynamic ticks and power savings
    * Real-time clock (RTC) subsystem
    * PTP and network time synchronization
    * Timer-related debugging

    **Interview Focus**: Explain hrtimer vs timer\_list, NOHZ operation
  </Accordion>

  <Accordion title="Module 22: Kernel Debugging Techniques" icon="bug">
    **Duration**: 12-14 hours | [Start Learning →](/courses/linux-internals/kernel-debugging)

    Tools and techniques for kernel debugging.

    * printk and dynamic debug
    * KGDB: kernel source-level debugging
    * Kdump and crash dump analysis
    * KASAN: kernel address sanitizer
    * UBSAN: undefined behavior sanitizer
    * KCSAN: concurrency sanitizer
    * KMSAN: memory sanitizer
    * ftrace for debugging
    * Kernel debugging with gdb
    * Reading and understanding oops/panic messages

    **Interview Focus**: Analyze kernel oops, explain KASAN operation
  </Accordion>

  <Accordion title="Module 23: Device Driver Model" icon="microchip">
    **Duration**: 14-16 hours | [Start Learning →](/courses/linux-internals/device-drivers)

    Understanding the Linux device model.

    * Device model: buses, devices, drivers
    * sysfs and kobjects
    * Platform devices and device tree
    * Character device drivers
    * Block device drivers
    * Network device drivers
    * PCI and USB subsystems
    * DMA mapping and coherent memory
    * Interrupt handling in drivers
    * IOMMU and device isolation

    **Interview Focus**: Explain device model, write simple char driver
  </Accordion>
</AccordionGroup>

***

## Capstone: Interview Preparation

<AccordionGroup>
  <Accordion title="Module 24: Real Interview Questions" icon="clipboard-question">
    **Duration**: 10-12 hours | [Start Learning →](/courses/linux-internals/interview-questions)

    Questions from actual infrastructure/observability interviews.

    * Datadog interview questions and solutions
    * Grafana Labs technical discussions
    * Cloudflare systems questions
    * FAANG infrastructure team questions
    * Debugging scenarios with solutions
    * System design with kernel considerations
    * Take-home assignment patterns
    * Live coding: eBPF and systems programming

    **Interview Focus**: Practice with real questions and detailed solutions
  </Accordion>

  <Accordion title="Module 25: Hands-on Projects" icon="hammer">
    **Duration**: 20-24 hours | [Start Learning →](/courses/linux-internals/projects)

    Build portfolio-worthy projects.

    * Project 1: Simple container runtime
    * Project 2: eBPF-based syscall tracer
    * Project 3: Custom I/O scheduler
    * Project 4: Network packet analyzer with XDP
    * Project 5: Memory profiler
    * Project 6: Process monitor with signals
    * Project 7: Kernel module for /proc filesystem
    * Code review and optimization
    * Writing technical documentation

    **Interview Focus**: Demonstrate practical skills with working code
  </Accordion>
</AccordionGroup>

***

## Learning Paths

<CardGroup cols={3}>
  <Card title="Observability Focus" icon="chart-line">
    **10-12 weeks**\
    Tracks 1, 2, 5 + eBPF projects\
    Focus: Interrupts, tracing, profiling\
    Target: Datadog, Grafana, Honeycomb
  </Card>

  <Card title="Infrastructure Focus" icon="server">
    **12-14 weeks**\
    Tracks 1-4, 6 + Security modules\
    Focus: Networking, containers, security\
    Target: Cloudflare, AWS, Platform teams
  </Card>

  <Card title="Complete Mastery" icon="rocket">
    **16-20 weeks**\
    All 7 tracks + all projects\
    Full kernel expertise\
    Target: Any senior infrastructure role
  </Card>
</CardGroup>

### Specialized Paths

<CardGroup cols={2}>
  <Card title="Container Runtime Developer" icon="docker">
    **8-10 weeks**\
    Modules: 1-5, 16-19, 24-25\
    Focus: Namespaces, cgroups, security\
    Target: Docker, Kubernetes, containerd teams
  </Card>

  <Card title="Performance Engineer" icon="gauge-high">
    **10-12 weeks**\
    Modules: 1-6, 8-9, 13-15, 20-22\
    Focus: Profiling, tracing, debugging\
    Target: Netflix, Performance teams
  </Card>
</CardGroup>

***

## Prerequisites & Environment Setup

<Steps>
  <Step title="Programming Skills">
    Strong C programming, comfortable reading kernel-style code
  </Step>

  <Step title="Linux Environment">
    Ubuntu 22.04+ or Fedora 38+ with root access (VM recommended for kernel experiments)
  </Step>

  <Step title="Tools Installation">
    ```bash theme={null}
    # Ubuntu/Debian
    sudo apt install linux-headers-$(uname -r) build-essential \
      bpftrace bpfcc-tools linux-tools-common linux-tools-generic \
      strace ltrace gdb perf systemtap

    # Kernel source (optional but recommended)
    sudo apt install linux-source
    ```
  </Step>

  <Step title="Recommended Resources">
    * "Linux Kernel Development" by Robert Love
    * "Understanding the Linux Kernel" by Bovet & Cesati
    * "BPF Performance Tools" by Brendan Gregg
    * kernel.org documentation
  </Step>
</Steps>

***

## What Makes This Course Different

<Note>
  **For Infrastructure & Observability Roles Specifically**

  This course focuses on what these companies actually ask:

  * **eBPF expertise**: The foundation of modern observability
  * **Performance analysis**: Skills for debugging production issues
  * **Container internals**: Understanding Kubernetes at the kernel level
  * **Networking stack**: Critical for edge/CDN and infrastructure companies
  * **Real interview prep**: Questions from actual Datadog, Grafana, Cloudflare interviews

  Unlike generic OS courses, every topic connects to practical infrastructure engineering.
</Note>

***

## Course Outcomes

By completing this course, you will be able to:

<Steps>
  <Step title="Technical Depth">
    Navigate and understand Linux kernel source code, including subsystems like scheduling, memory management, and networking
  </Step>

  <Step title="Observability Tools">
    Write eBPF programs for custom monitoring, tracing, and security enforcement
  </Step>

  <Step title="Performance Analysis">
    Profile and debug production systems using perf, ftrace, bpftrace, and kernel debugging tools
  </Step>

  <Step title="Container Expertise">
    Explain container isolation at the namespace/cgroup level and implement security policies
  </Step>

  <Step title="Concurrency Mastery">
    Understand kernel synchronization primitives (spinlocks, RCU, mutexes) and debug concurrency issues
  </Step>

  <Step title="Interview Confidence">
    Discuss Linux internals at the depth expected for senior infrastructure, observability, and SRE roles
  </Step>
</Steps>

***

## Quick Reference: All Modules

| Track   | Module | Topic                      | Duration |
| ------- | ------ | -------------------------- | -------- |
| **1**   | 1      | Kernel Architecture        | 10-12h   |
| **1**   | 2      | System Call Interface      | 12-14h   |
| **1**   | 3      | Kernel Data Structures     | 8-10h    |
| **1**   | 4      | Interrupts & Exceptions    | 12-14h   |
| **1**   | 5      | Synchronization Primitives | 14-16h   |
| **2**   | 4      | Process Subsystem          | 14-16h   |
| **2**   | 5      | Memory Management          | 16-18h   |
| **2**   | 6      | Virtual Memory             | 10-12h   |
| **2**   | 7      | Signals & IPC              | 12-14h   |
| **3**   | 7      | VFS & File Systems         | 14-16h   |
| **3**   | 8      | Block Layer & I/O          | 12-14h   |
| **3**   | 9      | Page Cache & Writeback     | 10-12h   |
| **4**   | 10     | Network Stack              | 16-18h   |
| **4**   | 11     | Socket Implementation      | 12-14h   |
| **4**   | 12     | Netfilter & Packet Flow    | 10-12h   |
| **5**   | 13     | eBPF Deep Dive             | 18-20h   |
| **5**   | 14     | Tracing Infrastructure     | 14-16h   |
| **5**   | 15     | Profiling & Flame Graphs   | 12-14h   |
| **6**   | 16     | Namespaces                 | 12-14h   |
| **6**   | 17     | Cgroups v1 & v2            | 12-14h   |
| **6**   | 18     | Container Runtimes         | 10-12h   |
| **6**   | 19     | Security Modules (LSM)     | 14-16h   |
| **7**   | 20     | Power & Thermal            | 10-12h   |
| **7**   | 21     | Time & Timers              | 10-12h   |
| **7**   | 22     | Kernel Debugging           | 12-14h   |
| **7**   | 23     | Device Driver Model        | 14-16h   |
| **Cap** | 24     | Interview Questions        | 10-12h   |
| **Cap** | 25     | Hands-on Projects          | 20-24h   |

**Total Estimated Time**: 300-350 hours

***

## Interview Deep-Dive

<AccordionGroup>
  <Accordion title="If you were interviewing at Datadog for an infrastructure engineering role, how would you explain the value of understanding Linux internals for building an observability product? Be specific about which subsystems matter most." icon="message">
    **Strong Answer:**

    * Observability products like Datadog's agent need to collect metrics, traces, and logs from every layer of the system with minimal overhead. This requires deep kernel knowledge in three critical areas.
    * First, eBPF and tracing: the Datadog agent uses eBPF programs to attach to kernel tracepoints and kprobes for syscall tracing, network flow monitoring, and security event detection. Understanding the BPF verifier, map types, and program lifecycle is essential for writing production-safe programs that do not crash hosts or exceed CPU budgets.
    * Second, the process and scheduling subsystem: monitoring CPU usage, context switches, and run queue latency requires understanding `task_struct`, the CFS scheduler's vruntime accounting, and how cgroups v2 CPU controller reports throttling metrics. Without this knowledge, you cannot interpret `cpu.stat` or explain why a container shows throttling despite low average CPU usage.
    * Third, the memory subsystem: understanding the OOM killer's scoring, the difference between RSS, page cache, and slab memory, and how cgroup memory accounting works is essential for building accurate container memory monitoring and for helping customers debug OOM kills.
    * The networking stack is also critical: for network performance monitoring (NPM), understanding sk\_buff lifecycle, TCP state machines, connection tracking, and XDP gives you the ability to instrument packet flows without significant overhead.

    **Follow-up:** What is the biggest challenge in building a kernel-level monitoring agent that runs on customer machines across thousands of different kernel versions?

    **Follow-up Answer:**

    * Kernel ABI instability is the fundamental challenge. Internal struct layouts, function signatures, and tracepoint arguments change between kernel versions. A kprobe on `do_sys_open` might work on kernel 5.4 but the function was renamed to `do_sys_openat2` in 5.15. eBPF CO-RE (Compile Once, Run Everywhere) with BTF solves this for struct field accesses by adjusting offsets at load time, but it requires the target kernel to be compiled with `CONFIG_DEBUG_INFO_BTF=y`. For older kernels without BTF, you need BCC-style runtime compilation or pre-compiled binaries per kernel version. The agent must also handle graceful degradation: if a specific eBPF feature is unavailable on an older kernel, fall back to less efficient methods (perf events, procfs polling) rather than failing entirely.
  </Accordion>

  <Accordion title="You are designing the curriculum for training new infrastructure engineers at a cloud company. They have CS fundamentals but no kernel experience. What three Linux subsystems would you prioritize first and why?" icon="message">
    **Strong Answer:**

    * First priority: the process subsystem and scheduling. Every debugging session starts with understanding what processes are running, why they are slow, and how the scheduler allocates CPU time. Engineers need to understand `task_struct`, the difference between threads and processes (clone flags), CFS vruntime, and CPU affinity. This is the foundation for interpreting `top`, `perf sched`, and cgroup CPU metrics. Without it, statements like "the container is throttled" are meaningless.
    * Second priority: memory management. Memory issues are the most common category of production incidents -- OOM kills, memory leaks, swap storms, and page cache eviction. Engineers must understand virtual memory (page tables, TLB), physical memory (zones, buddy allocator), the page cache's role in file I/O, and how cgroup memory limits interact with the OOM killer. This knowledge is needed to answer "why was my container killed?" and "why is my application slow after running for a week?"
    * Third priority: the system call interface and tracing. This is the bridge between user-space application behavior and kernel internals. Understanding syscall overhead, strace interpretation, and eBPF-based tracing gives engineers the ability to diagnose any problem by following the execution path from application code through libc into the kernel. Once engineers can trace a `read()` call from user space through VFS to the block layer, they can debug almost anything.
    * I would defer networking stack internals and advanced topics like synchronization primitives until the second phase, because they build on the foundations above.

    **Follow-up:** How would you assess whether an engineer has truly internalized this knowledge versus memorized it?

    **Follow-up Answer:**

    * I would give them a live debugging scenario on a test system: a container with an application experiencing intermittent latency spikes. An engineer who has memorized facts will check a checklist of commands. An engineer who understands the internals will form hypotheses ("this could be CPU throttling, memory reclaim, or I/O stalls"), select diagnostic tools that test each hypothesis (check `cpu.stat` for throttling, `vmstat` for reclaim, `biolatency` for I/O), interpret the results in context, and iterate. The key signal is whether they can explain why their diagnosis makes sense at the kernel level -- "the throttle count is high because CFS bandwidth enforcement dequeues tasks from the runqueue when quota is exhausted" versus "the throttle number is high so we need more CPU."
  </Accordion>

  <Accordion title="Compare the learning path for someone targeting an observability role at Grafana versus an infrastructure role at Cloudflare. Which kernel subsystems matter most for each?" icon="message">
    **Strong Answer:**

    * For Grafana (observability focus): the critical path is eBPF, tracing infrastructure, and the process/memory subsystems. Grafana's products (Grafana Agent, Pyroscope, Tempo) collect and analyze system-level telemetry. Engineers need deep eBPF expertise (program types, maps, verifier constraints, CO-RE for portability), profiling skills (perf record, flame graph generation, on-CPU vs off-CPU analysis), and understanding of kernel metrics sources (procfs, sysfs, cgroup files). The networking stack matters but primarily from an observability perspective (tracing TCP connections, not implementing protocols).
    * For Cloudflare (infrastructure/networking focus): the critical path is the networking stack, XDP/eBPF for packet processing, and security modules. Cloudflare operates one of the largest edge networks and handles DDoS mitigation, load balancing, and packet filtering at massive scale. Engineers need deep understanding of the packet receive/transmit path, NAPI, sk\_buff operations, netfilter/conntrack, and XDP programming. They also need to understand TCP congestion control (BBR vs CUBIC), socket tuning parameters, and RSS/RPS for multi-core packet distribution. Container internals matter less because Cloudflare's edge does not run customer containers in the traditional Kubernetes sense.
    * Shared requirements for both: strong syscall knowledge (the universal debugging tool), basic process and memory subsystem understanding, and the ability to read kernel source code. Both roles require production debugging skills with perf and bpftrace.

    **Follow-up:** If you had only 8 weeks to prepare for either role, what would your daily study plan look like?

    **Follow-up Answer:**

    * Weeks 1-2: kernel architecture, syscall mechanism, and process subsystem (foundation for everything). Spend 2 hours reading, 1 hour doing labs with `/proc`, strace, and perf. Weeks 3-4: for Grafana, dive into eBPF (write three programs: syscall counter, latency tracer, memory profiler) and tracing infrastructure. For Cloudflare, dive into the networking stack (trace a packet through the kernel, set up XDP, tune TCP). Weeks 5-6: memory management and cgroups for Grafana (container metrics, OOM analysis), or XDP/TC BPF programming and netfilter for Cloudflare. Weeks 7-8: practice interviews with real debugging scenarios, review all lab exercises, and build one portfolio project demonstrating the target skills.
  </Accordion>
</AccordionGroup>

***

Ready to master Linux internals? Start with [Kernel Architecture →](/courses/linux-internals/kernel-architecture)