Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Linux Internals & Kernel Mastery

A comprehensive, interview-focused curriculum designed for engineers targeting infrastructure, observability, and platform engineering roles at companies like Datadog, Grafana Labs, Honeycomb, Chronosphere, New Relic, Splunk, Cloudflare, and FAANG infrastructure teams.
Course Duration: 16-20 weeks (self-paced)
Modules: 25+ deep-dive modules covering the complete kernel
Target Roles: Infrastructure Engineer, Platform Engineer, SRE, Observability Engineer, Kernel Developer
Prerequisites: C programming, basic Linux command line, OS fundamentals
Hands-on Focus: 60% theory, 40% practical labs with real kernel code

​
Why This Course?

Infrastructure and observability companies need engineers who understand Linux at its deepest levels. They build tools that:
  • Monitor millions of systems in real-time
  • Trace application behavior with eBPF
  • Profile production workloads without overhead
  • Debug issues that span kernel and user space

Interview Ready

Deep technical discussions expected at Datadog, Grafana, Cloudflare, and similar companies

Kernel-First Approach

Understand the kernel source code, not just APIs — essential for building observability tools

eBPF & Tracing

Master the technology powering modern observability: eBPF, kprobes, tracepoints

Production Systems

Debug real issues with perf, ftrace, bpftrace — skills demanded in infrastructure roles

​
Target Companies & Roles

Company TypeExample CompaniesKey Focus Areas
ObservabilityDatadog, Grafana, Honeycomb, Chronosphere, New ReliceBPF, tracing, metrics, kernel events
InfrastructureCloudflare, Fastly, Akamai, Fly.ioNetworking stack, performance, syscalls
Cloud PlatformsAWS, GCP, Azure infrastructure teamsVirtualization, containers, scheduling
DatabaseCockroachDB, TiDB, ScyllaDB, ClickHouseI/O, memory management, storage
FAANG InfraMeta Infra, Google SRE, Netflix PlatformFull spectrum of kernel knowledge

​
Course Architecture

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                     LINUX INTERNALS & KERNEL MASTERY                                  │
│                          25+ Modules • 16-20 Weeks                                    │
├──────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                       │
│  TRACK 1: KERNEL FOUNDATIONS           TRACK 2: PROCESS & MEMORY                     │
│  ──────────────────────────            ────────────────────────                       │
│  □ Kernel Architecture                 □ Process Subsystem Deep Dive                 │
│  □ System Call Interface               □ Memory Management Internals                 │
│  □ Kernel Data Structures              □ Virtual Memory & Page Tables                │
│  □ Interrupts & Exceptions (NEW)       □ Signals & IPC (NEW)                         │
│  □ Synchronization Primitives (NEW)                                                  │
│                                                                                       │
│  TRACK 3: I/O & STORAGE                TRACK 4: NETWORKING STACK                     │
│  ─────────────────────                 ────────────────────────                       │
│  □ VFS & File Systems                  □ Network Stack Architecture                  │
│  □ Block Layer & I/O Scheduling        □ Socket Implementation                       │
│  □ Page Cache & Writeback              □ Packet Flow & Netfilter                     │
│                                                                                       │
│  TRACK 5: OBSERVABILITY & TRACING      TRACK 6: CONTAINERS & ISOLATION               │
│  ───────────────────────────────       ──────────────────────────────                │
│  □ eBPF Deep Dive                      □ Namespaces & Cgroups                        │
│  □ Tracing Infrastructure              □ Container Runtimes                          │
│  □ Profiling & Flame Graphs            □ Security Modules (LSM/SELinux) (NEW)        │
│                                                                                       │
│  TRACK 7: ADVANCED KERNEL (NEW)        CAPSTONE: INTERVIEW PREPARATION               │
│  ─────────────────────────────         ───────────────────────────────               │
│  □ Power & Thermal Management          □ Real Interview Questions                    │
│  □ Time Subsystem & Timers             □ System Design with Kernel                   │
│  □ Kernel Debugging Techniques         □ Debugging Scenarios                         │
│  □ Device Driver Model                 □ Hands-on Projects                           │
│                                                                                       │
└──────────────────────────────────────────────────────────────────────────────────────┘

​
Track 1: Kernel Foundations

Master the fundamental architecture before diving into subsystems.
Duration: 10-12 hours | Start Learning →Understanding how the Linux kernel is organized.
  • Monolithic vs microkernel design — why Linux chose monolithic
  • Kernel source tree organization and navigation
  • Boot process: BIOS/UEFI → bootloader → kernel initialization
  • Kernel address space layout (KASLR, kernel/user split)
  • Kernel threads vs user threads
  • Loadable kernel modules (LKMs) — architecture and lifecycle
  • Kernel configuration and compilation
Interview Focus: Explain Linux architecture choices, module loading process
Duration: 12-14 hours | Start Learning →The gateway between user space and kernel.
  • System call mechanism: syscall instruction, MSRs
  • System call table implementation and dispatching
  • Context switch on syscall entry/exit
  • vDSO (virtual dynamic shared object) optimization
  • Syscall overhead measurement and optimization
  • Adding a new system call (lab exercise)
  • seccomp-BPF: system call filtering
  • Compatibility layers: 32-bit on 64-bit, personality
Interview Focus: Trace a syscall from user space to kernel, explain vDSO
Duration: 8-10 hours | Start Learning →Building blocks used throughout the kernel.
  • Linked lists: list_head, circular doubly-linked
  • Red-black trees: rb_tree usage and implementation
  • Radix trees and XArray for efficient lookups
  • Hash tables in the kernel
  • Per-CPU variables and cache optimization
  • RCU (Read-Copy-Update) data structures
  • Memory allocation: kmalloc, vmalloc, slab allocator
Interview Focus: Implement kernel-style linked list, explain RCU
Duration: 12-14 hours | Start Learning →How the kernel responds to hardware and software events.
  • Interrupt Descriptor Table (IDT) and interrupt vectors
  • Hardware interrupts: IRQ handling and sharing
  • Top-half and bottom-half processing
  • Softirqs: high-priority deferred work
  • Tasklets: dynamically schedulable softirqs
  • Workqueues: process context deferred work
  • Threaded IRQs and IRQF_ONESHOT
  • Interrupt affinity and balancing (irqbalance)
  • Exception handling: page faults, general protection faults
  • NMI (Non-Maskable Interrupts) and MCE
Interview Focus: Explain top/bottom half split, softirq vs workqueue
Duration: 14-16 hours | Start Learning →Concurrency control in the Linux kernel.
  • Spinlocks: basic, reader-writer, raw spinlocks
  • Mutexes vs spinlocks: when to use each
  • Semaphores and completions
  • Read-Copy-Update (RCU) deep dive
  • Sequence locks (seqlocks)
  • Per-CPU variables and preemption control
  • Memory barriers and ordering
  • Atomic operations and compare-and-swap
  • Lockdep: the kernel lock validator
  • Common deadlock patterns and avoidance
Interview Focus: Explain RCU mechanics, spinlock vs mutex choice

​
Track 2: Process & Memory

The two most critical subsystems in the kernel.
Duration: 14-16 hours | Start Learning →How Linux manages processes and threads.
  • task_struct — the process descriptor in detail
  • Process creation: fork(), vfork(), clone(), clone3()
  • Copy-on-write implementation
  • Process termination and zombie handling
  • Thread implementation: NPTL, futex
  • Scheduler architecture: CFS, deadline, real-time
  • Scheduler classes and policies
  • Load balancing and CPU migration
  • NUMA-aware scheduling
  • CPU affinity and isolation (isolcpus, taskset)
Interview Focus: Explain CFS implementation, clone flags, CPU isolation
Duration: 16-18 hours | Start Learning →Linux memory management is complex but fascinating.
  • Physical memory organization: zones, nodes, pages
  • Buddy allocator for page allocation
  • Slab allocator: SLUB, SLAB, SLOB
  • kmalloc vs vmalloc vs kzalloc
  • Page tables: 4-level (5-level) page tables
  • TLB management and shootdowns
  • Virtual memory areas (VMAs) and mm_struct
  • Memory mapping: mmap() implementation
  • Huge pages: transparent and explicit
  • Memory reclaim: kswapd, direct reclaim
  • OOM killer: scoring and behavior
  • Memory cgroups and limits
Interview Focus: Explain buddy allocator, TLB shootdown, OOM killer
Duration: 10-12 hours | Start Learning →Deep dive into address translation.
  • x86-64 virtual address space layout
  • Page table walks and hardware support
  • Page fault handling: minor, major, invalid
  • Demand paging and lazy allocation
  • Copy-on-write mechanics
  • Shared memory implementation
  • Memory-mapped I/O (MMIO)
  • IOMMU and DMA mapping
  • Memory protection keys (MPK)
Interview Focus: Trace a page fault, explain copy-on-write
Duration: 12-14 hours | Start Learning →Inter-process communication mechanisms.
  • Signal delivery and handling
  • Signal masks and blocking
  • Real-time signals vs standard signals
  • sigaction vs signal semantics
  • Pipes and FIFOs internals
  • System V IPC: message queues, semaphores, shared memory
  • POSIX IPC mechanisms
  • Unix domain sockets for IPC
  • eventfd, signalfd, timerfd
  • D-Bus and modern IPC patterns
Interview Focus: Explain signal delivery, compare IPC mechanisms

​
Track 3: I/O & Storage

Understanding the storage stack from applications to disks.
Duration: 14-16 hours | Start Learning →The Virtual File System layer that unifies all file systems.
  • VFS architecture: superblock, inode, dentry, file
  • File system registration and mounting
  • Path lookup: namei and dcache
  • Inode operations and file operations
  • ext4 internals: extents, journaling, delayed allocation
  • XFS: B+ trees, allocation groups, delayed logging
  • Btrfs: copy-on-write, checksumming, snapshots
  • Overlay filesystems and union mounts
  • Pseudo filesystems: procfs, sysfs, debugfs
Interview Focus: Explain VFS abstraction, compare ext4 vs XFS
Duration: 12-14 hours | Start Learning →How I/O requests flow through the kernel.
  • Block device architecture
  • Bio and request structures
  • Multi-queue block layer (blk-mq)
  • I/O schedulers: mq-deadline, BFQ, kyber, none
  • Request merging and plugging
  • Write barriers and ordering
  • NVMe driver architecture
  • io_uring: modern async I/O
  • Direct I/O vs buffered I/O
  • I/O prioritization (ionice, cgroups)
Interview Focus: Explain blk-mq, io_uring benefits, NVMe design
Duration: 10-12 hours | Start Learning →Caching and write-back mechanisms.
  • Page cache organization and lookup
  • Read-ahead: adaptive and explicit
  • Dirty page tracking
  • Writeback: background and periodic
  • flusher threads and dirty ratios
  • fsync, fdatasync, sync semantics
  • Direct I/O and O_DIRECT
  • Memory pressure and cache eviction
  • Cgroup writeback throttling
Interview Focus: Explain dirty ratio tuning, fsync guarantees

​
Track 4: Networking Stack

Critical for infrastructure and observability roles.
Duration: 16-18 hours | Start Learning →Linux networking from NIC to application.
  • Network stack layers in Linux
  • sk_buff: the network buffer structure
  • Packet reception: NAPI, interrupt coalescing
  • Packet transmission: qdisc, TSO, GSO
  • Protocol handlers: L2, L3, L4
  • Routing subsystem: FIB, routing cache
  • Neighbor subsystem: ARP/NDP
  • Traffic control (tc) and queueing disciplines
  • XDP (eXpress Data Path) introduction
  • AF_XDP for zero-copy networking
Interview Focus: Trace packet flow, explain NAPI, XDP use cases
Duration: 12-14 hours | Start Learning →How sockets work under the hood.
  • Socket architecture and socket types
  • TCP implementation: connection management
  • TCP congestion control algorithms (Cubic, BBR)
  • TCP buffer management
  • UDP implementation
  • Unix domain sockets
  • Socket options and tuning
  • SO_REUSEPORT and load balancing
  • Kernel bypass: DPDK concepts
Interview Focus: Explain TCP congestion control, socket tuning
Duration: 10-12 hours | Start Learning →Packet filtering and network security.
  • Netfilter hooks and chains
  • Connection tracking (conntrack)
  • iptables architecture and rule processing
  • nftables: the modern replacement
  • NAT implementation
  • eBPF for networking: TC and XDP programs
  • Network namespaces and veth pairs
  • Bridge, macvlan, ipvlan devices
  • Container networking under the hood
Interview Focus: Explain conntrack, container networking

​
Track 5: Observability & Tracing

The core skills for observability engineering roles.
Duration: 18-20 hours | Start Learning →The technology revolutionizing observability.
  • eBPF architecture and virtual machine
  • BPF program types and attach points
  • BPF maps: hash, array, ringbuf, perf buffer
  • BPF verifier: safety guarantees
  • Helper functions and kernel integration
  • libbpf and BPF CO-RE (Compile Once, Run Everywhere)
  • Writing eBPF programs in C
  • bpftrace: high-level tracing language
  • BCC (BPF Compiler Collection) tools
  • eBPF for networking: XDP, TC
  • eBPF for security: LSM hooks
  • Production eBPF: overhead and safety
Interview Focus: Write eBPF program, explain verifier, map types
Duration: 14-16 hours | Start Learning →Kernel tracing mechanisms and tools.
  • Static tracepoints: how they work
  • Dynamic tracing: kprobes and uprobes
  • Function tracing: ftrace framework
  • Event tracing: trace events and filters
  • perf events subsystem
  • perf_event_open() interface
  • Hardware performance counters (PMU)
  • LTTng and kernel ring buffers
  • User-space tracing: USDT probes
  • Distributed tracing integration
Interview Focus: Instrument kernel function, explain tracepoints vs kprobes
Duration: 12-14 hours | Start Learning →Performance analysis for production systems.
  • CPU profiling: sampling vs tracing
  • perf record/report/annotate
  • Flame graphs: creation and interpretation
  • Off-CPU analysis: blocking time
  • Memory profiling and leak detection
  • Cache miss analysis
  • Lock contention profiling
  • Latency analysis and histograms
  • Continuous profiling in production
  • Profiling containers and Kubernetes pods
Interview Focus: Analyze flame graph, explain off-CPU profiling

​
Track 6: Containers & Isolation

Understanding container technology at the kernel level.
Duration: 12-14 hours | Start Learning →The foundation of container isolation.
  • Namespace types: mnt, uts, ipc, net, pid, user, cgroup, time
  • Creating namespaces: clone(), unshare(), setns()
  • PID namespace: init process, orphan reaping
  • Mount namespace: propagation, pivot_root
  • Network namespace: veth, bridges, routing
  • User namespace: UID/GID mapping, capabilities
  • Namespace interaction with cgroups
  • Rootless containers
Interview Focus: Explain namespace types, rootless container challenges
Duration: 12-14 hours | Start Learning →Resource control and accounting.
  • Cgroups v1 architecture and controllers
  • Cgroups v2 unified hierarchy
  • CPU controller: shares, quota, period
  • Memory controller: limits, OOM handling
  • I/O controller: BFQ, throttling
  • PID controller: process limits
  • Cgroup namespaces
  • systemd and cgroups integration
  • Kubernetes resource management
  • Container resource limits in practice
Interview Focus: Configure resource limits, explain v1 vs v2 differences
Duration: 10-12 hours | Start Learning →How container runtimes work.
  • OCI runtime specification
  • runc internals: container creation flow
  • containerd architecture
  • Docker daemon and containerd relationship
  • seccomp-BPF in containers
  • Capabilities and privilege dropping
  • AppArmor and SELinux profiles
  • rootfs and overlay filesystems
  • gVisor and Kata Containers (isolation alternatives)
  • Container escape vulnerabilities
Interview Focus: Explain container creation, security boundaries
Duration: 14-16 hours | Start Learning →Linux Security Modules framework and implementations.
  • LSM framework architecture and hooks
  • SELinux: type enforcement, MLS, RBAC
  • AppArmor: path-based MAC
  • Capabilities system deep dive
  • seccomp-BPF: syscall filtering
  • Integrity Measurement Architecture (IMA)
  • Extended Verification Module (EVM)
  • Landlock: unprivileged sandboxing
  • Security namespaces and user namespaces
  • Container security hardening patterns
Interview Focus: Explain LSM architecture, seccomp-BPF filter design

​
Track 7: Advanced Kernel Topics

Deep dives into specialized kernel subsystems.
Duration: 10-12 hours | Start Learning →Understanding how Linux manages power and thermal.
  • CPUFreq governors and scaling drivers
  • CPUIdle: idle states and menu/teo governors
  • P-states and C-states on x86
  • Thermal zones and cooling devices
  • Intel RAPL for power measurement
  • Runtime PM for device power management
  • System suspend and hibernate
  • Wake-on-LAN and wake sources
  • Power profiling with PowerTOP
  • Energy-aware scheduling (EAS)
Interview Focus: Explain cpufreq/cpuidle interaction, power optimization
Duration: 10-12 hours | Start Learning →Timekeeping and timer infrastructure.
  • Clocksources and clock event devices
  • High-resolution timers (hrtimers)
  • Timer wheel for low-resolution timers
  • POSIX timers and interval timers
  • timerfd and polling on time
  • NOHZ (tickless) operation
  • Dynamic ticks and power savings
  • Real-time clock (RTC) subsystem
  • PTP and network time synchronization
  • Timer-related debugging
Interview Focus: Explain hrtimer vs timer_list, NOHZ operation
Duration: 12-14 hours | Start Learning →Tools and techniques for kernel debugging.
  • printk and dynamic debug
  • KGDB: kernel source-level debugging
  • Kdump and crash dump analysis
  • KASAN: kernel address sanitizer
  • UBSAN: undefined behavior sanitizer
  • KCSAN: concurrency sanitizer
  • KMSAN: memory sanitizer
  • ftrace for debugging
  • Kernel debugging with gdb
  • Reading and understanding oops/panic messages
Interview Focus: Analyze kernel oops, explain KASAN operation
Duration: 14-16 hours | Start Learning →Understanding the Linux device model.
  • Device model: buses, devices, drivers
  • sysfs and kobjects
  • Platform devices and device tree
  • Character device drivers
  • Block device drivers
  • Network device drivers
  • PCI and USB subsystems
  • DMA mapping and coherent memory
  • Interrupt handling in drivers
  • IOMMU and device isolation
Interview Focus: Explain device model, write simple char driver

​
Capstone: Interview Preparation

Duration: 10-12 hours | Start Learning →Questions from actual infrastructure/observability interviews.
  • Datadog interview questions and solutions
  • Grafana Labs technical discussions
  • Cloudflare systems questions
  • FAANG infrastructure team questions
  • Debugging scenarios with solutions
  • System design with kernel considerations
  • Take-home assignment patterns
  • Live coding: eBPF and systems programming
Interview Focus: Practice with real questions and detailed solutions
Duration: 20-24 hours | Start Learning →Build portfolio-worthy projects.
  • Project 1: Simple container runtime
  • Project 2: eBPF-based syscall tracer
  • Project 3: Custom I/O scheduler
  • Project 4: Network packet analyzer with XDP
  • Project 5: Memory profiler
  • Project 6: Process monitor with signals
  • Project 7: Kernel module for /proc filesystem
  • Code review and optimization
  • Writing technical documentation
Interview Focus: Demonstrate practical skills with working code

​
Learning Paths

Observability Focus

10-12 weeks
Tracks 1, 2, 5 + eBPF projects
Focus: Interrupts, tracing, profiling
Target: Datadog, Grafana, Honeycomb

Infrastructure Focus

12-14 weeks
Tracks 1-4, 6 + Security modules
Focus: Networking, containers, security
Target: Cloudflare, AWS, Platform teams

Complete Mastery

16-20 weeks
All 7 tracks + all projects
Full kernel expertise
Target: Any senior infrastructure role

​
Specialized Paths

Container Runtime Developer

8-10 weeks
Modules: 1-5, 16-19, 24-25
Focus: Namespaces, cgroups, security
Target: Docker, Kubernetes, containerd teams

Performance Engineer

10-12 weeks
Modules: 1-6, 8-9, 13-15, 20-22
Focus: Profiling, tracing, debugging
Target: Netflix, Performance teams

​
Prerequisites & Environment Setup

1

Programming Skills

Strong C programming, comfortable reading kernel-style code
2

Linux Environment

Ubuntu 22.04+ or Fedora 38+ with root access (VM recommended for kernel experiments)
3

Tools Installation

# Ubuntu/Debian
sudo apt install linux-headers-$(uname -r) build-essential \
  bpftrace bpfcc-tools linux-tools-common linux-tools-generic \
  strace ltrace gdb perf systemtap

# Kernel source (optional but recommended)
sudo apt install linux-source
4

Recommended Resources

  • “Linux Kernel Development” by Robert Love
  • “Understanding the Linux Kernel” by Bovet & Cesati
  • “BPF Performance Tools” by Brendan Gregg
  • kernel.org documentation

​
What Makes This Course Different

For Infrastructure & Observability Roles SpecificallyThis course focuses on what these companies actually ask:
  • eBPF expertise: The foundation of modern observability
  • Performance analysis: Skills for debugging production issues
  • Container internals: Understanding Kubernetes at the kernel level
  • Networking stack: Critical for edge/CDN and infrastructure companies
  • Real interview prep: Questions from actual Datadog, Grafana, Cloudflare interviews
Unlike generic OS courses, every topic connects to practical infrastructure engineering.

​
Course Outcomes

By completing this course, you will be able to:
1

Technical Depth

Navigate and understand Linux kernel source code, including subsystems like scheduling, memory management, and networking
2

Observability Tools

Write eBPF programs for custom monitoring, tracing, and security enforcement
3

Performance Analysis

Profile and debug production systems using perf, ftrace, bpftrace, and kernel debugging tools
4

Container Expertise

Explain container isolation at the namespace/cgroup level and implement security policies
5

Concurrency Mastery

Understand kernel synchronization primitives (spinlocks, RCU, mutexes) and debug concurrency issues
6

Interview Confidence

Discuss Linux internals at the depth expected for senior infrastructure, observability, and SRE roles

​
Quick Reference: All Modules

TrackModuleTopicDuration
11Kernel Architecture10-12h
12System Call Interface12-14h
13Kernel Data Structures8-10h
14Interrupts & Exceptions12-14h
15Synchronization Primitives14-16h
24Process Subsystem14-16h
25Memory Management16-18h
26Virtual Memory10-12h
27Signals & IPC12-14h
37VFS & File Systems14-16h
38Block Layer & I/O12-14h
39Page Cache & Writeback10-12h
410Network Stack16-18h
411Socket Implementation12-14h
412Netfilter & Packet Flow10-12h
513eBPF Deep Dive18-20h
514Tracing Infrastructure14-16h
515Profiling & Flame Graphs12-14h
616Namespaces12-14h
617Cgroups v1 & v212-14h
618Container Runtimes10-12h
619Security Modules (LSM)14-16h
720Power & Thermal10-12h
721Time & Timers10-12h
722Kernel Debugging12-14h
723Device Driver Model14-16h
Cap24Interview Questions10-12h
Cap25Hands-on Projects20-24h
Total Estimated Time: 300-350 hours

​
Interview Deep-Dive

Strong Answer:
  • Observability products like Datadog’s agent need to collect metrics, traces, and logs from every layer of the system with minimal overhead. This requires deep kernel knowledge in three critical areas.
  • First, eBPF and tracing: the Datadog agent uses eBPF programs to attach to kernel tracepoints and kprobes for syscall tracing, network flow monitoring, and security event detection. Understanding the BPF verifier, map types, and program lifecycle is essential for writing production-safe programs that do not crash hosts or exceed CPU budgets.
  • Second, the process and scheduling subsystem: monitoring CPU usage, context switches, and run queue latency requires understanding task_struct, the CFS scheduler’s vruntime accounting, and how cgroups v2 CPU controller reports throttling metrics. Without this knowledge, you cannot interpret cpu.stat or explain why a container shows throttling despite low average CPU usage.
  • Third, the memory subsystem: understanding the OOM killer’s scoring, the difference between RSS, page cache, and slab memory, and how cgroup memory accounting works is essential for building accurate container memory monitoring and for helping customers debug OOM kills.
  • The networking stack is also critical: for network performance monitoring (NPM), understanding sk_buff lifecycle, TCP state machines, connection tracking, and XDP gives you the ability to instrument packet flows without significant overhead.
Follow-up: What is the biggest challenge in building a kernel-level monitoring agent that runs on customer machines across thousands of different kernel versions?Follow-up Answer:
  • Kernel ABI instability is the fundamental challenge. Internal struct layouts, function signatures, and tracepoint arguments change between kernel versions. A kprobe on do_sys_open might work on kernel 5.4 but the function was renamed to do_sys_openat2 in 5.15. eBPF CO-RE (Compile Once, Run Everywhere) with BTF solves this for struct field accesses by adjusting offsets at load time, but it requires the target kernel to be compiled with CONFIG_DEBUG_INFO_BTF=y. For older kernels without BTF, you need BCC-style runtime compilation or pre-compiled binaries per kernel version. The agent must also handle graceful degradation: if a specific eBPF feature is unavailable on an older kernel, fall back to less efficient methods (perf events, procfs polling) rather than failing entirely.
Strong Answer:
  • First priority: the process subsystem and scheduling. Every debugging session starts with understanding what processes are running, why they are slow, and how the scheduler allocates CPU time. Engineers need to understand task_struct, the difference between threads and processes (clone flags), CFS vruntime, and CPU affinity. This is the foundation for interpreting top, perf sched, and cgroup CPU metrics. Without it, statements like “the container is throttled” are meaningless.
  • Second priority: memory management. Memory issues are the most common category of production incidents — OOM kills, memory leaks, swap storms, and page cache eviction. Engineers must understand virtual memory (page tables, TLB), physical memory (zones, buddy allocator), the page cache’s role in file I/O, and how cgroup memory limits interact with the OOM killer. This knowledge is needed to answer “why was my container killed?” and “why is my application slow after running for a week?”
  • Third priority: the system call interface and tracing. This is the bridge between user-space application behavior and kernel internals. Understanding syscall overhead, strace interpretation, and eBPF-based tracing gives engineers the ability to diagnose any problem by following the execution path from application code through libc into the kernel. Once engineers can trace a read() call from user space through VFS to the block layer, they can debug almost anything.
  • I would defer networking stack internals and advanced topics like synchronization primitives until the second phase, because they build on the foundations above.
Follow-up: How would you assess whether an engineer has truly internalized this knowledge versus memorized it?Follow-up Answer:
  • I would give them a live debugging scenario on a test system: a container with an application experiencing intermittent latency spikes. An engineer who has memorized facts will check a checklist of commands. An engineer who understands the internals will form hypotheses (“this could be CPU throttling, memory reclaim, or I/O stalls”), select diagnostic tools that test each hypothesis (check cpu.stat for throttling, vmstat for reclaim, biolatency for I/O), interpret the results in context, and iterate. The key signal is whether they can explain why their diagnosis makes sense at the kernel level — “the throttle count is high because CFS bandwidth enforcement dequeues tasks from the runqueue when quota is exhausted” versus “the throttle number is high so we need more CPU.”
Strong Answer:
  • For Grafana (observability focus): the critical path is eBPF, tracing infrastructure, and the process/memory subsystems. Grafana’s products (Grafana Agent, Pyroscope, Tempo) collect and analyze system-level telemetry. Engineers need deep eBPF expertise (program types, maps, verifier constraints, CO-RE for portability), profiling skills (perf record, flame graph generation, on-CPU vs off-CPU analysis), and understanding of kernel metrics sources (procfs, sysfs, cgroup files). The networking stack matters but primarily from an observability perspective (tracing TCP connections, not implementing protocols).
  • For Cloudflare (infrastructure/networking focus): the critical path is the networking stack, XDP/eBPF for packet processing, and security modules. Cloudflare operates one of the largest edge networks and handles DDoS mitigation, load balancing, and packet filtering at massive scale. Engineers need deep understanding of the packet receive/transmit path, NAPI, sk_buff operations, netfilter/conntrack, and XDP programming. They also need to understand TCP congestion control (BBR vs CUBIC), socket tuning parameters, and RSS/RPS for multi-core packet distribution. Container internals matter less because Cloudflare’s edge does not run customer containers in the traditional Kubernetes sense.
  • Shared requirements for both: strong syscall knowledge (the universal debugging tool), basic process and memory subsystem understanding, and the ability to read kernel source code. Both roles require production debugging skills with perf and bpftrace.
Follow-up: If you had only 8 weeks to prepare for either role, what would your daily study plan look like?Follow-up Answer:
  • Weeks 1-2: kernel architecture, syscall mechanism, and process subsystem (foundation for everything). Spend 2 hours reading, 1 hour doing labs with /proc, strace, and perf. Weeks 3-4: for Grafana, dive into eBPF (write three programs: syscall counter, latency tracer, memory profiler) and tracing infrastructure. For Cloudflare, dive into the networking stack (trace a packet through the kernel, set up XDP, tune TCP). Weeks 5-6: memory management and cgroups for Grafana (container metrics, OOM analysis), or XDP/TC BPF programming and netfilter for Cloudflare. Weeks 7-8: practice interviews with real debugging scenarios, review all lab exercises, and build one portfolio project demonstrating the target skills.
Ready to master Linux internals? Start with Kernel Architecture →