Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Modern Operating System Features

Modern operating systems continue to evolve with new abstractions and optimizations. Understanding these cutting-edge features is essential for building high-performance systems and impressing in senior interviews.
Interview Frequency: High for performance-critical roles
Key Topics: io_uring, eBPF, modern schedulers, memory tiering
Time to Master: 15-20 hours

​
io_uring: Modern Async I/O

io_uring (added in Linux 5.1) is a revolutionary async I/O interface that provides high-performance, low-latency I/O without system call overhead. Think of traditional I/O like ordering food at a restaurant where you have to walk to the kitchen for every dish. io_uring is like having a conveyor belt between your table and the kitchen — you place orders on one belt, and finished dishes arrive on another, all without leaving your seat.

​
Why io_uring?

┌─────────────────────────────────────────────────────────────────┐
│                    I/O EVOLUTION                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Blocking I/O          Async I/O (aio)        io_uring        │
│   ────────────          ──────────────         ────────         │
│   read() blocks         Limited to O_DIRECT    Full async       │
│   One I/O at a time     Complex API            Ring buffers     │
│   Thread per request    Kernel thread pool     Zero-copy        │
│   Syscall per I/O       Syscall per I/O        Batched syscalls│
│                                                                  │
│   Performance (NVMe SSD, QD=1):                                 │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  Blocking read():     ~20,000 IOPS                       │  │
│   │  aio:                 ~100,000 IOPS                      │  │
│   │  io_uring:            ~400,000+ IOPS                     │  │
│   └──────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
io_uring Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    io_uring RING BUFFERS                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   User Space                   Kernel Space                     │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │                                                         │   │
│   │   Submission Queue (SQ)      Completion Queue (CQ)     │   │
│   │   ┌─────────────────┐       ┌─────────────────┐        │   │
│   │   │ SQE1: read()    │       │ CQE1: result=64 │        │   │
│   │   │ SQE2: write()   │       │ CQE2: result=32 │        │   │
│   │   │ SQE3: sendmsg() │       │ (waiting...)    │        │   │
│   │   │ (empty slots)   │       │                 │        │   │
│   │   └────────┬────────┘       └────────▲────────┘        │   │
│   │            │                         │                 │   │
│   └────────────│─────────────────────────│─────────────────┘   │
│                │                         │                      │
│          io_uring_enter()          Results placed              │
│          (optional, or poll)       after completion            │
│                │                         │                      │
│   ─────────────│─────────────────────────│────────────────────  │
│                │                         │                      │
│   Kernel:      ▼                         │                      │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  Process SQEs                                            │   │
│   │  • Execute I/O operations                                │   │
│   │  • Write CQEs when complete                              │   │
│   │  • Optional: polling mode (no syscalls!)                 │   │
│   └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│   Key Benefits:                                                  │
│   • Shared memory: No copy between user/kernel                 │
│   • Batched: Submit many ops, one syscall (or none)            │
│   • Linked: Chain dependent operations                          │
│   • Fixed buffers: Pre-registered for zero-copy                │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
io_uring Example

#include <liburing.h>
#include <fcntl.h>
#include <string.h>

int main() {
    struct io_uring ring;
    struct io_uring_sqe *sqe;
    struct io_uring_cqe *cqe;
    char buffer[1024];
    
    // Initialize ring with 8 entries -- this sets up the shared-memory
    // region between user space and kernel (the "conveyor belts")
    io_uring_queue_init(8, &ring, 0);
    
    int fd = open("test.txt", O_RDONLY);
    
    // Get a slot in the submission queue -- like grabbing
    // a blank order slip from the conveyor belt
    sqe = io_uring_get_sqe(&ring);
    
    // Fill in what we want: "read from this fd into this buffer"
    io_uring_prep_read(sqe, fd, buffer, sizeof(buffer), 0);
    sqe->user_data = 1;  // Tag so we can match completions to requests
    
    // Push the order into the kernel -- may not even need a syscall
    // if SQPOLL mode is enabled (kernel polls the ring autonomously)
    io_uring_submit(&ring);
    
    // Block until the kernel places a result on the completion ring
    io_uring_wait_cqe(&ring, &cqe);
    
    // cqe->res mirrors the return value of a normal read():
    // positive = bytes read, negative = -errno
    if (cqe->res > 0) {
        printf("Read %d bytes\n", cqe->res);
    }
    
    // Tell the kernel we have consumed this completion entry,
    // freeing the slot for future results
    io_uring_cqe_seen(&ring, cqe);
    
    close(fd);
    io_uring_queue_exit(&ring);
    return 0;
}

​
io_uring Operations

// Supported operations (partial list)
io_uring_prep_read()        // Read from file
io_uring_prep_write()       // Write to file
io_uring_prep_readv()       // Vectored read
io_uring_prep_writev()      // Vectored write
io_uring_prep_send()        // Send on socket
io_uring_prep_recv()        // Receive on socket
io_uring_prep_accept()      // Accept connection
io_uring_prep_connect()     // Connect socket
io_uring_prep_openat()      // Open file
io_uring_prep_close()       // Close file
io_uring_prep_statx()       // Stat file
io_uring_prep_poll_add()    // Poll for events
io_uring_prep_timeout()     // Set timeout
io_uring_prep_link_timeout()// Linked timeout

// Advanced features
io_uring_register_buffers() // Pre-register for zero-copy
io_uring_register_files()   // Pre-register file descriptors
IORING_SETUP_SQPOLL         // Kernel polling (no syscalls!)
IOSQE_IO_LINK               // Chain operations

​
eBPF: Programmable Kernel

eBPF (extended Berkeley Packet Filter) allows running sandboxed programs in the kernel without changing kernel code or loading modules. If the kernel is a fortified castle, eBPF is a safe, inspected drone that you can send inside the walls to observe and report back — without ever opening the gates to untrusted code. The verifier acts as the castle guard, ensuring every drone program terminates, never accesses invalid memory, and cannot crash the host.

​
eBPF Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    eBPF ARCHITECTURE                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   User Space                                                     │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  eBPF Program (C)                                        │   │
│   │       │ clang/llvm                                       │   │
│   │       ▼                                                  │   │
│   │  eBPF Bytecode                                           │   │
│   │       │ bpf() syscall                                    │   │
│   └───────│─────────────────────────────────────────────────┘   │
│           │                                                      │
│   ────────│──────────────────────────────────────────────────── │
│           ▼                                                      │
│   Kernel Space                                                   │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  Verifier                                                │   │
│   │  • Checks safety                                         │   │
│   │  • No loops (or bounded)                                 │   │
│   │  • Valid memory access                                   │   │
│   │  • Terminates                                            │   │
│   └─────────────────────────────────────────────────────────┘   │
│           │ Verified                                             │
│           ▼                                                      │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  JIT Compiler                                            │   │
│   │  • Compiles to native code                               │   │
│   │  • Near-native performance                               │   │
│   └─────────────────────────────────────────────────────────┘   │
│           │                                                      │
│           ▼                                                      │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  Attach to Hook Points                                   │   │
│   │  • kprobes (any kernel function)                         │   │
│   │  • tracepoints (stable kernel events)                    │   │
│   │  • XDP (network packet processing)                       │   │
│   │  • tc (traffic control)                                  │   │
│   │  • cgroup (resource control)                             │   │
│   │  • LSM (security)                                        │   │
│   └─────────────────────────────────────────────────────────┘   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
eBPF Use Cases

┌─────────────────────────────────────────────────────────────────┐
│                    eBPF USE CASES                                │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Category        │ Examples                                     │
│   ────────────────┼─────────────────────────────────────────────│
│   Observability   │ • Performance monitoring                    │
│                   │ • Distributed tracing (Cilium Hubble)       │
│                   │ • Custom metrics                            │
│                   │ • Runtime debugging                          │
│                                                                  │
│   Networking      │ • Load balancing (Cilium, Katran)           │
│                   │ • Packet filtering (XDP firewalls)          │
│                   │ • DDoS mitigation                           │
│                   │ • Service mesh (Cilium)                     │
│                                                                  │
│   Security        │ • Runtime security (Falco, Tetragon)        │
│                   │ • Syscall filtering                         │
│                   │ • Container security                        │
│                                                                  │
│   Profiling       │ • Continuous profiling (Parca, Pyroscope)   │
│                   │ • CPU flame graphs                          │
│                   │ • Off-CPU analysis                          │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
eBPF Example

// Simple eBPF program to count syscalls.
// This attaches to every syscall entry and tallies how often
// each syscall number is invoked -- a live histogram of kernel API usage.
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

// A hash map shared between the eBPF program (kernel side) and
// your user-space loader.  Think of it as a scoreboard that both
// the referee (kernel) and the audience (user space) can read.
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 256);
    __type(key, u32);         // syscall number (e.g., 1 = write)
    __type(value, u64);       // invocation count
} syscall_counts SEC(".maps");

// SEC() tells the loader which hook point to attach to.
// "tracepoint/raw_syscalls/sys_enter" fires on every syscall entry.
SEC("tracepoint/raw_syscalls/sys_enter")
int trace_syscall(struct trace_event_raw_sys_enter *ctx) {
    u32 syscall_id = ctx->id;
    u64 *count = bpf_map_lookup_elem(&syscall_counts, &syscall_id);
    
    if (count) {
        // Atomic increment -- safe even though multiple CPUs
        // may hit this path concurrently
        __sync_fetch_and_add(count, 1);
    } else {
        // First time seeing this syscall: initialize counter
        u64 init_val = 1;
        bpf_map_update_elem(&syscall_counts, &syscall_id, &init_val, BPF_ANY);
    }
    
    // Return 0 to tell the kernel "continue normally"
    return 0;
}

// The verifier requires a GPL-compatible license to access
// certain helper functions (e.g., bpf_trace_printk)
char LICENSE[] SEC("license") = "GPL";

​
eBPF Maps

// Map types for sharing data

// Hash map (key-value store)
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);
    __type(value, struct stats);
} hash_map SEC(".maps");

// Array (indexed access)
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 256);
    __type(key, u32);
    __type(value, u64);
} array_map SEC(".maps");

// Per-CPU array (no locking needed)
struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 256);
    __type(key, u32);
    __type(value, u64);
} percpu_map SEC(".maps");

// Ring buffer (efficient event streaming)
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);  // 256KB
} events SEC(".maps");

// LRU hash (automatic eviction)
struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 10000);
    __type(key, struct flow_key);
    __type(value, struct flow_stats);
} lru_map SEC(".maps");

​
Modern Schedulers

​
EEVDF (Earliest Eligible Virtual Deadline First)

Linux 6.6+ replaces CFS with EEVDF. The core insight: CFS was “fair on average” but could produce short-term latency spikes for tasks that needed quick bursts of CPU. EEVDF adds a deadline concept so that latency-sensitive tasks (like audio or UI rendering) get their time slice when they actually need it, not just eventually. 
┌─────────────────────────────────────────────────────────────────┐
│                    EEVDF vs CFS                                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   CFS (Completely Fair Scheduler)                               │
│   ────────────────────────────────                              │
│   • Virtual runtime tracking                                    │
│   • Red-black tree for task selection                           │
│   • Good average case, but:                                     │
│     - Latency spikes possible                                   │
│     - Unfair to short-running tasks                             │
│                                                                  │
│   EEVDF (Earliest Eligible Virtual Deadline First)              │
│   ─────────────────────────────────────────────────             │
│   • Each task has a virtual deadline                            │
│   • Eligible: Has runnable time slice                           │
│   • Pick task with earliest deadline                            │
│   • Better latency guarantees                                   │
│   • Fairer to short tasks                                       │
│                                                                  │
│   Key Concept:                                                   │
│   • lag = service_received - service_expected                   │
│   • Positive lag: Got more than fair share                      │
│   • Negative lag: Got less than fair share                      │
│   • Eligible: lag ≤ 0 (hasn't exceeded fair share)             │
│   • Pick eligible task with earliest deadline                   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
Scheduler Classes

┌─────────────────────────────────────────────────────────────────┐
│                    LINUX SCHEDULER CLASSES                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Priority  │ Class          │ Policy         │ Use Case        │
│   ──────────┼────────────────┼────────────────┼──────────────── │
│   Highest   │ stop_sched     │ (internal)     │ Stop machine    │
│             │ dl_sched       │ SCHED_DEADLINE │ Hard real-time  │
│             │ rt_sched       │ SCHED_FIFO     │ Soft real-time  │
│             │                │ SCHED_RR       │                 │
│             │ fair_sched     │ SCHED_NORMAL   │ Regular tasks   │
│   Lowest    │ idle_sched     │ SCHED_IDLE     │ Very low prio   │
│                                                                  │
│   SCHED_DEADLINE:                                                │
│   • Earliest Deadline First (EDF)                               │
│   • Parameters: runtime, deadline, period                       │
│   • Guaranteed CPU time if schedulable                          │
│   • Used for hard real-time tasks                               │
│                                                                  │
│   Example:                                                       │
│   struct sched_attr attr = {                                    │
│       .sched_policy = SCHED_DEADLINE,                           │
│       .sched_runtime = 10 * 1000 * 1000,   // 10ms              │
│       .sched_deadline = 30 * 1000 * 1000,  // 30ms              │
│       .sched_period = 30 * 1000 * 1000,    // 30ms              │
│   };                                                             │
│   sched_setattr(0, &attr, 0);                                   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
Core Scheduling

For security (Spectre/Meltdown), run only trusted code together on SMT siblings. SMT (Simultaneous Multi-Threading, marketed as “Hyper-Threading”) shares execution resources — ALUs, caches, branch predictors — between two logical cores on the same physical core. This sharing creates side-channel leakage: a malicious thread can probe the branch predictor or cache to extract secrets from its sibling. Core Scheduling solves this by ensuring that only threads belonging to the same trust domain (same “cookie”) run simultaneously on SMT siblings: 
# Enable core scheduling
echo 1 > /sys/kernel/debug/sched/core_sched_enabled

# Create core scheduling group
# Only threads in same cookie run on SMT siblings

prctl(PR_SCHED_CORE, PR_SCHED_CORE_CREATE, 0, PIDTYPE_TGID, 0);

# Kernel ensures:
# - SMT siblings run tasks with same cookie
# - Mitigates some side-channel attacks

​
Memory Management Advances

​
Memory Tiering

┌─────────────────────────────────────────────────────────────────┐
│                    MEMORY TIERING                                │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Modern systems have heterogeneous memory:                     │
│                                                                  │
│   Tier 0 (Fastest)                                               │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  HBM (High Bandwidth Memory) / Local DRAM                │  │
│   │  ~100 ns latency                                         │  │
│   └──────────────────────────────────────────────────────────┘  │
│                              │                                   │
│                              ▼ Page migration                    │
│   Tier 1 (Medium)                                                │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  CXL Memory / Remote NUMA                                 │  │
│   │  ~200-500 ns latency                                     │  │
│   └──────────────────────────────────────────────────────────┘  │
│                              │                                   │
│                              ▼ Page migration                    │
│   Tier 2 (Slowest)                                               │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  Persistent Memory (PMEM) / Compressed Memory            │  │
│   │  ~1000+ ns latency                                       │  │
│   └──────────────────────────────────────────────────────────┘  │
│                                                                  │
│   Kernel features:                                               │
│   • kswapd: Demotes cold pages                                  │
│   • NUMA balancing: Promotes hot pages                          │
│   • Memory tiering (Linux 5.15+): Explicit tier management     │
│                                                                  │
│   # Check memory tiers                                           │
│   cat /sys/devices/system/node/node*/memory_tier                │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
MGLRU (Multi-Gen LRU)

Linux 6.1+ introduces Multi-Generational LRU for better page reclaim: 
┌─────────────────────────────────────────────────────────────────┐
│                    MGLRU vs Classic LRU                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Classic LRU:                                                   │
│   ┌────────────────────────────────────────────────────────┐    │
│   │  Active List ◄──► Inactive List ──► Reclaim            │    │
│   └────────────────────────────────────────────────────────┘    │
│   • Two lists only                                               │
│   • Age information limited                                      │
│   • Scanning overhead for large memory                          │
│                                                                  │
│   MGLRU:                                                         │
│   ┌────────────────────────────────────────────────────────┐    │
│   │  Gen 0   Gen 1   Gen 2   Gen 3    ──►  Reclaim         │    │
│   │ (oldest)                 (youngest)                     │    │
│   └────────────────────────────────────────────────────────┘    │
│   • Multiple generations (typically 4)                          │
│   • Better age tracking                                          │
│   • Faster page aging                                            │
│   • Reduces CPU overhead for scanning                           │
│   • Better for large memory systems                              │
│                                                                  │
│   Enable MGLRU:                                                  │
│   echo 1 > /sys/kernel/mm/lru_gen/enabled                       │
│                                                                  │
│   Benefits:                                                      │
│   • 10-20% better throughput under memory pressure              │
│   • Lower tail latency                                          │
│   • Better workload isolation                                    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
Filesystem Innovations

​
SMR and ZNS Support

┌─────────────────────────────────────────────────────────────────┐
│                    ZONED STORAGE                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Traditional SSD:                                               │
│   ┌────┬────┬────┬────┬────┬────┬────┬────┐                    │
│   │ R/W│ R/W│ R/W│ R/W│ R/W│ R/W│ R/W│ R/W│  Random access    │
│   └────┴────┴────┴────┴────┴────┴────┴────┘                    │
│                                                                  │
│   Zoned Namespace (ZNS) SSD / SMR HDD:                          │
│   ┌──────────────────────┬──────────────────────┐               │
│   │     Zone 1           │     Zone 2           │               │
│   │ ████████░░░░░░░░░░░░│░░░░░░░░░░░░░░░░░░░░│              │
│   │ ──────► Sequential   │ Write pointer        │               │
│   └──────────────────────┴──────────────────────┘               │
│                                                                  │
│   Rules:                                                         │
│   • Must write sequentially within a zone                       │
│   • Can only reset (erase) entire zone                          │
│   • Better SSD endurance and cost                               │
│                                                                  │
│   Filesystem support:                                            │
│   • Btrfs: Native zone support                                  │
│   • f2fs: Flash-Friendly FS with zone support                   │
│   • ZoneFS: Exposes zones as files                              │
│                                                                  │
│   # Check zone info                                              │
│   cat /sys/block/nvme0n1/queue/zoned                            │
│   # Values: none, host-aware, host-managed                      │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
FUSE and Filesystem in Userspace

┌─────────────────────────────────────────────────────────────────┐
│                    FUSE ARCHITECTURE                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   User Space                                                     │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  Application                                             │   │
│   │     │ VFS operation (open, read, write)                 │   │
│   └─────│───────────────────────────────────────────────────┘   │
│         │                                                        │
│   ──────│────────────────────────────────────────────────────── │
│         ▼                                                        │
│   Kernel                                                         │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  VFS Layer                                               │   │
│   │     │                                                    │   │
│   │     ▼                                                    │   │
│   │  FUSE Kernel Module                                      │   │
│   │     │                                                    │   │
│   └─────│───────────────────────────────────────────────────┘   │
│         │ /dev/fuse                                              │
│   ──────│────────────────────────────────────────────────────── │
│         ▼                                                        │
│   User Space                                                     │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │  FUSE Daemon (Your Filesystem)                           │   │
│   │  • libfuse                                               │   │
│   │  • Implement operations                                  │   │
│   └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│   Use cases:                                                     │
│   • sshfs: Mount remote filesystem over SSH                    │
│   • s3fs: Mount S3 bucket as filesystem                        │
│   • rclone: Mount cloud storage                                 │
│   • Avoids kernel development for custom filesystems           │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
Security Features

​
Landlock

Linux Security Module for unprivileged sandboxing. Unlike seccomp (which filters syscalls) or namespaces (which isolate resources), Landlock lets a regular, unprivileged process voluntarily restrict its own filesystem access — similar to how a mobile app declares which folders it needs. This is defense-in-depth: even if an attacker exploits a bug in your application, Landlock limits what damage they can do. Practical tip: Landlock is especially useful for processing untrusted input (PDF rendering, image conversion). Sandbox the worker before it touches the data. 
#include <linux/landlock.h>
#include <sys/syscall.h>

// Create ruleset
struct landlock_ruleset_attr ruleset_attr = {
    .handled_access_fs = LANDLOCK_ACCESS_FS_READ_FILE |
                         LANDLOCK_ACCESS_FS_WRITE_FILE,
};

int ruleset_fd = syscall(SYS_landlock_create_ruleset,
                         &ruleset_attr, sizeof(ruleset_attr), 0);

// Add rule: Allow read from /usr
struct landlock_path_beneath_attr path_beneath = {
    .allowed_access = LANDLOCK_ACCESS_FS_READ_FILE,
    .parent_fd = open("/usr", O_PATH),
};

syscall(SYS_landlock_add_rule, ruleset_fd,
        LANDLOCK_RULE_PATH_BENEATH, &path_beneath, 0);

// Enforce ruleset
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
syscall(SYS_landlock_restrict_self, ruleset_fd, 0);

// Now process can only read from /usr
// All other file access will fail with EACCES

​
io_uring Security

// io_uring can be restricted for security

// IORING_REGISTER_RESTRICTIONS
struct io_uring_restriction res[] = {
    // Only allow specific operations
    { .opcode = IORING_RESTRICTION_SQE_OP,
      .sqe_op = IORING_OP_READ },
    { .opcode = IORING_RESTRICTION_SQE_OP,
      .sqe_op = IORING_OP_WRITE },
    // Require fixed files only
    { .opcode = IORING_RESTRICTION_SQE_FLAGS_REQUIRED,
      .sqe_flags = IOSQE_FIXED_FILE },
};

io_uring_register_restrictions(ring, res, ARRAY_SIZE(res));
io_uring_enable_rings(ring);

// Some systems disable io_uring entirely
// sysctl kernel.io_uring_disabled=2

​
Interview Questions

io_uring advantages:
  1. Shared memory rings: No copying between user/kernel
  2. Batched submissions: Multiple ops per syscall
  3. Polling mode: Zero syscalls possible (SQPOLL)
  4. Zero-copy: Pre-registered buffers
  5. Async everything: File, network, timers all async
Performance gains:
  • Syscall overhead eliminated or amortized
  • Better cache utilization
  • 4-10x IOPS improvement for NVMe
When to use:
  • High-performance servers
  • Storage-intensive applications
  • When syscall overhead matters
eBPF (extended Berkeley Packet Filter):Run verified, sandboxed programs in kernel:
  • No kernel recompilation
  • No kernel modules
  • Safe: Verified before execution
  • Fast: JIT compiled
Use cases:
  1. Observability: Trace syscalls, functions, performance
  2. Networking: XDP for fast packet processing, load balancing
  3. Security: Runtime threat detection, syscall filtering
  4. Profiling: CPU, memory, off-CPU analysis
How it works:
  • Write in C or bpftrace
  • Compile to BPF bytecode
  • Kernel verifier checks safety
  • JIT compiles to native code
  • Attach to hook points (kprobes, tracepoints, XDP)
XDP (eXpress Data Path):eBPF programs that run at the NIC driver level, before sk_buff allocation.Actions:
  • XDP_DROP: Drop packet (fastest firewall)
  • XDP_PASS: Continue to network stack
  • XDP_TX: Send back out same interface
  • XDP_REDIRECT: Send to different interface/CPU
Performance:
  • Millions of packets per second per core
  • 10-100x faster than iptables for simple filtering
Use cases:
  • DDoS mitigation (drop bad traffic early)
  • Load balancing (Facebook’s Katran)
  • Traffic filtering
  • Packet modification
Limitations:
  • Limited to packet processing
  • Must handle raw packets
  • Driver support required for best performance

​
Production Caveats and Patterns

The features in this chapter are powerful and dangerous. They are powerful because they unlock performance and observability that were impossible a few years ago; dangerous because they expose new attack surfaces, lifecycle complexity, and footguns that production teams keep stepping on.
Caveat 1: io_uring lifecycle bugs are subtle, and a misuse of submission/completion ring management can cause use-after-free, double-free, or silent data corruption. Each SQE (submission queue entry) references user buffers and file descriptors; the kernel reads these asynchronously, often after the submitting code has moved on. If you free the buffer before the kernel completes the operation, you get a UAF. Multiple CVEs in 2021-2023 — privilege escalation in registered buffers, race conditions in cancellation — led Google’s Android team to disable io_uring entirely in production Android kernels. Some container runtimes block io_uring_setup via seccomp by default for the same reason.
Pattern: Pre-register buffers and files; use linked SQEs for ordered work; treat io_uring as an ownership transfer. When you submit an SQE, conceptually you hand the buffer and the FD to the kernel until the matching CQE arrives. Use io_uring_register_buffers() for fixed-buffer mode; lifetimes become explicit. For ordered work (read-then-write), use IOSQE_IO_LINK so the kernel chains them and you only see one completion. Audit your seccomp profile — if you do not need io_uring for untrusted workloads, block io_uring_setup (syscall 425) and io_uring_enter (426). For Rust, use io-uring or tokio-uring crates with their lifetime checking; for C, add valgrind-style instrumented testing of the submission/completion paths.
Caveat 2: userfaultfd is a powerful primitive used by CRIU, Postgres, and some VMMs — and a recurring kernel exploit primitive. It lets a userspace process intercept page faults and supply pages on demand, enabling live migration, memory snapshotting, and fast clone. The flip side: it gives attackers fine-grained control over kernel memory access timing, which has been weaponized in dozens of exploits (Dirty Pipe variants, page-cache races, refcount bugs). Linux 5.11+ added the unprivileged_userfaultfd sysctl that defaults to 0 — root-only — specifically to limit attack surface.
Pattern: Disable unprivileged_userfaultfd unless you have a legitimate need; if you do, scope it tightly. Check /proc/sys/vm/unprivileged_userfaultfd. The default of 0 is correct for most production servers. If you run software that requires userfaultfd (CRIU for container live migration, Postgres for some experimental features, some database engines for snapshotting), grant it via capability (CAP_SYS_PTRACE) or set the sysctl to 1 only on hosts running that software. Pair with seccomp policies that allow userfaultfd syscall only for the specific binary that needs it. Audit kernel security advisories quarterly — this primitive resurfaces in CVE lists every year.
Caveat 3: memfd_secret (Linux 5.14+) creates memory invisible to root processes, but it does not protect against kernel exploits. It maps anonymous memory that is not exposed in /proc/<pid>/mem, not visible to ptrace, not swappable, and not accessible to the kernel direct-map (so even kernel code cannot trivially read it). Used for secrets in cryptographic libraries that want strong defense against attacker-with-root scenarios. The catch: a kernel exploit that can call arbitrary kernel functions can still extract the memory, because the kernel itself can re-map the pages. It is a defense-in-depth feature, not an absolute guarantee.
Pattern: Use memfd_secret for short-lived, high-value secrets and pair with mlock + secure-erase. Allocate via memfd_secret() (passing flags=FD_CLOEXEC), ftruncate to size, mmap to map, then write secret material directly. Before unmap, use explicit_bzero() (or volatile-write loop) to wipe. This stack is what modern crypto libraries (recent OpenSSL builds, libsodium) use for in-memory key material. Combine with hardware features (AMD SME, Intel TME) for memory encryption at rest in DRAM. Do not rely on memfd_secret alone for compliance scenarios — it raises the bar for memory disclosure attacks but does not satisfy FIPS 140-3 by itself.
Caveat 4: BPF-LSM, SELinux, and AppArmor are all Linux Security Modules — and you must understand which one you are betting on. BPF-LSM lets you write LSM hooks in eBPF: fine-grained, dynamic, but new (Linux 5.7+) and less mature than the alternatives. SELinux is the heavyweight: comprehensive policy language, used in RHEL/Android, vast policy ecosystem, but notoriously hard to author correctly. AppArmor is path-based, simpler to write, used in Ubuntu/SUSE, but less expressive than SELinux. You can stack BPF-LSM with one of the others; you cannot stack SELinux and AppArmor.
Pattern: Pick one mainline LSM as your security baseline, add BPF-LSM for application-specific rules. For Red Hat / OpenShift / Android: SELinux is the right default; the policy ecosystem and tooling are mature. For Ubuntu / SUSE / general distros: AppArmor is the practical default. For new hardening (your application has a small set of unusual constraints), use BPF-LSM to express them in code — this is what tools like Tetragon and Inspektor Gadget do. Audit which LSM is enabled with cat /sys/kernel/security/lsm. Do not run with no LSM in production; the difference between “AppArmor in complain mode” and “no LSM” is observable in incident response timelines.

​
Summary

┌─────────────────────────────────────────────────────────────────┐
│              MODERN OS FEATURES SUMMARY                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  io_uring:                                                       │
│  • Ring buffer based async I/O                                  │
│  • Minimal syscall overhead                                     │
│  • Supports all I/O types                                       │
│                                                                  │
│  eBPF:                                                           │
│  • Safe kernel programmability                                  │
│  • Tracing, networking, security                                │
│  • XDP for fast packet processing                               │
│                                                                  │
│  Schedulers:                                                     │
│  • EEVDF: Better fairness and latency                          │
│  • SCHED_DEADLINE: Hard real-time                               │
│  • Core scheduling: SMT security                                │
│                                                                  │
│  Memory:                                                         │
│  • MGLRU: Better page reclaim                                   │
│  • Memory tiering: CXL and PMEM                                 │
│                                                                  │
│  Security:                                                       │
│  • Landlock: Unprivileged sandboxing                           │
│  • Improved io_uring restrictions                               │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

​
Interview Deep-Dive

Strong Answer:The key bottleneck with epoll + synchronous read/write at this scale is syscall overhead. At 1M IOPS, if each operation requires a syscall for submission and another for completion, you are burning 2M syscalls per second. Each syscall costs 100-200ns (mode switch, KPTI CR3 swap on Intel), so you lose 200-400ms of CPU time per second just on transitions — that is 20-40% of a single core doing nothing but entering and exiting the kernel.
  • Why io_uring helps: The submission queue (SQ) and completion queue (CQ) are shared memory ring buffers between user space and kernel. You can batch 32 or 64 I/O operations into a single io_uring_enter() call, or in SQPOLL mode, eliminate syscalls entirely because a kernel thread polls the SQ autonomously. At 1M IOPS, SQPOLL means the kernel thread continuously drains SQEs without any user-to-kernel transition. The result is 4-10x IOPS improvement over epoll+read for NVMe workloads.
  • Pre-registered buffers and files: io_uring_register_buffers() and io_uring_register_files() let you pre-register memory regions and file descriptors. This skips per-operation validation (fget/fput for fd lookup, get_user_pages for buffer pinning) and enables true zero-copy paths.
Migration risks:
  • Error handling changes: With synchronous I/O, errors are returned inline. With io_uring, errors appear asynchronously in CQEs. Your entire error-handling architecture needs redesigning.
  • Backpressure: If the application submits faster than the device can complete, the SQ fills up. You need to handle io_uring_get_sqe() returning NULL gracefully.
  • Debugging difficulty: Async I/O is harder to trace. Standard strace does not show io_uring operations well. You need bpftrace or perf to observe the ring buffer activity.
Security considerations:
  • io_uring has been a major attack surface: Multiple CVEs in 2021-2023 (privilege escalation, use-after-free in registered buffers). Google’s Android team disabled io_uring entirely in Android kernels. Some container runtimes block io_uring_setup via seccomp by default.
  • SQPOLL runs a kernel thread with user-specified parameters: This thread runs at kernel privilege continuously. If you enable it, ensure the process is trusted and resource-limited via cgroups.
  • Mitigation: Use IORING_REGISTER_RESTRICTIONS to limit which operations the ring can perform. In containerized environments, audit whether your seccomp profile allows io_uring_setup and io_uring_enter.
Follow-up: In what scenario would io_uring actually perform worse than epoll?For workloads with very few concurrent I/O operations (say, a single-threaded CLI tool doing sequential reads), io_uring adds complexity without benefit. The ring buffer setup cost and the overhead of managing SQEs/CQEs exceeds the savings from batching when you are only submitting one operation at a time. Also, for network I/O on older kernels (before 5.7 or so), io_uring’s socket support was incomplete and buggy — epoll was more reliable. The crossover point where io_uring wins is typically around queue depth 4-8+.
Strong Answer:The eBPF verifier is the gatekeeper that makes running untrusted code inside the kernel safe. Before any eBPF program executes, the verifier performs static analysis on the bytecode to prove several safety properties:
  • Termination: The program must provably terminate. Originally this meant no loops at all. Since Linux 5.3, bounded loops are allowed if the verifier can prove a finite upper bound on iterations (e.g., a loop with a counter that decrements to zero). The verifier tracks the possible range of every register and proves the loop bound statically.
  • Memory safety: Every memory access must be within bounds. The verifier tracks the type and valid range of every register. If you have a pointer to an sk_buff data region, the verifier knows the bounds (ctx->data to ctx->data_end) and rejects any access that could go out of bounds. This is why you see those if ((void *)(eth + 1) > data_end) return XDP_PASS; checks everywhere — the verifier requires them.
  • No invalid pointers: You cannot dereference a register that might be NULL without checking first. The verifier tracks NULL-ability through conditional branches.
  • Stack safety: The eBPF stack is limited to 512 bytes. The verifier ensures no stack overflow.
  • Helper function safety: eBPF programs can only call pre-approved kernel helper functions (bpf_map_lookup_elem, bpf_probe_read, etc.). Each helper has a type signature the verifier checks.
Limitations the verifier has:
  • Complexity limit: The verifier has a maximum instruction limit it will analyze (1 million verified instructions as of recent kernels). Complex programs with many branches can hit this limit and be rejected even if they are correct. This forces you to simplify control flow or split programs.
  • Conservative analysis: The verifier over-approximates. It may reject a program that is actually safe because it cannot prove the safety statically. For example, a perfectly safe pointer arithmetic expression might be rejected because the verifier loses track of the value range through a complex series of operations.
  • No floating point: eBPF has no floating-point support. All math is integer.
  • Limited data structures: You cannot dynamically allocate memory inside an eBPF program. All data sharing goes through pre-defined BPF maps.
The practical implication is that writing eBPF programs is like writing code for a very strict compiler. You have to structure your code to “help” the verifier understand safety. Frameworks like libbpf and the BPF CO-RE (Compile Once, Run Everywhere) approach abstract some of this complexity.Follow-up: How do BPF maps enable communication between eBPF programs and user space, and what are the concurrency implications?BPF maps are kernel-managed data structures (hash tables, arrays, ring buffers, LRU caches, etc.) shared between eBPF programs running in kernel context and user-space programs via the bpf() syscall. For concurrency: per-CPU maps (BPF_MAP_TYPE_PERCPU_HASH, BPF_MAP_TYPE_PERCPU_ARRAY) eliminate locking entirely because each CPU has its own copy — ideal for counters and statistics. Regular maps use RCU internally for reads and spinlocks for writes, so readers never block. Ring buffers (BPF_MAP_TYPE_RINGBUF, added in Linux 5.8) are the preferred way to stream events from kernel to user space because they support variable-length records and are more efficient than the older perf buffer approach. The gotcha is that if user space cannot drain the ring buffer fast enough, events are silently dropped — you must monitor the drop counter.
Strong Answer:CFS (Completely Fair Scheduler) tracks each task’s vruntime — the CPU time consumed, weighted by priority — and always picks the task with the lowest vruntime to run next. Over long periods, this is mathematically fair. The problem is short-term unfairness, which CFS calls “lag.”
  • The CFS problem: Imagine two tasks, A (interactive, short bursts) and B (CPU-bound, long bursts). Task A sleeps for 50ms, wakes up, and needs 2ms of CPU. Under CFS, A’s vruntime fell behind while it slept, so it gets picked next — good. But if 20 tasks all wake up simultaneously (common in event-driven servers), CFS picks them in vruntime order, and the last few tasks might wait 10-20ms even though they only need a 1ms burst. CFS has no notion of urgency or deadline — it only knows “who has consumed the least so far.”
  • Latency spikes for short tasks: Audio processing, UI rendering, and network packet handling are highly sensitive to scheduling latency. CFS’s sched_latency parameter (default 6ms for 8 tasks) provides a target, but under load it is a best-effort guarantee. A task that needs CPU now has no way to express that urgency.
EEVDF (Earliest Eligible Virtual Deadline First) fixes this by adding a deadline dimension:
  • Virtual deadline: Each task gets a virtual deadline calculated as vruntime + (time_slice / weight). This represents “by when should this task have received its fair share.”
  • Eligibility: A task is “eligible” if its lag is less than or equal to zero — meaning it has not exceeded its fair share. A task that has been running too much (positive lag) becomes ineligible temporarily.
  • Selection: Among all eligible tasks, EEVDF picks the one with the earliest virtual deadline. This means a short-burst task that just woke up gets a near-term deadline and is scheduled quickly, while a long-running task that has been consuming CPU gets a farther-out deadline and yields.
The practical result is tighter latency bounds. In benchmarks, EEVDF reduced worst-case scheduling latency for interactive tasks by 30-50% compared to CFS under the same load. Audio engineers and real-time Linux users had been patching around CFS’s latency issues for years; EEVDF addresses the root cause at the algorithm level.Follow-up: Does EEVDF eliminate the need for SCHED_DEADLINE and real-time scheduling classes?No. EEVDF operates within the fair scheduling class (SCHED_NORMAL). It improves latency for general-purpose tasks but does not provide hard deadline guarantees. SCHED_DEADLINE uses EDF (Earliest Deadline First) with admission control — the kernel mathematically proves that all admitted tasks can meet their deadlines before allowing them to run. If you need provable guarantees (audio processing with 5ms deadlines, robotics control loops), you still need SCHED_DEADLINE or SCHED_FIFO. EEVDF makes the “normal” class good enough that many soft real-time workloads no longer need to reach for RT scheduling, which is the real win.
Strong Answer:These three mechanisms operate at fundamentally different layers and are complementary, not competing:
  • Seccomp (syscall filtering): Operates at the syscall boundary. A seccomp-BPF filter inspects each syscall number and its arguments, and decides allow/deny/kill. It answers the question: “Which kernel APIs can this process invoke?” Seccomp cannot distinguish between files — if you allow open(), you allow opening any file. It is coarse-grained for filesystem access but excellent for reducing the kernel attack surface (blocking dangerous syscalls like ptrace, mount, kexec_load).
  • Namespaces (resource isolation): Change what the process can see. A PID namespace makes a process think it is PID 1. A mount namespace gives it a different filesystem tree. A network namespace gives it a separate network stack. Namespaces answer: “What does the world look like to this process?” They are powerful but require root or CAP_SYS_ADMIN to create (except user namespaces), and they are coarse — you get a whole new view, not fine-grained access control.
  • Landlock (filesystem access control): Operates at the VFS layer. An unprivileged process can voluntarily restrict its own filesystem access to specific directories and specific operations (read, write, execute, make_dir, etc.). It answers: “Which files and directories can this process access, and how?” The key differentiator is that Landlock is self-imposed and unprivileged — no root needed, no special setup.
When to choose each:
  • Processing untrusted input (PDF renderer, image converter): Use Landlock to restrict filesystem access to only the input/output directories, plus seccomp to block unnecessary syscalls. This is defense-in-depth: even if an attacker gets code execution through a parsing bug, they cannot read /etc/shadow (Landlock blocks it) and cannot call ptrace (seccomp blocks it).
  • Container isolation (Docker, Kubernetes): Use all three. Namespaces provide the fundamental isolation (separate PID, mount, network views), seccomp profiles block dangerous syscalls, and newer container runtimes are adding Landlock for fine-grained filesystem restrictions within the container.
  • Reducing attack surface of a network daemon: Seccomp is the primary tool. Block everything the daemon does not need. A web server does not need mount, reboot, kexec_load, or ptrace. Drop those with seccomp-BPF.
The practical tip: these mechanisms compose. A well-sandboxed application uses prctl(PR_SET_NO_NEW_PRIVS) first (prevents privilege escalation via setuid binaries), then applies a seccomp filter, then applies Landlock rules. Each layer catches what the others miss.Follow-up: What is the performance cost of Landlock vs seccomp?Seccomp-BPF runs a small BPF program on every syscall entry, which adds roughly 10-50ns per syscall depending on filter complexity. Landlock hooks into the VFS at specific operations (open, mkdir, etc.) and checks the rules only for filesystem operations, so its overhead is negligible for non-filesystem syscalls. For filesystem-heavy workloads, Landlock adds a small constant cost per VFS operation to walk the ruleset tree. In benchmarks, both are well under 1% overhead for typical workloads. The cost of not using them — a privilege escalation vulnerability — is infinitely more expensive.
Strong Answer Framework:
  1. State the architectural difference. epoll is an event-readiness API: you ask the kernel to notify you when an FD is ready for I/O, then you do a syscall (read, write) to actually transfer data. Two syscalls per I/O on the hot path. io_uring is a completion-based API: you submit the work (read this, write that) into a shared ring; the kernel performs the work asynchronously and posts a completion when done. With SQPOLL, zero syscalls per I/O.
  2. Identify where io_uring wins. High-fanout, high-IOPS workloads where syscall overhead dominates. Storage servers handling 500k+ IOPS on NVMe see 4-10x improvement over epoll+pread. Network proxies handling millions of QPS see 2-5x. Workloads that batch many I/Os at once (read 100 files in parallel) — io_uring lets you submit them all in one ring write, reducing context switches.
  3. Identify where epoll is fine or better. Single-threaded servers with moderate concurrency (under 10k QPS): the syscall savings are not measurable. Workloads where I/O is dominated by network round-trip time, not local syscall cost (HTTP servers waiting on DNS, proxies behind slow upstreams). Workloads where you need cross-language portability or run on older kernels (epoll is universal; io_uring is Linux 5.1+, with usable network support more like 5.5+).
  4. Identify where io_uring is wrong. Untrusted code execution — io_uring has had multiple privilege-escalation CVEs and is blocked in some environments (Android, recent ChromeOS). Workloads with very few concurrent operations (a CLI tool sequentially reading files): the ring setup cost exceeds the syscall savings. Code where async lifecycle management would be a major architectural lift you cannot afford.
  5. State the crossover. Roughly: queue depth 4-8 is the break-even point. Below that, epoll is comparable or better. Above that, io_uring’s batching and zero-syscall properties dominate. Use latency targets and IOPS goals to pick the threshold. Always benchmark on your actual hardware and workload, not on synthetic numbers.
Real-World Example: ScyllaDB (a Cassandra-compatible NoSQL database written in Seastar) was an early io_uring adopter and reported 30-40% throughput improvements on NVMe-backed workloads vs their previous AIO + epoll implementation. The improvement came primarily from eliminating the per-IOP context switch — at 200k+ IOPS per core, even 200ns of syscall overhead matters. By contrast, the Linux team at Cloudflare benchmarked io_uring for their HTTP edge proxy in 2022 and concluded that for their specific workload (TLS termination, then forward to upstream), the bottleneck was the upstream network round-trip, not the local I/O syscall path. They stayed on epoll, citing operational maturity and audit simplicity.
Senior follow-up: What is SQPOLL mode and when does it pay off? SQPOLL has the kernel run a thread that polls the submission ring continuously, draining SQEs without any user-to-kernel transition. Zero syscalls per I/O. The cost is a kernel thread always running (or running while there is work, in newer kernels) — effectively burning a core. SQPOLL pays off above some threshold (often 100k+ IOPS) where the saved syscall cost exceeds the dedicated-core cost. Below that, the dedicated thread is wasted.
Senior follow-up: Why do some teams disable io_uring entirely? Security. The interface is large, complex, and has had a steady stream of CVEs since 2020. Google’s Android team disabled io_uring in production Android kernels (post-Pixel 7 era) because the attack surface in the kernel was unacceptable on devices that run untrusted apps. ChromeOS made similar decisions for renderer processes. For server workloads in trusted environments, the trade-off favors enabling io_uring; for edge/mobile/multi-tenant, it often does not.
Senior follow-up: How does io_uring change the design of an async runtime like Tokio? Tokio’s main loop on Linux is built on epoll. Adding io_uring is non-trivial because the lifetime models are different: epoll lifetimes match the future’s lifetime (you wake up, do read, return); io_uring requires the buffer to outlive the operation, which conflicts with borrow-based lifetime in Rust. tokio-uring uses owned buffers (Bytes) to side-step this. The result is a slightly different async API. Most apps stay on the epoll-backed Tokio for portability and simplicity; performance-critical apps reach for tokio-uring or glommio (an io_uring-native runtime).
Common Wrong Answers:
  1. “io_uring is always faster than epoll.” Untrue at low queue depths and untrue for workloads where syscall cost is not the bottleneck.
  2. “io_uring replaces epoll entirely.” No — they coexist on every modern Linux system. epoll is older and more universally supported; io_uring is newer and more capable for specific workloads.
  3. “io_uring is just async I/O.” It is more general — io_uring supports network I/O, file I/O, opening/closing files, statx, and more. It is closer to “async syscall framework” than “async I/O.”
Further Reading:
  • Jens Axboe’s “Efficient IO with io_uring” paper (2019) — the canonical introduction by the author.
  • Glauber Costa’s posts on Seastar and io_uring — design lessons from a real high-IOPS database.
  • LWN.net articles “Ringing in a new asynchronous I/O API” and follow-ups — evolution of the API and the security history.
Strong Answer Framework:
  1. State what older tools could and could not do. strace traces syscalls one process at a time with high overhead (ptrace per call, often 10-100x slowdown). ftrace gives you static tracepoints with low overhead but limited filtering — you typically capture a lot and filter offline. kprobes/uprobes let you attach to arbitrary kernel/user functions, but the actions are limited to “log this event” or “emit a counter.” None of these support arbitrary in-kernel logic at the trace point.
  2. State what eBPF adds. A safe, JIT-compiled, in-kernel program runs at the trace point. It can read function arguments and return values, walk kernel data structures, aggregate into BPF maps (per-CPU counters, histograms), and emit only the records you actually want. The result: you can answer questions like “what is the latency distribution of this syscall, broken down by process and bucket?” with ~1% overhead — something that was previously impossible without instrumenting the application or running a sampling profiler.
  3. List the categories of new capability. (a) Performance analysis at production load — BCC’s biolatency shows block-I/O latency histograms across the entire fleet without measurable overhead. (b) Security observability — Falco and Tetragon attach to syscalls and report anomalies in real time. (c) Custom networking — Cilium replaces iptables with eBPF for service-mesh-grade policy at line rate. (d) Programmatic profiling — bpftrace one-liners replace what used to require kernel modules.
  4. Identify the limits. eBPF programs have a 1M-instruction verifier limit, a 512-byte stack, no dynamic allocation, no floating point. They can only call helper functions, not arbitrary kernel functions. Some kernel data is invisible to BPF (encrypted memory regions, certain BPF-LSM hook contexts). The verifier rejects programs it cannot prove safe, so you sometimes have to refactor to help it.
  5. State the operational shift. eBPF lets you ship observability tools as small, declarative programs (BCC, bpftrace, libbpf-tools) that run anywhere a recent kernel runs. You no longer need to wait for the next kernel release to add a new metric. Production diagnosis that took weeks (kernel module, peer review, deploy, monitor) now takes minutes (write a bpftrace one-liner, run, learn).
Real-World Example: Netflix uses eBPF extensively for production diagnosis. Brendan Gregg’s bcc and bpftrace toolkits are the de facto standard for Linux performance debugging. A specific example from his book BPF Performance Tools: diagnosing a “tail latency anomaly” in a microservice. Old approach: enable detailed JVM logging, capture for hours, re-deploy with extra metrics, repeat. New approach: a 10-line bpftrace script attached to the JVM safepoint kprobes that aggregated stop-the-world pauses by stack trace, run for 5 minutes in production, root cause identified. The total time from “I noticed a problem” to “I have a fix” went from days to hours.
Senior follow-up: How does eBPF enable observability without code changes to the application? Tools like USDT (User-Statically-Defined Tracing) markers exist in many runtimes (Java JDK, Python, Node, JVM, MySQL) and let eBPF attach without recompilation. uprobes go further — attach to any user-space symbol, including third-party binaries. Cilium Hubble uses tc-bpf at the network device level to observe all traffic to/from a pod with no app awareness. The application is unaware it is being measured — which is exactly what you want for production diagnosis.
Senior follow-up: What is the difference between BCC and libbpf, and why does it matter operationally? BCC compiles BPF programs at runtime using clang/LLVM in the target kernel. Pro: works with the running kernel’s headers. Con: requires LLVM at runtime (~50 MB), slow startup, fragile across kernel versions. libbpf with CO-RE (Compile Once, Run Everywhere) compiles ahead-of-time using BTF (BPF Type Format) for kernel-version portability. Pro: small binaries, fast startup. Con: requires kernel BTF (mainline since 5.4 with CONFIG_DEBUG_INFO_BTF). For production fleet deployment, libbpf-CO-RE is now the standard.
Senior follow-up: Where does eBPF fit relative to OpenTelemetry? They are complementary. OpenTelemetry is a standardized observability data format (traces, metrics, logs) and SDK. eBPF is a mechanism for collecting data in the kernel and userspace. Tools like Pixie and Parca use eBPF to collect data and emit it in OTel-compatible formats. The eBPF layer answers “how do I observe without changing the app”; OTel answers “how do I make all my observability data interoperable across vendors.”
Common Wrong Answers:
  1. “eBPF is just a faster strace.” Massively understates it. eBPF lets you run arbitrary safe code at trace points — aggregations, conditionals, data structure walks. strace is logging, eBPF is computation.
  2. “eBPF replaces all monitoring agents.” No — agents (Datadog, Prometheus exporters) handle metric shipping, dashboarding, alerting. eBPF is a data source those agents can use. They coexist.
  3. “eBPF is unsafe because it runs kernel code.” The verifier exists specifically to make it safe. The class of safety properties eBPF guarantees (no kernel crashes, no infinite loops, no out-of-bounds memory access) is provable — which is more than you can say for kernel modules.
Further Reading:
  • Brendan Gregg, BPF Performance Tools (2019) — comprehensive book on production performance analysis with eBPF.
  • “Learning eBPF” by Liz Rice (O’Reilly, 2023) — accessible introduction with practical examples.
  • ebpf.io — the canonical home for tooling, examples, and the eBPF Foundation’s resources.
Strong Answer Framework:
  1. State the threat model precisely. Confidential computing protects code and data from a privileged attacker on the same machine — the cloud provider, the host OS, the hypervisor, a malicious admin. The trust boundary is the CPU package itself: you trust the hardware vendor and the silicon, and you do not trust anything outside the CPU (RAM, disk, BIOS, hypervisor, host kernel). The use case is: “I want to run my workload in someone else’s cloud and not trust the cloud provider with my data.”
  2. Explain Intel SGX (Software Guard Extensions). Process-level enclaves: a process can carve out an enclave region, and only enclave code can read enclave memory. The CPU encrypts enclave memory with a hardware-derived key. Any access from outside the enclave (kernel, hypervisor, other processes) sees ciphertext. Limitation: enclave size historically capped at ~128 MB, with anything beyond paged in/out (slow). SGX2 (Ice Lake+) raised this to several GB but still has limits.
  3. Explain AMD SEV-SNP / Intel TDX. VM-level confidential computing: an entire VM runs encrypted, with memory access from the hypervisor returning ciphertext. Larger blast radius (full VM, not just an enclave) but easier to retrofit existing applications — you do not have to re-architect into “trusted enclave + untrusted host.” SEV-SNP adds attestation and integrity protection. TDX is Intel’s equivalent. AWS Nitro Enclaves, Azure DCsv3 / DCdsv3, GCP Confidential VM all use this layer.
  4. Explain Arm CCA (Confidential Compute Architecture). Newer (rolling out in 2024-2025), Realm-based: a VM-style isolation backed by Arm’s Realm Management Extension (RME). Conceptually similar to TDX/SEV-SNP but designed for Arm’s heterogeneous compute model. Used for data-center deployments and increasingly for edge/AI inference workloads.
  5. Identify the gaps. This is where senior judgment shows.
    • Side channels. SGX has had repeated side-channel breaks: Foreshadow (2018), Plundervolt (2019), SGAxe / CrossTalk (2020). SEV-SNP and TDX have had cache-timing and speculative-execution attacks. The hardware vendors patch them, but new ones keep appearing. Confidential computing is robust against direct memory access and most direct attacks; it is fragile against microarchitectural side channels.
    • Attestation trust chain. You must verify the attestation report comes from a genuine CPU running the firmware version you expect. This requires bootstrapping trust through the CPU vendor’s PKI (Intel Attestation Service, AMD’s KDS). If that PKI is compromised or the attestation library has bugs, the whole chain falls.
    • Software bugs in enclaves. The enclave code itself can have bugs. Confidential computing protects from external attackers, not from a vulnerable implementation inside the enclave.
    • I/O. Anything that leaves the enclave (network, disk) is not protected by these mechanisms. You need TLS for network, encrypted storage for disk, and careful design for any data crossing the boundary.
Real-World Example: Signal’s “Private Contact Discovery” service used Intel SGX from 2017 to provide a server-side contact-discovery feature where the server provably could not see users’ contact lists. The enclave hashed and matched contacts; the host OS saw only encrypted enclave memory. After several SGX side-channel breaks, Signal moved off SGX in 2022 in favor of Oblivious RAM techniques for stronger guarantees. The lesson: confidential computing is a meaningful defense, but it is not a substitute for cryptographic protocols that do not require trusting the hardware.
Senior follow-up: What is remote attestation, and why is it the linchpin of confidential computing? Attestation is the process by which a verifier proves that the code running in an enclave/VM is what they expect. The CPU signs a quote containing a measurement (hash) of the enclave code, the enclave configuration, and a nonce from the verifier. The verifier checks the signature against the CPU vendor’s PKI. Without attestation, you have isolated computation, but you do not know what code is running there — a malicious operator could replace the binary. Most production confidential computing deployments tie attestation to key release: the secret keys are delivered to the enclave only after attestation succeeds.
Senior follow-up: When should a team NOT use confidential computing? When the threat model does not include the cloud provider or the host OS. If you trust your provider (most internal use cases) and your concern is application-layer attackers, confidential computing adds operational complexity (attestation infrastructure, debugging is harder, performance overhead is 10-30% in many workloads) without a corresponding security benefit. Use TLS, encryption-at-rest, strong access controls, and avoid the operational burden.
Senior follow-up: How does confidential computing interact with confidential AI / private inference? This is one of the fastest-growing use cases. ML inference on user data (medical, financial, legal) using a model running in a confidential VM lets you process sensitive data without the cloud operator seeing inputs or outputs. NVIDIA’s H100 and H200 GPUs added confidential-computing support (NVIDIA Confidential Computing) so the entire CPU+GPU pipeline is protected. Watch for products from Apple, Google, Microsoft on confidential AI; this is the area where confidential computing is moving from niche to mainstream.
Common Wrong Answers:
  1. “Confidential computing means encrypted data.” Encryption-at-rest exists everywhere. Confidential computing is encryption-during-use — data is protected even while the CPU is operating on it. That is a much harder property.
  2. “SGX is dead because of side-channel attacks.” SGX is deprecated for client (consumer) CPUs but still actively developed for server. SGX2 with a larger enclave size and TDX as a successor are alive. The narrative “dead” is overstated; the narrative “fragile against side channels” is accurate.
  3. “Confidential computing replaces trust in the cloud provider.” It reduces trust scope to the CPU vendor, not eliminates trust. You now trust Intel/AMD/Arm. For some threat models that is a meaningful improvement; for others (state-level adversaries who might pressure the silicon vendor) it is not.
Further Reading:
  • Confidential Computing Consortium whitepapers — ccc.io has architectural overviews and threat-model docs.
  • Costan and Devadas, “Intel SGX Explained” (2016) — the most thorough public treatment of SGX internals.
  • “AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More” (AMD whitepaper, 2020) — canonical reference for SEV-SNP design.