Skip to main content
Linux Memory Management - Virtual memory, buddy allocator, and page tables

Memory Management Internals

Memory management is one of the most complex subsystems in the Linux kernel. Understanding it deeply is crucial for infrastructure engineers debugging OOM issues, optimizing container resource limits, and understanding application performance.
Interview Frequency: Very High (especially at observability/infrastructure companies)
Key Topics: Buddy allocator, slab, zones, OOM killer, cgroups
Time to Master: 16-18 hours

Physical Memory Organization

Memory Zones

Linux organizes physical memory into zones based on addressing constraints: Physical Memory Zones

Viewing Zone Information


Buddy Allocator

Why Power-of-2 Allocation?

The buddy allocator is the kernel’s physical page allocator. But why does it use powers of 2? The problem: If you allow arbitrary-sized allocations, memory becomes fragmented. You might have 100 free pages scattered around, but can’t allocate a contiguous 64-page block. The solution: Only allow power-of-2 sized blocks (1, 2, 4, 8, 16… pages). This enables:
  1. Fast splitting: A 16-page block splits perfectly into two 8-page blocks
  2. Fast coalescing: Two adjacent 8-page blocks merge back into one 16-page block
  3. Simple math: Finding a block’s “buddy” is just XOR with the block size
What is a “buddy”? Two blocks are buddies if:
  • They’re the same size
  • They’re adjacent in memory
  • Together they form the next larger power-of-2 block
Example: Pages 0-7 and pages 8-15 are buddies (together = pages 0-15). The buddy system manages physical page allocation. Buddy Allocator Mechanism

How Buddy Allocation Works

Buddy Allocator Code

Fragmentation Problem


Slab Allocator (SLUB)

The Problem with Buddy Allocator for Small Objects

The buddy allocator works great for page-sized allocations (4KB+), but what about small objects like task_struct (2.6KB) or dentry (192 bytes)? Problems:
  1. Waste: Allocating a full 4KB page for a 192-byte object wastes 95% of memory
  2. Performance: Buddy allocator has global locks - contention on multi-core systems
  3. Cache efficiency: Small objects from different pages = poor cache locality
The solution: SLUB (Slab Allocator Unqueued)
  • Pre-allocate pages and divide them into same-sized objects
  • Per-CPU caches for lockless fast path
  • Object caches for common kernel structures (task_struct, inode, dentry, etc.)
For small object allocations (smaller than a page).

SLUB Architecture

SLUB Allocator Architecture

SLUB Fast Path

Viewing Slab Information


Page Tables and Virtual Memory

4-Level Page Tables (x86-64)

x86-64 Page Table Walk

TLB (Translation Lookaside Buffer)

TLB Caching Mechanism

Page Fault Handling

When Do Page Faults Happen?

Page faults aren’t errors - they’re a normal part of how Linux manages memory efficiently. Here are real-world scenarios: Minor fault examples:
  • You run ./myprogram - the first time code executes, it’s not in memory yet (demand paging)
  • After fork(), child writes to memory - triggers copy-on-write
  • You mmap() a file but haven’t read it yet - first access brings it in
Major fault examples:
  • Your laptop suspended with apps open - resuming reads pages back from swap
  • You open a 1GB video file with mmap() - reading it triggers disk I/O
  • System under memory pressure swapped out your idle browser - switching back triggers swap-in
Invalid fault examples:
  • char *p = NULL; *p = 'x'; - accessing NULL pointer
  • Buffer overflow writing past array bounds
  • Accessing freed memory (use-after-free)

Monitoring Page Faults


Memory Reclaim

kswapd and Direct Reclaim

Swappiness

The vm.swappiness parameter controls how aggressively the kernel swaps out anonymous memory relative to reclaiming page cache.
  • Range: 0 to 100 (default 60)
  • High value (100): Aggressively swap out anonymous memory to keep page cache (file data) in memory. Good for I/O heavy workloads.
  • Low value (0): Avoid swapping anonymous memory unless absolutely necessary. Good for latency-sensitive applications (databases).
  • Misconception: Setting to 0 does NOT disable swap. It just tells the kernel to prefer reclaiming file pages first.

OOM Killer

Design Philosophy

When the system runs completely out of memory, Linux faces an impossible choice: which process to kill? The goal: Kill the “least important” process that frees the most memory. The challenge: How do you define “least important”? The OOM killer uses a heuristic:
  • Memory usage is the primary factor (kill big memory hogs)
  • oom_score_adj lets you override (protect critical services)
  • Root processes get a slight bonus (system services are important)
  • Recently started processes are slightly preferred (haven’t done much work yet)
Why not just kill the biggest process? Because that might be your database with critical data. The scoring system tries to balance memory freed vs. importance.

OOM Score Calculation


Memory Cgroups

Critical for container resource management.

Cgroup v2 Memory Controller

Memory Cgroup OOM


Huge Pages

Why Huge Pages Matter

Configuring Huge Pages


Lab Exercises

Objective: Understand memory allocation patterns
Objective: Understand OOM killer behavior
Objective: Profile page faults

Interview Questions

Answer:Minor fault:
  • Page is in memory but not mapped in page table
  • No disk I/O required
  • Examples: First touch of allocated memory, COW after fork
  • Cost: ~1-10 microseconds
Major fault:
  • Page not in memory, must read from disk
  • Examples: Swap-in, reading memory-mapped file
  • Cost: ~1-10 milliseconds (1000x slower!)
Production impact:
  • Minor faults: Normal, usually not a concern
  • Major faults: Serious performance problem, indicates memory pressure or working set too large
Monitoring:
Answer:OOM score calculation:
Factors:
  1. Memory usage (primary factor)
  2. oom_score_adj (-1000 to +1000)
  3. Root processes get slight preference
Selection process:
  1. Calculate score for all processes
  2. Select process with highest score
  3. Send SIGKILL to that process
  4. Wait for memory to free
Protection strategies:
  • Set oom_score_adj = -1000 for critical services
  • Use memory cgroups to limit container memory
  • Enable vm.panic_on_oom for critical systems
Answer:When to use kmalloc:
  • Small allocations (<4 MB)
  • DMA buffers
  • Performance-critical paths
  • Interrupt context
When to use vmalloc:
  • Large allocations
  • Module loading
  • Memory that doesn’t need DMA
  • Non-critical paths
Answer:Memory cgroup controls:
  • memory.max: Hard limit (OOM if exceeded)
  • memory.high: Soft limit (throttling)
  • memory.low: Best-effort protection
  • memory.min: Hard protection
Container behavior:
  1. Container requests resources (Kubernetes requests)
  2. Scheduler places based on available memory
  3. Cgroup limits enforced at runtime
  4. Exceeding memory.max → container OOM (not host)
OOM handling:
  • Default: Kill one process in cgroup
  • memory.oom.group = 1: Kill entire cgroup
  • Docker default: Restart policy determines behavior
Best practices:
  • Set memory.max to prevent runaway containers
  • Set memory.high slightly below max for graceful throttling
  • Monitor container memory usage with cAdvisor/Prometheus

Key Takeaways

Buddy Allocator

Manages physical pages efficiently with power-of-2 coalescing

Slab Allocator

Optimizes small object allocation with caching and per-CPU pools

OOM Killer

Protects system by killing processes based on memory score

Memory Cgroups

Enable container memory limits with per-cgroup OOM handling

Interview Deep-Dive

Strong Answer:
  • The container was killed by the cgroup OOM killer, not the system-wide OOM killer. Cgroups v2 memory controller enforces a per-cgroup memory.max limit that is independent of host-level free memory. When the container’s memory.current exceeds memory.max, the kernel first attempts to reclaim pages from within that cgroup (file-backed pages, reclaimable slab). If reclaim fails to free enough, the cgroup OOM killer selects and kills a process within the cgroup.
  • To investigate, I would first check the cgroup memory stats: cat /sys/fs/cgroup/<path>/memory.stat to see the breakdown of anon (heap, stack), file (page cache charged to this cgroup), kernel (slab, page tables, socket buffers), and sock (network buffers). A common surprise is that page cache from file reads is charged to the container’s cgroup, so a database doing sequential scans can fill its memory limit with page cache before the application’s heap is even close to the limit.
  • I would also check memory.events for the oom_kill counter and correlate with dmesg for the detailed OOM output, which shows per-process RSS and the scoring breakdown.
  • Prevention strategies: set memory.high below memory.max to create a throttling buffer where the kernel aggressively reclaims before hitting the hard limit. For databases, consider using O_DIRECT to bypass page cache, or tune vm.swappiness within the cgroup. Always set the JVM’s -Xmx below the container’s memory limit, leaving headroom for native allocations, page tables, and kernel memory charged to the cgroup.
Follow-up: How does the buddy allocator’s fragmentation interact with cgroup memory limits?Follow-up Answer:
  • The buddy allocator operates on physical pages at the zone level, not per-cgroup. However, cgroup memory limits affect the availability of pages for allocation within a cgroup context. When a cgroup is near its memory.max, the kernel may need to reclaim pages before allocating new ones. If the cgroup’s pages are heavily fragmented (many small free chunks but no contiguous blocks), allocations requesting higher-order pages (like transparent huge pages needing order-9 = 2MB) will fail even if the cgroup has reclaimable memory. The kernel will attempt compaction within the cgroup’s pages, but this adds latency. This is why disabling transparent huge pages (echo never > /sys/kernel/mm/transparent_hugepage/enabled) is often recommended for latency-sensitive containerized workloads.
Strong Answer:
  • When user space calls malloc(), glibc either services it from its internal free list or calls mmap() or brk() for new pages. The kernel’s mmap() handler creates a vm_area_struct (VMA) describing the new mapping but does not allocate any physical pages. The page table entries for this range remain empty. This is lazy allocation — the kernel defers physical allocation until the page is actually accessed.
  • When the process first writes to the allocated address, the CPU walks the page table, finds no valid PTE, and raises a page fault exception (interrupt 14 on x86-64). The CPU pushes the error code and faulting address (CR2 register) onto the kernel stack.
  • The kernel’s do_page_fault() handler in arch/x86/mm/fault.c reads CR2 to get the faulting address, then calls find_vma() to locate the VMA containing that address. If no VMA exists, or the access violates VMA permissions, the kernel sends SIGSEGV.
  • For a valid anonymous mapping, handle_mm_fault() walks the page table levels (PGD, P4D, PUD, PMD, PTE), allocating intermediate page table pages as needed. At the PTE level, it calls do_anonymous_page(), which allocates a physical page via alloc_pages() (the buddy allocator), zeroes it (for security), creates a PTE mapping the virtual address to the physical page, and inserts it into the page table.
  • This is a minor fault: no disk I/O was needed, the cost is roughly 2-10 microseconds.
Follow-up: How does transparent huge page allocation change this flow?Follow-up Answer:
  • When THP is enabled and the faulting address falls within a suitably aligned VMA, the PMD-level handler (do_huge_pmd_anonymous_page()) attempts to allocate an order-9 compound page (2MB) instead of a regular 4KB page. If the buddy allocator has a free 2MB block, it maps the entire 2MB region with a single PMD entry, eliminating 511 page table entries and dramatically reducing TLB misses. If the 2MB allocation fails due to fragmentation, the kernel falls back to regular 4KB pages. The khugepaged kernel thread runs in the background, scanning for opportunities to collapse adjacent 4KB pages into 2MB huge pages, which involves migrating pages to achieve contiguity.
Strong Answer:
  • kswapd is the kernel’s background memory reclaim daemon. It wakes up when free memory in a zone drops below the low watermark and reclaims pages until free memory reaches the high watermark. If kswapd is consuming 30% CPU, the system is under sustained memory pressure — applications are allocating faster than kswapd can reclaim.
  • The implications are severe: kswapd reclaim involves scanning LRU lists (active/inactive for both anonymous and file-backed pages), which is CPU-intensive. Worse, if kswapd cannot keep up, application threads enter direct reclaim during alloc_pages(), meaning they block in the allocator trying to free pages before they can proceed. This causes unpredictable latency spikes across all applications.
  • To diagnose, I would check /proc/meminfo for the breakdown: if Cached is high, file pages are consuming memory and can be reclaimed. If AnonPages is high, the working set genuinely exceeds physical memory and swap is being used. I would check /proc/vmstat for pgscan_kswapd vs pgscan_direct — a high ratio of direct reclaim to kswapd reclaim means kswapd is falling behind.
  • Fixes depend on the cause: if excessive page cache, tune vm.vfs_cache_pressure higher to reclaim dentries and inodes more aggressively. If anonymous memory pressure, either add physical memory, reduce the workload, or add swap (swap prevents OOM but adds latency). If the problem is specific containers, tighten their memory.high limits so they self-throttle before causing system-wide pressure.
Follow-up: How does the multi-gen LRU (MGLRU) improve on the traditional two-list LRU for reclaim decisions?Follow-up Answer:
  • The traditional LRU uses two lists per page type (active and inactive) and makes binary decisions: a page is either recently used or not. This leads to poor decisions when the working set changes, because pages must cycle through both lists before being reclaimed. MGLRU (merged in kernel 6.1) introduces multiple generations of pages, tracking access recency with higher fidelity. It uses hardware-assisted page table scanning (checking accessed bits) to build a multi-generational age distribution, allowing the kernel to reclaim the truly coldest pages first. In practice, MGLRU reduces page fault rates and improves throughput under memory pressure, especially for workloads with large, dynamic working sets like databases.

Next: Virtual Memory & Address Translation →