Linux Internals Deep Dive
If you love understanding how things actually work, this chapter is for you. If you just want to run commands and get things done, feel free to skip ahead. No judgment.This chapter takes you beneath the surface of Linux. We will explore how the kernel manages processes, understand how system calls bridge user space and kernel space, and demystify the virtual filesystem. This knowledge is what transforms a Linux user into a Linux engineer.
Why Internals Matter
Understanding Linux internals helps you:- Debug performance issues when top and htop are not enough
- Write better software that works with the kernel, not against it
- Ace interviews where internals questions are common
- Understand containers since Docker relies on kernel features
- Troubleshoot production systems at a deeper level
User Space vs Kernel Space
The most fundamental concept: Linux divides memory into two distinct spaces.- Security: Buggy user programs cannot crash the kernel
- Stability: One process cannot corrupt another
- Abstraction: Applications do not need to know hardware details
System Calls: The Bridge
When a user program needs kernel services (read file, open network connection, create process), it makes a system call.Anatomy of a System Call
Common System Calls
Tracing System Calls
strace is one of the most powerful debugging tools in Linux. It shows you exactly what a process is asking the kernel to do, in real time. When a program hangs, crashes, or behaves strangely, strace reveals what is happening at the system call level — which files it is trying to open, which network connections it is making, and where it is getting stuck.
Process Management
What is a Process?
A process is a running program. It includes:- Code: The program instructions
- Data: Variables and heap
- Stack: Function calls and local variables
- Registers: CPU state
- File descriptors: Open files, sockets
- Memory mappings: Virtual memory layout
Process Control Block (PCB)
The kernel maintains atask_struct for each process:
Process States
The Scheduler
Linux uses the Completely Fair Scheduler (CFS) for normal processes:Real-Time Scheduling
For time-critical tasks, Linux provides real-time schedulers:Memory Management
Virtual Memory
Every process gets its own virtual address space:Page Tables
Virtual addresses translate to physical addresses via page tables:The Page Cache
Linux aggressively caches file data in RAM:- Read a file? It stays in cache for future reads
- Write a file? Goes to cache first, flushed to disk later
- Running low on memory? Cache pages are evicted first
Memory Allocation
When a process requests memory:The Virtual Filesystem (VFS)
Linux abstracts all filesystems through a common interface.VFS Architecture
Key VFS Concepts
Inode: Metadata about a file (permissions, size, timestamps, block pointers). Does NOT contain the filename.Everything is a File
This Unix philosophy extends to:Networking Internals
Understanding the network stack is essential for debugging connectivity problems, container networking, and performance tuning. When your application cannot connect to a database or your API latency spikes, this knowledge tells you where to look.The Network Stack
Think of the network stack like a mail-room in a large building. Your application writes a letter (data), the mail-room staff put it in an envelope (TCP/UDP), add a street address (IP), put it in a delivery truck (Ethernet frame), and the truck drives to the destination (physical network). Each layer handles one responsibility and passes the packet down.Socket Buffers
Data does not go straight from your application to the wire. It passes through kernel buffers at each layer. These buffers are where performance tuning happens — and where problems hide.Netfilter and iptables
Netfilter is the kernel’s packet filtering framework. It provides hooks at five points in the packet processing path where you can inspect, modify, or drop packets.iptables (and its successor nftables) is the user-space tool for configuring these hooks. This is also the foundation for Docker and Kubernetes networking — every port mapping, every Service VIP, every NetworkPolicy is implemented as Netfilter rules under the hood.
Interview Deep Dive Questions
What happens when you run a program?
What happens when you run a program?
Explain the difference between processes and threads
Explain the difference between processes and threads
What is a context switch?
What is a context switch?
How does Linux handle memory overcommit?
How does Linux handle memory overcommit?
Explain the purpose of /proc and /sys
Explain the purpose of /proc and /sys
What is the OOM killer and how does it work?
What is the OOM killer and how does it work?
Exploring Internals Yourself
The best way to internalize these concepts is to poke around on a live system. Every command below reads from /proc or /sys — virtual filesystems that expose kernel internals as ordinary files. Nothing here is dangerous to read.Key Takeaways
- User space and kernel space are separated - for security and stability
- System calls are the bridge - only way to request kernel services
- Everything is a file - devices, processes, kernel data exposed as files
- Virtual memory provides isolation - each process has its own address space
- CFS scheduler ensures fairness - virtual runtime tracks CPU usage
- Page cache makes I/O fast - files cached in RAM automatically
- VFS abstracts filesystems - same interface for ext4, NFS, procfs
- Namespaces and cgroups enable containers - isolation and resource limits
Interview Deep-Dive
A Java application in a Docker container is getting OOMKilled even though the JVM heap is set to 2GB and the container limit is 4GB. What is happening?
A Java application in a Docker container is getting OOMKilled even though the JVM heap is set to 2GB and the container limit is 4GB. What is happening?
- The JVM uses more than heap. Total memory includes heap, metaspace, thread stacks (~1MB per thread), JIT code cache, direct byte buffers, and native memory from libraries. A 2GB heap easily results in 3-4GB total.
- The cgroup memory limit counts all process memory, not just heap. When total RSS exceeds the limit, the OOM killer terminates the process.
- Fix: use
-XX:MaxRAMPercentage=75.0instead of-Xmx2g. This reserves 25% for non-heap usage. On Java 11+,-XX:+UseContainerSupportis enabled by default. - Diagnostic:
jcmd <pid> VM.native_memory summaryshows JVM memory breakdown.cat /sys/fs/cgroup/memory/memory.usage_in_bytesshows cgroup-level usage.
oom_score per process based on RSS, root ownership, and oom_score_adj (-1000 to 1000). Higher score = killed first. In Kubernetes, the kubelet sets oom_score_adj by QoS class: Guaranteed gets -997 (protected), BestEffort gets 1000 (killed first). Check with cat /proc/<pid>/oom_score.Explain fork() and exec() at the kernel level. Why are they two separate system calls?
Explain fork() and exec() at the kernel level. Why are they two separate system calls?
fork()creates a copy of the calling process. The child gets a new PID but inherits memory mappings, file descriptors, and environment. Modern Linux uses copy-on-write: pages are shared until one process writes, making fork() fast even for large processes.exec()replaces the current process image with a new program. Loads the ELF binary, sets up a fresh stack, jumps to the entry point. PID stays the same.- Separation gives you a window between fork and exec to configure the child: redirect file descriptors (how shell pipes work), change working directory, set environment variables, drop privileges. A single “create process” call would need a massive options struct.
- This also enables fork-without-exec for pre-forking servers (Apache, PostgreSQL) where workers run the same code with shared file descriptors.
Your application response time doubled overnight. Using strace and other tools, how do you identify whether the bottleneck is CPU, disk I/O, network, or locks?
Your application response time doubled overnight. Using strace and other tools, how do you identify whether the bottleneck is CPU, disk I/O, network, or locks?
- Start with
vmstat 1 5. Highr+ lowwa= CPU-bound. Highwaorb= I/O-bound. Non-zerosi/so= memory pressure (swapping). - CPU-bound:
htopto find the hot process, thenperf top -p <pid>for function-level profiling. - I/O-bound:
iostat -xz 1to identify the saturated disk (highawait,%utilabove 80%).iotopfor per-process I/O. - Neither:
strace -c -p <pid>to see where time is spent. High time infutex()= lock contention. High time inepoll_wait()orrecv()= waiting on network. - Network:
ss -tn state establishedfor connection counts.curl -w "DNS: %{time_namelookup} Connect: %{time_connect} Total: %{time_total}\n" -o /dev/null -s <URL>for timing breakdown.
futex() indicate?futex() underlies userspace synchronization primitives (mutexes, condition variables). High futex time means threads are contending for locks. Diagnose with perf lock report for native code, jstack for Java thread dumps, or pprof blocking profiles for Go. The fix is usually reducing lock scope, using lock-free structures, or reducing parallelism.Ready to master the command line? Next up: Linux Permissions where we will dive deep into users, groups, and access control.