OS Fundamentals & System Call Internals
Operating Systems exist to manage hardware and provide a safe abstraction for applications. A “Senior” engineer must understand the physical transition between these two worlds.0. What is an Operating System?
At the highest level, an Operating System is a resource manager and isolation layer:- Resource Manager:
- Multiplexes CPU time between many processes.
- Allocates and reclaims memory, files, sockets, and devices.
- Schedules and prioritizes work according to policy (throughput, latency, fairness, deadlines).
- Isolation & Protection Layer:
- Prevents one program from corrupting another program’s memory.
- Prevents untrusted code from directly touching hardware.
- Enforces security boundaries (user vs kernel, containers, VMs).
- Streets/highways ⇢ CPU cores and buses.
- Buildings ⇢ processes.
- Rooms ⇢ threads.
- Zoning rules and permits ⇢ permissions and security policies.
- Traffic lights ⇢ synchronization and scheduling.
0.1 Core Responsibilities
Every mainstream OS (Linux, Windows, macOS, BSD, RTOS variants) implements the same core ideas:- Abstraction: Present simple interfaces (files, sockets, processes) instead of raw devices and registers.
- Virtualization: Make a single physical CPU and memory look like many virtual CPUs and address spaces.
- Isolation: Ensure faults in one address space do not corrupt others.
- Coordination: Provide primitives (locks, signals, pipes, futexes) so concurrent entities can cooperate.
- Accounting: Track which process used how much CPU, memory, I/O; enforce quotas and limits.
0.2 Types of Operating Systems
Monolithic Kernels
Microkernels
Hybrid Kernels
Real-Time OS
0.5 From Source Code to Running Process
To make OS fundamentals concrete, walk through what happens when you compile and run a simple C program:Step 1: Compilation and Linking
- Preprocessing
- Compilation
- Assembly
- Linking
#include and macros into a single translation unit:Step 2: Shell Creates a New Process
When you run:- Your shell (itself a process) parses the command.
- The shell calls
fork():- The kernel creates a child process by copying the parent’s PCB and page tables (copy-on-write).
- Parent and child now both exist; they differ only in the return value of
fork().
Step 3: Child Calls execve()
In the child:
- The shell calls
execve("./main", ...). - The kernel:
- Reads the ELF headers from disk.
- Allocates a new address space.
- Maps code, data, stack, and shared libraries into that space.
- Sets up the initial user stack with
argc/argvand environment. - Sets the program counter to the C runtime entry point (
_start).
execve():
Step 4: C Runtime → main → Exit
- The C runtime (
crt1.o) runs first, initializing the runtime and calling yourmain(). - Your code executes (
printf("hello\n")), which itself issues syscalls under the hood (write()on stdout). - When
mainreturns, the runtime callsexit(), which:- Flushes stdio buffers.
- Invokes the
exit_groupsyscall. - Lets the kernel tear down the process (free memory, close FDs, reap the PCB).
1. The Kernel vs. User Space
The CPU hardware enforces the boundary.1.1 Privilege Levels (Protection Rings)
- x86/x86-64
- ARM
- RISC-V
HLT- halt the CPUCLI/STI- disable/enable interruptsMOV CR3, reg- change page tablesLGDT/LIDT- load GDT/IDTIN/OUT- direct hardware I/O (on some systems)
1.2 What Each Mode Can Do
2. System Call Evolution (x86-64)
How does a program ask the kernel for help?2.1 Legacy: INT 0x80 (i386)
In the 32-bit era, applications used a software interrupt.
- CPU saves registers (CS, EIP, EFLAGS).
- CPU looks up interrupt vector 0x80 in the IDT (Interrupt Descriptor Table).
- CPU jumps to kernel’s interrupt handler.
- Handler switches to kernel stack.
- Handler calls the appropriate syscall function.
- Handler returns using
IRET.
- Interrupt controller overhead
- Full register save/restore
- Stack switching
- Permission checks
2.2 Modern: SYSCALL (AMD) / SYSENTER (Intel)
x86-64 introduced a dedicated instruction for syscalls.
SYSCALL Instruction (AMD64):
- No IDT lookup: CPU jumps to address stored in
IA32_LSTARMSR (Model Specific Register). - No stack lookup: Uses
IA32_KERNEL_GS_BASEfor per-CPU data. - Minimal save: Only saves
RIPandRFLAGStoRCXandR11.
- Direct jump (no table lookup)
- Minimal register save/restore
- No interrupt controller involved
2.3 ARM64: SVC Instruction
- Saves PC to
ELR_EL1 - Saves PSTATE to
SPSR_EL1 - Jumps to exception vector
- Kernel dispatches based on
x8
2.4 RISC-V: ECALL Instruction
3. The vDSO (Virtual Dynamic Shared Object)
Some system calls are called thousands of times per second (e.g.,gettimeofday(), clock_gettime()). Switching to kernel mode every time is a massive waste of CPU.
3.1 How it Works
The vDSO is a special page of memory that the kernel maps into every user process’s address space. This page contains:- Code: Executable functions that run in user mode.
- Data: Read-only kernel data (like current time).
gettimeofday() Without vDSO:
gettimeofday() With vDSO:
3.2 Implementation Details
Kernel Side (sets up vDSO):3.3 Why Some Syscalls Can’t Use vDSO
Safe for vDSO:- Read-only operations
- No side effects
- Data changes slowly or predictably
write()- modifies kernel statefork()- creates processmmap()- changes address spaceopen()- allocates file descriptor
3.4 Finding the vDSO
4. vsyscall: The Legacy Fixed Address
Before the vDSO, there was vsyscall.
4.1 The Problem with vsyscall
4.2 Modern Linux Solution
vsyscall is now emulated:5. Kernel Entry: entry_SYSCALL_64
When the SYSCALL instruction is executed, the CPU jumps to this assembly entry point in the kernel.
5.1 Complete Entry Path (x86-64)
5.2 The C Dispatcher
5.3 The Syscall Table
5.4 Complete Flow Diagram
6. Processes vs. Threads vs. Kernel Tasks
Before we dive into system call micro-details, it is critical to distinguish the units of execution the kernel manages:Process
- Own virtual address space
- Own page tables (
mm_struct) - Own resources (FDs, signals, cwd)
- Heavyweight context switch
Thread
- Shares address space with process
- Own stack and registers
- Own TID
- Lightweight context switch (same CR3)
Kernel Thread
- No user address space
- Lives entirely in kernel
- Examples: kswapd, kworker
- No context switch overhead for syscalls
6.1 Visualization
Think of a process as a house, and threads as people inside the house:6.2 Why It Matters for System Calls
Example:read() system call
- Issued by a thread
- CPU time charged to the process
- May block only this thread, not the whole process
- Other threads can continue executing
fork() system call
- Creates a new process with copy-on-write address space
- The child initially has only one thread (the caller)
- Even though parent had 5 threads, they don’t get copied
- Child’s single thread continues from the
fork()return point
7. System Call Deep Dive: Real Linux Examples
7.1 Example: write() System Call
User space call:
glibc/sysdeps/unix/sysv/linux/write.c):
arch/x86/entry/syscall_64.c):
fs/read_write.c):
fs/read_write.c):
7.2 Example: getpid() - Fast Path
User space:
7.3 Example: open() - Complex Path
fs/namei.c):
- String parsing (
/home/user/file.txt→ components) - Dentry cache lookups (hot path)
- Inode cache lookups
- Disk reads (cold path, if not cached)
- Permission checks (each directory component)
- File allocation
- FD table modification
- Hot (all cached): 1-2 μs
- Cold (disk reads): 1-10 ms
8. Performance Analysis of System Calls
8.1 Measuring Syscall Overhead
Microbenchmark:8.2 Using perf to Analyze
8.3 Syscall Batching Strategies
Bad: Many small syscalls9. Security Implications
9.1 Spectre/Meltdown and Syscalls
Meltdown (2018) exploited speculative execution:9.2 Syscall Filtering with seccomp
seccomp-BPF: Filter which syscalls a process can make- Docker containers
- Chrome sandbox
- systemd service hardening
10. Interview Deep Dive Questions
Q1: Explain exactly what happens during a system call on x86-64
Q1: Explain exactly what happens during a system call on x86-64
- Application loads syscall number into RAX
- Arguments into RDI, RSI, RDX, R10, R8, R9
- Executes
SYSCALLinstruction
Q2: Why does the vDSO exist and what syscalls benefit from it?
Q2: Why does the vDSO exist and what syscalls benefit from it?
gettimeofday(), this overhead dominates.Solution: vDSO (Virtual Dynamic Shared Object)How it works:- Kernel maps a page of executable code into every process
- This page contains implementations of certain syscalls
- Kernel periodically updates read-only data in this page
- Libc resolves these functions to vDSO instead of syscalls
- Calls execute entirely in user space (no mode switch)
gettimeofday()- reads kernel’s time dataclock_gettime()- reads clock datagetcpu()- reads current CPU numbertime()- simplified time call
- Must be read-only (no side effects)
- Data must be safely readable from user space
- Data changes must be atomic/consistent
- Cannot require kernel state modifications
Q3: What is SWAPGS and why is it critical for security?
Q3: What is SWAPGS and why is it critical for security?
- GS register points to per-CPU data structures
- In user mode: GS points to user thread-local storage (TLS)
- In kernel mode: GS points to kernel per-CPU area
- Spectre v1 exploited missing SWAPGS
- CPU could speculatively execute kernel code with user GS
- Kernel would read wrong data, leak information
- SWAPGS must be first instruction in syscall entry
- Must happen before any GS-relative memory access
- Hardware barriers prevent speculative execution reordering
Q4: Compare the cost of system calls vs function calls
Q4: Compare the cost of system calls vs function calls
- Push return address
- Branch (usually predicted correctly)
- Return (predicted via RAS - Return Address Stack)
- Mode switch overhead: ~50 cycles
- Register save/restore: ~20 cycles
- TLB effects (if PCID not used): ~50 cycles
- Cache effects (kernel code not in L1): ~50+ cycles
- Privilege level transition (Ring 3 → Ring 0 → Ring 3)
- Page table switch (if KPTI enabled)
- TLB flush (if PCID not supported)
- Cache pollution (kernel code evicts user code)
- Security checks and barriers
- Batch operations (writev vs many writes)
- Use vDSO when available
- Use memory mapping (mmap) to avoid read/write syscalls
- Use io_uring for async I/O with minimal syscalls
Q5: How do system calls differ across architectures?
Q5: How do system calls differ across architectures?
-
Calling convention:
- x86-64 uses different registers for syscalls vs function calls
- ARM/RISC-V use same registers for both
-
Syscall numbering:
- Each architecture has different syscall numbers
writeis #1 on x86-64, #64 on ARM64/RISC-V- Forces architecture-specific syscall tables
-
Mode switching:
- x86: Ring 0 vs Ring 3
- ARM: EL0 vs EL1
- RISC-V: U-mode vs S-mode
-
Performance:
- Similar overhead (~100-200 cycles)
- RISC architectures slightly cleaner (fewer legacy modes)
- All benefit from vDSO equally
11. Hands-On Practice
Lab 1: Tracing System Calls
Lab 2: Minimal Syscall (No libc)
Lab 3: Benchmark Syscall Overhead
Lab 4: Examine vDSO
Summary for Senior Engineers
System Calls Aren't Free
vDSO is Magic
SWAPGS is Critical
Syscall Table is Kernel's API
- Privilege separation is enforced by hardware (CPU rings/modes)
- System calls are the only legitimate way to cross the user/kernel boundary
- vDSO eliminates syscall overhead for frequently-used operations
- Security mechanisms (KPTI, SWAPGS, seccomp) protect the syscall interface
- Performance matters: Modern systems minimize context switches
Production Caveats: What Goes Wrong at the User-Kernel Boundary
The textbook explanation makes syscalls look clean. Production reality is messier. Most kernel-related performance bugs and security incidents trace back to specific failure modes around mode switching, syscall semantics, and kernel architecture choices.Senior Interview Questions: Kernel Architecture and Syscall Semantics
Explain monolithic versus microkernel using Linux versus L4. Where does each architecture win, and why has Linux's design dominated despite the theoretical elegance of microkernels?
Explain monolithic versus microkernel using Linux versus L4. Where does each architecture win, and why has Linux's design dominated despite the theoretical elegance of microkernels?
- Define the architectural difference. Monolithic kernels (Linux) run drivers, file systems, network stack, and scheduler all in kernel mode (ring 0). They communicate via direct function calls and shared data structures. Microkernels (L4, seL4, QNX) keep only the absolute minimum (address space management, IPC, scheduling) in kernel mode and move everything else (file systems, drivers, networking) to user-space servers.
- Performance reality. A function call in a monolithic kernel is roughly 5 cycles. The equivalent operation in a microkernel is an IPC round-trip: typically 1 to 10 microseconds even on optimized L4 (sub-microsecond), with multiple context switches and cache effects. For an operation invoked 100K times per second (typical for a busy file server), the difference is 1 to 10 percent of CPU pure overhead.
- Reliability reality. A bug in a Linux driver can kernel-panic the whole system. A bug in a microkernel driver crashes the driver process; the kernel can restart it. seL4 takes this further: it is formally verified, with mathematical proof that the kernel itself contains no bugs. For mission-critical systems (defense, medical, automotive), this matters more than performance.
- Linux’s middle path. Linux is mostly monolithic but with three escape valves: loadable kernel modules (drivers can crash without rebuilding the kernel),
eBPF(sandboxed user-supplied code in kernel space), and user-mode helpers (FUSE, vhost-user, CUSE). Drivers that need isolation run as VFIO-passed user-space processes. This pragmatic hybrid captures most microkernel benefits at a fraction of the cost. - Why monolithic won market share. Linux’s monolithic design was good enough plus orders of magnitude faster than 1990s-era microkernels (Mach was the cautionary tale). By the time microkernel IPC got fast (L4 in 2000s), Linux had already eaten desktop, server, mobile, and embedded markets. Network effects and ecosystem gravity outweigh architectural elegance.
- “Microkernels are objectively safer and the industry is wrong.” Andy Tanenbaum’s argument from the 1992 Tanenbaum-Torvalds debate. It misses that “safer” without “fast and ecosystem-rich” does not win market share. seL4 is safer; almost no one runs it.
- “Linux is monolithic so it cannot be reliable.” Linux runs Google, Facebook, AWS, and most of the planet’s infrastructure with five to seven nines availability. Reliability is engineering practice (testing, fuzzing, eBPF-based verification, hardening), not architecture per se.
- Liedtke, “On µ-kernel construction” (1995) — the foundational L4 paper.
- Klein et al., “seL4: formal verification of an OS kernel” (SOSP 2009) — how you actually prove a kernel correct.
- Linus Torvalds vs Andy Tanenbaum, USENET archive (1992) — the original “Linux is obsolete” debate, still worth reading for the framing.
What is the cost of a syscall, and how do modern OSes minimize it (vDSO, io_uring, etc.)? Quantify the difference.
What is the cost of a syscall, and how do modern OSes minimize it (vDSO, io_uring, etc.)? Quantify the difference.
- Baseline cost on x86-64. A bare
syscallinstruction round-trip is roughly 100 cycles (about 30ns at 3 GHz). With KPTI (Meltdown mitigation) it jumps to 200 to 500 cycles because of the CR3 page-table switch. With Spectre mitigations (IBPB, IBRS) and KPTI fully enforced, it can hit 1000+ cycles. The variance comes from PCID support (mitigates KPTI cost), microarchitectural state, and what the syscall actually does. - Where the time goes. Roughly: 30 cycles for the privilege transition itself (SWAPGS, register save), 50 to 100 cycles for the page-table switch and TLB effects under KPTI, 20 to 50 cycles for the dispatcher (validate syscall number, look up in
sys_call_table, security checks via LSM hooks), then the actual syscall body, then the reverse. The fixed overhead is 100 to 500 cycles regardless of what the syscall does. - vDSO eliminates the overhead entirely for read-only operations.
clock_gettime,gettimeofday,getcpu,timeare mapped as user-space code that reads kernel-maintained data via a seqlock. Cost: roughly 10 to 20 cycles. Speedup: 10 to 30x. - io_uring batches syscalls. Submit a queue of operations (read, write, accept, recv) and reap completions, with a single (or zero, with
IORING_SETUP_SQPOLL) syscall. For 100 ops, cost goes from 100 syscalls (10K to 50K cycles) to 1 syscall (100 to 500 cycles): 50 to 100x reduction. Used by databases (Ceph, ScyllaDB), high-perf web (proxygen). - Other techniques.
eBPFfor in-kernel processing without round-tripping to userspace (XDP for networking, BPF LSM for security). Kernel bypass via DPDK/RDMA/SPDK skips the syscall entirely for I/O. Shared memory + lock-free queues for IPC where syscalls are not needed (futexonly on contention).
IORING_SETUP_SQPOLL doubled their throughput on NVMe storage workloads. The gain was almost entirely from eliminating syscalls, not from faster I/O. They went from roughly 250K IOPS per core to 500K IOPS per core on the same hardware.- “Syscalls always cost about 1 microsecond.” Wrong by an order of magnitude in either direction depending on architecture, mitigations, and what the syscall does. The honest answer is “between 30ns and 1 microsecond, measure it on your kernel.”
- “io_uring makes everything faster.” No. For low-throughput workloads, the queue management overhead exceeds the saved syscall cost. io_uring is for high-throughput, not for general use.
- Jens Axboe, “Efficient I/O with io_uring” (kernel.org, 2019) — the original design document.
- Brendan Gregg, “Linux System Call Performance” (LWN-style write-up, 2018) — how to measure syscall overhead in production.
- Linux kernel
Documentation/userspace-api/vsyscall.rstandDocumentation/ABI/stable/vdso— the canonical references.
Walk me through what SWAPGS does and why it became a security-critical instruction after Spectre. What is the LFENCE doing in the modern kernel entry path?
Walk me through what SWAPGS does and why it became a security-critical instruction after Spectre. What is the LFENCE doing in the modern kernel entry path?
- What SWAPGS does mechanically. It atomically swaps the GS base register between the user-mode value (TLS pointer) and the kernel-mode value (per-CPU data pointer). On entry to the kernel, the kernel needs to access per-CPU data structures (current
task_struct, kernel stack pointer, etc.) immediately. Those are addressed viags:offset. SWAPGS makes this work without a separate setup instruction. - Why it is security-critical. Before SWAPGS executes, the GS register still points at user-controlled data. If the CPU speculatively executes a memory access using
gs:offsetbefore SWAPGS retires, it dereferences attacker-controlled memory. The kernel then reads from wherever the user pointed GS, potentially leaking data through cache side channels. - The Spectre v1 SWAPGS variant. Researchers found that the CPU’s speculation engine could speculatively execute the kernel entry path with the wrong GS value, even though architecturally SWAPGS happens first. The speculative reads completed, polluted the cache, and leaked data to a measuring attacker — even though the speculation was eventually discarded.
- The LFENCE mitigation. LFENCE is a load fence — it serializes loads, preventing the CPU from speculatively executing loads after the LFENCE until prior loads complete. Placing
LFENCEimmediately afterSWAPGSguarantees that any subsequentgs:offsetaccess happens with the correct (kernel) GS value. - The performance cost. LFENCE serializes the pipeline; on a modern OoO core that costs roughly 10 to 30 cycles per syscall. Across millions of syscalls per second, this is real overhead. The kernel applies the fence selectively (only on entry, only on architectures that need it) to minimize impact.
TPIDR_EL0 and TPIDR_EL1) so there is no swap operation at the same level. RISC-V’s sscratch register serves a similar role. The x86 design (one GS base, swap on entry) is a relic of the original AMD64 design that became a footgun under speculation.mitigations=off on the kernel command line restores 10 to 30 percent throughput. For multi-tenant systems (cloud, containers from untrusted images), never turn them off.- “SWAPGS is just a privilege transition instruction.” It is more specific than that; it does not change the privilege level (the CPL change from
SYSCALLdoes that). It only swaps a register. The conflation of “kernel transition” with SWAPGS leads to confusion about what each piece actually protects. - “LFENCE prevents Spectre.” LFENCE prevents one specific class of Spectre variants involving speculative loads after a barrier. It does not prevent BTB-based variants (those need retpolines or IBRS).
- Bitdefender’s original SWAPGS variant disclosure (August 2019).
- Linux kernel
arch/x86/entry/entry_64.S— read the actual assembly with comments. - Mark Brand, Project Zero, “Speculative buffer overflows: attacks and defenses” — background on speculation-based attacks.
Interview Deep-Dive
Explain the full lifecycle of a system call from user space to kernel and back. What are the performance implications, and how does the vDSO optimize hot-path calls?
Explain the full lifecycle of a system call from user space to kernel and back. What are the performance implications, and how does the vDSO optimize hot-path calls?
- User-space setup: The C library (glibc) places the syscall number in RAX and arguments in RDI, RSI, RDX, R10, R8, R9. Then it executes the
syscallinstruction. - Hardware transition: The CPU reads the target address from the IA32_LSTAR MSR (set at boot by the kernel), saves the return address in RCX and flags in R11, switches to ring 0, and jumps to
entry_SYSCALL_64. - Kernel entry: The kernel executes
SWAPGSto load the per-CPU kernel data area, saves the user stack pointer, loads the kernel stack, and pushes a full register frame (pt_regs). On systems with KPTI (Kernel Page Table Isolation, the Meltdown mitigation), the kernel must also switch CR3 to the kernel page table, which invalidates TLB entries unless PCIDs are used. - Dispatch: The kernel indexes into
sys_call_table[RAX]and calls the appropriate handler (e.g.,ksys_write()). - Return: Reverse the process — restore registers,
SWAPGSback, switch CR3 if KPTI, executesysretqto return to user space at the address saved in RCX.
clock_gettime(), glibc routes to the vDSO function, which reads a memory-mapped time value updated by the kernel’s timer interrupt — no mode switch at all. Cost drops from 200 cycles to about 20 cycles. The functions typically available via vDSO are gettimeofday, clock_gettime, getcpu, and time.The key insight for interviews: not every “system call” is actually a system call. The vDSO makes some of the most frequently called functions essentially free, which is why you do not see them in strace output — strace only intercepts actual syscall instructions.Follow-up: If vDSO runs in user space with kernel data, how does the kernel keep the time value up to date without a race condition?The kernel uses a seqlock pattern. The vDSO page contains a sequence counter and the time data. The kernel’s timer interrupt updates the time data and increments the sequence counter (odd during write, even when stable). The vDSO reader code loops: read the sequence counter, read the time, read the sequence counter again. If the counter changed or was odd, retry. This guarantees the reader always gets a consistent snapshot without any locks or atomic instructions on the read path. The retry is almost never needed because the timer interrupt is very brief.What are the three core purposes of an OS -- abstraction, multiplexing, and isolation -- and can you give a concrete example of a production failure caused by a breakdown in each?
What are the three core purposes of an OS -- abstraction, multiplexing, and isolation -- and can you give a concrete example of a production failure caused by a breakdown in each?
-
Abstraction (hiding hardware complexity): The OS presents uniform interfaces (files, sockets, processes) regardless of underlying hardware. A production failure from broken abstraction: a cloud provider migrated VMs from Intel to AMD hosts. Applications using
RDTSCdirectly (bypassing the OS clock abstraction) started producing incorrect timestamps because TSC behavior differs between CPU vendors. The fix was to useclock_gettime()(the proper OS abstraction) instead of raw hardware instructions. Lesson: when you bypass the OS abstraction, you take on hardware portability risk. - Multiplexing (sharing resources among competing users): The OS divides CPU, memory, I/O, and network among processes. A production failure from broken multiplexing: a noisy neighbor on a shared Kubernetes node consumed all available I/O bandwidth (no blkio cgroup limits were set). The database on the same node saw query latency spike from 5ms to 500ms because its fsync calls were queued behind the neighbor’s bulk writes. The OS was multiplexing the I/O device fairly by default (CFQ scheduler), but “fair” meant the database got equal share, not prioritized share. Fix: set blkio cgroup weights and move latency-sensitive workloads to dedicated nodes.
- Isolation (preventing interference between processes): The OS ensures one process cannot corrupt another’s memory or resources. A production failure from broken isolation: the Meltdown vulnerability (2018) showed that speculative execution could leak kernel memory to user space, breaking the fundamental isolation between kernel and user. A malicious process could read passwords, encryption keys, and other secrets from kernel memory at roughly 500KB/s. The fix (KPTI) restored isolation but cost 5-30% performance on syscall-heavy workloads. This is arguably the most expensive isolation failure in computing history.
A junior engineer asks: 'If system calls are slow, why doesn't the kernel just run everything in user space?' How would you explain the necessity of the kernel/user-space boundary?
A junior engineer asks: 'If system calls are slow, why doesn't the kernel just run everything in user space?' How would you explain the necessity of the kernel/user-space boundary?
- Mutual distrust: Your web browser, your editor, and a random npm package you installed all run as user-space processes. None of them should be able to read each other’s memory, delete each other’s files, or monopolize the CPU. The kernel/user-space boundary, enforced by hardware (CPU privilege rings), is the mechanism that makes this isolation real. Without it, any process could overwrite any other process’s memory with a simple pointer dereference.
- Hardware protection requires privilege: Certain operations — modifying page tables, programming the interrupt controller, accessing I/O ports, halting the CPU — would allow a single process to break the entire system if performed incorrectly. The hardware restricts these to ring 0 (kernel mode). The kernel acts as a trusted intermediary that validates requests before executing privileged operations.
- Resource accounting: The kernel tracks who owns what — which process has which memory pages, file descriptors, and CPU time. This accounting is what enables fair scheduling, memory limits, and cgroups. If everything ran in user space with equal privilege, there would be no authority to enforce limits.
- Crash containment: When a user-space program dereferences a NULL pointer, the kernel catches the fault and kills just that process. If that code were running in kernel mode, a NULL dereference would panic the entire system.
Next: CPU Architectures & Microarchitecture →