System calls are the only legitimate way for user-space programs to request services from the kernel. Understanding syscalls deeply is essential for observability engineers who trace application behavior and infrastructure engineers who debug performance issues.
Interview Frequency: Very High (especially at observability companies) Key Topics: syscall mechanism, vDSO, seccomp, overhead analysis Time to Master: 12-14 hours
When an application makes a system call, the CPU must transition from user mode (Ring 3) to kernel mode (Ring 0). This is a privileged operation that involves:
Saving user context: All registers are saved so we can return to exactly where we left off
Switching stacks: User stack → Kernel stack (each process has both)
Changing privilege level: Ring 3 → Ring 0 (CPU enforces this)
Executing kernel code: The actual syscall handler runs
Returning to user mode: Restore context and switch back to Ring 3
This transition is expensive (200-500 CPU cycles) because of security checks, context switching, and cache effects. Understanding this overhead is crucial for writing performant systems.
// These MSRs are set during boot:// MSR_LSTAR (0xC0000082) - Syscall entry point addresswrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);// MSR_STAR - Segment selectors for syscall/sysret// MSR_SYSCALL_MASK - Flags to clear on syscall
Before vDSO, getting the current time was surprisingly expensive:The problem: Applications call gettimeofday() or clock_gettime() millions of times per second:
Web servers log every request with timestamps
Databases track transaction times
Profilers measure code execution
Games render frames with timing
The cost: Each call was a full syscall (~200-500 cycles overhead) just to read a number that the kernel updates periodically anyway.The insight: Time is read-only data that changes slowly (milliseconds). Why context switch to read it?The solution: vDSO maps kernel data into user space. Applications read time directly from memory - no syscall needed!vDSO is a kernel optimization that provides certain syscalls without entering kernel mode:
Copy
┌─────────────────────────────────────────────────────────────────────────────┐│ vDSO MECHANISM │├─────────────────────────────────────────────────────────────────────────────┤│ ││ Without vDSO (slow): ││ ┌────────────┐ syscall ┌────────────┐ ││ │ User Space │ ──────────────→ │ Kernel │ ││ │ │ ←────────────── │ │ ││ └────────────┘ return └────────────┘ ││ ~100-200 cycles overhead ││ ││ With vDSO (fast): ││ ┌────────────┐ function call ┌────────────┐ ││ │ User Space │ ──────────────→ │ vDSO │ (still user space!) ││ │ │ ←────────────── │ (shared) │ ││ └────────────┘ return └────────────┘ ││ ~10-20 cycles (no mode switch) ││ │└─────────────────────────────────────────────────────────────────────────────┘
// Benchmark: clock_gettime performance#include <time.h>void benchmark() { struct timespec ts; // This uses vDSO (~20ns) for (int i = 0; i < 1000000; i++) { clock_gettime(CLOCK_MONOTONIC, &ts); }}
Interview Insight: “gettimeofday/clock_gettime are the most frequently called syscalls in many applications. vDSO makes them essentially free, which is why you rarely see them in syscall traces as performance problems.”
Syscalls are one of the most expensive operations you can do in user space. Here’s why:The fundamental problem: CPU privilege levels. User code runs in ring 3 (unprivileged), kernel code runs in ring 0 (privileged). Switching between them is expensive.What makes it expensive:
Mode switch: CPU must save all registers, switch stacks, change privilege level
Security checks: Validate arguments, check permissions, run seccomp filters
Cache pollution: Kernel code evicts user code from CPU caches
Spectre mitigations: KPTI adds extra overhead (page table switching)
Real-world impact: A program doing 100,000 syscalls/second spends 2-5% of CPU time just on syscall overhead, before doing any actual work.Understanding syscall overhead is crucial for performance:
Containers provide isolation, but they share the same kernel. This creates a security risk:The threat: A compromised container could:
Use ptrace() to inspect other processes
Use mount() to escape the container
Use reboot() to crash the host
Use kexec_load() to replace the kernel
Use clock_settime() to break time-based security
The challenge: Containers need some syscalls to function, but not all ~300+ syscalls.The solution: seccomp-BPF filters syscalls before they execute. Even if an attacker gains code execution in a container, dangerous syscalls are blocked at the kernel level.seccomp-BPF allows filtering syscalls for security:
Docker uses seccomp to restrict container syscalls:
Copy
# View Docker's default seccomp profiledocker info --format '{{ .SecurityOptions }}'# Run container with custom seccomp profiledocker run --security-opt seccomp=profile.json myimage# Run without seccomp (less secure)docker run --security-opt seccomp=unconfined myimage
For production, use eBPF-based tracing (covered in Track 5):
Copy
# Using bpftracesudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s opened %s\n", comm, str(args->filename)); }'# Using perfsudo perf trace ls
// Different syscall tables for different ABIs// arch/x86/entry/syscall_64.c - 64-bit// arch/x86/entry/syscall_32.c - 32-bit compat// Check ABI in seccompstruct seccomp_data { __u32 arch; // AUDIT_ARCH_X86_64 or AUDIT_ARCH_I386 ...};
