Documentation Index
Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
Use this file to discover all available pages before exploring further.
The System Boot Process
Booting an operating system is a series of “handoffs” where each stage initializes more hardware and increases the CPU’s capability until the full kernel is in control. Think of it like a relay race: the firmware sprints the first leg, hands the baton to the bootloader, which hands it to the kernel, which finally hands it to userspace. Each runner can only do what the previous runner set up for it — the bootloader cannot load drivers because the kernel has not initialized the driver model yet, and the kernel cannot mount the real root filesystem because the storage driver might live inside initramfs. Understanding this chain of trust and capability is where the “magic” of the hardware-software boundary happens.Mastery Level: Senior Systems Engineer
Key Internals: Reset Vector
Prerequisites: CPU Architectures, Memory Management
Key Internals: Reset Vector
0xFFFFFFF0, CR0/CR4/EFER registers, GDT/IDT layout, Page Table bootstrappingPrerequisites: CPU Architectures, Memory Management
1. The Reset Vector: The CPU’s First Breath
When you press the power button, the CPU is in a state of “Real Mode” (16-bit) but with a twist. It does not start at address0x0000.
- The Address: On x86-64, the CPU begins execution at
0xFFFFFFF0(16 bytes below the 4GB mark). This is analogous to a newborn’s first reflex — it is hardwired, not learned. Every x86 CPU ever made wakes up reaching for this exact address. - The Hidden Base: While in 16-bit mode the address space is normally 1MB, at reset, the Code Segment (CS) register has a hidden base of
0xFFFF0000. Thus,CS:IPpoints to the top of the 4GB space, which is mapped by the motherboard to the Flash ROM containing the BIOS or UEFI.
2. Firmware: BIOS vs. UEFI
The firmware’s job is to perform POST (Power-On Self Test) and find a bootable device. Think of the firmware as the building superintendent who turns on the lights, checks the plumbing, and unlocks the front door before the tenants (the OS) arrive for the day.2.1 Legacy BIOS (Basic Input/Output System)
- Mode: Runs entirely in 16-bit Real Mode.
- Disk Format: Uses MBR (Master Boot Record). The BIOS reads the first 512-byte sector of the disk and jumps to it.
- Limitations: Max 2TB disks, 4 primary partitions, slow interrupt-based I/O.
2.2 Modern UEFI (Unified Extensible Firmware Interface)
- Mode: Runs in 32-bit or 64-bit mode from the start.
- Disk Format: Uses GPT (GUID Partition Table) and a dedicated EFI System Partition (ESP).
- The Protocol: Instead of jumping to a sector, UEFI understands filesystems (FAT32) and loads PE/COFF executables (e.g.,
grubx64.efi).
- Boot Services: Used by the bootloader (e.g., to read files). Terminated once the OS starts.
- Runtime Services: Available even after the OS boots (e.g., setting UEFI variables, NVRAM).
3. The Road to 64-bit: Mode Transitions
The most complex part of the boot process is transitioning the CPU from its 16-bit legacy state to full 64-bit Long Mode.Step 1: Real Mode to Protected Mode (32-bit)
Think of this transition as upgrading from a bicycle to a car — you gain speed and capability, but you need to learn new rules of the road first (segment selectors instead of direct segment registers).- Disable Interrupts:
cli(Clear Interrupt Flag) to prevent interrupts before the IDT is ready. An interrupt arriving now would jump to an undefined handler and triple-fault. - Enable A20 Gate: A legacy hack to allow addressing above 1MB. Without it, address bit 20 is forced to zero, and any address above 1MB wraps around.
- Load GDT: The Global Descriptor Table defines how memory segments work. The CPU cannot enter Protected Mode without one — it would not know the privilege level or limits of any segment.
- Set CR0.PE: Set the Protection Enable bit in the
CR0register. This single bit-flip changes how the CPU interprets every subsequent memory access. - Far Jump: A special jump that flushes the CPU pipeline and loads the new 32-bit Code Segment. This is necessary because the pipeline still contains instructions decoded under Real Mode rules.
Step 2: Protected Mode to Long Mode (64-bit)
This is upgrading from the car to a jet — the CPU gains a vastly larger address space (256 TB virtual), but the runway checklist is strict: paging is mandatory, and the CPU will refuse to take off without it.- Set CR4.PAE: Physical Address Extension is required for 64-bit mode. PAE widens page table entries from 32 to 64 bits, which is a prerequisite for the wider address space.
- Setup Page Tables: You must enable paging to enter Long Mode. The kernel builds a minimal “Identity Map” (Virtual Address = Physical Address) for the first 1GB of memory. Without this, the very next instruction fetch after enabling paging would fail — the CPU would try to translate the instruction pointer through page tables that do not map it.
- Set EFER.LME: Set the Long Mode Enable bit in the Extended Feature Enable Register. This arms the transition but does not activate it yet.
- Set CR0.PG: Enable Paging. Combined with EFER.LME, the CPU is now in “Compatibility Mode” — a transitional state that runs 32-bit code under 64-bit paging.
- Final Far Jump: A jump to a 64-bit code segment officially enters Long Mode. The pipeline flush ensures the CPU starts decoding 64-bit instructions.
4. Kernel Data Structures: GDT and IDT
The kernel must define how it will handle memory and interrupts before it can do anything else.4.1 The GDT (Global Descriptor Table)
The GDT is an array of 8-byte descriptors. Even in “Flat” 64-bit mode where segmentation is mostly unused, the GDT is required to define:- Kernel Code Segment: Rings 0, Executable, Readable.
- Kernel Data Segment: Rings 0, Readable, Writable.
- User Code/Data Segments: Rings 3.
- TSS (Task State Segment): Points to the stack to use when an interrupt occurs.
4.2 The IDT (Interrupt Descriptor Table)
The IDT maps interrupt vectors (0-255) to handler functions.- Vectors 0-31: Reserved for CPU exceptions (Divide by Zero, Page Fault, etc.).
- Vectors 32-255: Available for hardware interrupts and system calls.
- Gate Types: Interrupt Gates (clear IF), Trap Gates (don’t clear IF).
5. The Kernel Entry Sequence (Linux)
Once the bootloader (GRUB) loads the kernel into memory, it jumps to the kernel’s entry point.5.1 Decompression (head_64.S)
The Linux kernel is actually a self-extracting executable (vmlinuz — the “z” literally stands for “zipped”). Think of it like a zip file that contains its own unzip utility at the front.
- The early code decompresses the “real” kernel image into a higher memory address. Modern kernels use LZ4 or ZSTD for fast decompression.
- It sets up a temporary stack — just enough to run C code.
- It jumps to the decompressed kernel’s entry point, leaving the compressed image behind.
5.2 The start_kernel() Function
This is the “Big Bang” of the operating system. It is architecture-independent C code (finally, after pages of assembly) that:
setup_arch(): Handles CPU-specific initialization — detecting features, calibrating timers, building the memory map from firmware-provided tables (E820 or UEFI memory map).mm_init(): Initializes the full Buddy Allocator and Slab Allocator. Before this point, memory allocation is done through a crude “memblock” early allocator.sched_init(): Sets up the scheduler and the “Idle” task (the task that runs when there is nothing else to do — it puts the CPU into a low-power halt state).rest_init(): Spawns Process 1 (init) and Process 2 (kthreadd). From this moment, the system is running real processes with a real scheduler.
start_kernel() does and how long each step takes, boot with initcall_debug on the kernel command line. The kernel will print timestamps for every initialization function, making it straightforward to identify slow subsystem init.
6. The Handover to User Space
6.1 Initramfs (Initial RAM Filesystem)
The kernel cannot mount the real root disk immediately — it is a chicken-and-egg problem: the storage driver needed to read the disk might itself be on the disk. Initramfs solves this by giving the kernel a tiny “starter kit” already in memory. Think of it like a toolbox that a mechanic brings to a broken-down car. The car has all the tools in the trunk, but you cannot open the trunk until the car is running. The toolbox has just enough to get the engine started.- The bootloader loads a small CPIO archive into memory (initrd/initramfs).
- The kernel mounts this as
/. - It runs
/initfrom the ramfs, which loads necessary drivers (NVMe, RAID, LVM, LUKS encryption) and finally “Switches Root” to the real disk.
lsinitrd (on Fedora/RHEL) or lsinitramfs (on Debian/Ubuntu). If your system fails to boot with “unable to mount root fs,” the most common cause is a missing storage driver in the initramfs. Rebuild it with dracut or update-initramfs.
6.2 PID 1: systemd / SysV init
The final step is to execute the first user-space process — the “root of the process tree.” Every process on the system is a descendant of PID 1.- Path:
/sbin/init(or whateverinit=kernel parameter specifies). On modern distributions this is almost alwayssystemd. - The PID 1 Rule: This process is the ancestor of all others. If it ever exits, the kernel triggers a Kernel Panic. This is by design: PID 1 is responsible for reaping orphaned child processes. Without it, zombie processes would accumulate with no parent to collect their exit status.
init=/bin/bash on the kernel command line. The kernel will drop you into a root shell instead of starting systemd, letting you fix configuration issues, repair filesystems, or reset passwords. Remember to exec /sbin/init when done to start the system normally.
7. Interview Deep Dive: Senior Level
What is the 'A20 Gate' and why is it still relevant?
What is the 'A20 Gate' and why is it still relevant?
On the original 8086, memory addresses wrapped around at 1MB. When the 80286 arrived, it could address more, but some old programs relied on the wrap-around bug. IBM added a gate on the 20th address line (A20) to manually enable/disable the wrap-around. Even today, x86 CPUs start with A20 disabled for compatibility, and the bootloader must enable it to access more than 1MB of RAM.
How does the kernel find its own code in memory before paging is enabled?
How does the kernel find its own code in memory before paging is enabled?
The kernel is compiled as a Position Independent Executable (PIE) or uses Relative Addressing. Early boot code uses the Instruction Pointer (RIP) relative instructions to find data. Once the kernel sets up the initial page tables and enables paging, it can “jump” into the virtual address space it has defined.
Explain the 'Identity Mapping' during boot.
Explain the 'Identity Mapping' during boot.
When the kernel enables paging (setting
CR0.PG), the CPU immediately begins interpreting all addresses as virtual. If the kernel didn’t “Identity Map” (Map Virtual 0x1234 to Physical 0x1234) the code it is currently executing, the very next instruction fetch would fail because the MMU wouldn’t know where to find the code, resulting in an immediate crash.8. Advanced Practice
- GDT Inspector: Use
gdband QEMU (-s -S) to inspect the GDT of a booting kernel. Use the commandmonitor info gdt. - Early Printk: Add a
printk("Hello from early boot!");to thestart_kernelfunction in a Linux source tree and compile it. Observe when the message appears during boot. - UEFI Shell: Boot into a UEFI shell and use the
lsandmapcommands to see how the firmware sees your disks and partitions.
9. Checklist: From Power-On to Login Prompt
Use this quick reference to understand (or debug) the complete boot sequence:Phase-by-Phase Checklist
Debugging Boot Issues
| Symptom | Phase | Debug Command |
|---|---|---|
| No display, fans spin | Firmware | Check POST codes, clear CMOS |
| ”No bootable device” | Firmware/ESP | efibootmgr -v, check ESP mount |
| GRUB rescue prompt | Bootloader | Boot from live USB, reinstall GRUB |
| Kernel panic early | Kernel early | Add debug to kernel cmdline |
| initramfs drops to shell | Initramfs | Check /dev, load missing drivers |
| systemd fails | Init system | systemctl --failed, journalctl -b |
Quick Boot Time Analysis
Next: Memory Management & Allocators →
Production Caveats: Where the Boot Sequence Bites Real Systems
Boot is the part of the stack with the worst observability. Logs do not exist yet. Console output may not redirect anywhere. The kernel is half-initialized. Everything assumes the previous stage worked, and when it does not, you get a blinking cursor with no diagnostic.Senior Interview Questions: Boot Process Mastery
BIOS versus UEFI -- what changed and why? Be specific about the security and capability differences, not just 'UEFI is newer.'
BIOS versus UEFI -- what changed and why? Be specific about the security and capability differences, not just 'UEFI is newer.'
Strong Answer Framework:Common Wrong Answers:
- The architectural difference. BIOS is a 16-bit Real Mode firmware that boots a 512-byte sector via the MBR. UEFI is a 32 or 64-bit firmware that runs PE/COFF executables (
.efifiles) from the FAT-formatted EFI System Partition. The shift is from “execute a sector” to “load and execute a program.” - Disk capacity. BIOS with MBR caps disks at 2.2TB (32-bit LBA). UEFI with GPT supports disks up to 9.4 zettabytes and 128 partitions natively. This alone forced the migration — enterprise storage demanded GPT a decade ago.
- Boot speed. UEFI can boot in parallel: device init, driver loading, and bootloader load can overlap. BIOS boots strictly serially through INT calls. A modern UEFI server boots firmware-to-bootloader in 5 to 15 seconds; equivalent BIOS would be 30 to 60. Some of this is actually firmware quality, but UEFI’s architecture enables it.
- Security. UEFI has Secure Boot: signed-image verification before execution. BIOS has nothing comparable — any sector at LBA 0 with the magic bytes runs. Secure Boot enables a measurable trust chain (firmware verifies bootloader, bootloader verifies kernel, kernel verifies modules). For server workloads handling regulated data, this is increasingly mandatory.
- Programming model. UEFI provides Boot Services (filesystem access, network, graphics) and Runtime Services (NVRAM variables, time, reset). Bootloaders can be full C programs that call these services. BIOS only had INT calls and 16-bit conventions; everything was assembly or near-assembly.
- The migration cost. UEFI brought new failure modes: corrupted NVRAM (
efivars) bricking machines, signed-blob mistakes locking out admins, BootHole-style attacks on bootloader code that does file parsing. The complexity is real.
Senior follow-up: What does the “Compatibility Support Module” (CSM) do, and why is it being phased out? CSM is a UEFI shim that emulates BIOS — letting a UEFI firmware boot a legacy BIOS-style MBR. Useful during the migration era, but it is a huge attack surface (basically running 16-bit Real Mode code from the firmware). Intel mandated CSM removal from new platforms starting in 2020. Modern firmware is UEFI-only; if you have Windows 7 disks, they will not boot.
Senior follow-up: Is UEFI’s complexity worth it for an embedded system or appliance? Often not. For a single-purpose appliance (router, embedded controller), U-Boot or a direct flash bootloader is simpler, smaller, and easier to audit. UEFI shines when you need a flexible boot environment (multi-OS, network boot, recovery) on commodity hardware. For a static workload, simpler is better.
Senior follow-up: What is the smallest credible attack surface on a UEFI system? The firmware update mechanism. Most UEFI firmware can be reflashed via the OS (CapsuleUpdate). A kernel-level attacker can persist by reflashing firmware with a backdoor. Mitigations: Boot Guard (Intel), Platform Secure Boot (PSB on AMD), and signed firmware updates. None of these eliminate the risk entirely; all of them require correct OEM implementation.
- “UEFI is faster because BIOS is single-threaded.” Misleading. Boot speed is dominated by hardware init (DRAM training, PCIe link training), not by the firmware language. UEFI’s parallelism helps but is not the dominant factor.
- “Secure Boot makes systems unhackable.” Secure Boot only protects the boot path. After the OS loads, normal kernel exploits still apply. Secure Boot is one layer, not the answer.
- UEFI Specification, version 2.10 (uefi.org) — the formal reference.
- Adam Williamson’s “UEFI Boot for Dummies” series on the Fedora wiki.
- Eclypsium’s BootHole technical write-up (2020) — shows what an actual UEFI attack looks like.
How does Secure Boot work end to end, and what attacks does it prevent versus not prevent?
How does Secure Boot work end to end, and what attacks does it prevent versus not prevent?
Strong Answer Framework:Common Wrong Answers:
- The trust hierarchy. Four NVRAM databases: PK (Platform Key, single, OEM-controlled), KEK (Key Exchange Keys, allow updating db/dbx), db (allowlist of trusted signing certificates), dbx (denylist of revoked signatures). The firmware will only execute boot-stage binaries whose signature chains to a cert in db, unless the binary’s hash is also in dbx (which overrides).
- The chain of trust at boot. UEFI firmware verifies
shimx64.efiagainst db (signed by Microsoft’s UEFI CA). shim contains the distro’s MOK (Machine Owner Key) and verifiesgrubx64.efiagainst MOK. GRUB verifies the kernel against MOK or kernel keyring. Modern kernels then verify their own modules against the kernel keyring. Each link breaks if its signature is invalid. - What Secure Boot prevents. Bootkit malware (replacing the bootloader to persist below the OS), unauthorized OS substitution (booting a malicious kernel that masquerades as the legitimate one), evil-maid attacks (briefly accessing a machine to install a tampered bootloader). Combined with TPM-sealed disk encryption, also prevents extracting data from a stolen disk.
- What Secure Boot does not prevent. Vulnerabilities in signed bootloaders (BootHole was a buffer overflow in
grub.cfgparsing — the GRUB binary was correctly signed, but it processed attacker-controlled data unsafely). User-space malware (Secure Boot stops here once the kernel is loaded). Kernel exploits (privilege escalation, container escapes). Hardware attacks (DMA, JTAG, reflashing the firmware itself with a flasher). Physical-layer attacks (cold boot RAM extraction). - The dbx update problem. When a vulnerability is found in a signed bootloader, the response is to add the bootloader’s hash to dbx. But if the same bootloader is installed across millions of machines, mass-revoking it disrupts everything. Vendors must coordinate revocations carefully, ship updated bootloaders first, then revoke old ones. The lag from disclosure to safe revocation is often months to years.
Senior follow-up: What is the difference between Secure Boot and Measured Boot? Secure Boot rejects untrusted binaries at load time. Measured Boot computes hashes of every loaded component into TPM PCRs (Platform Configuration Registers) but does not gate execution. Measured Boot enables remote attestation: a server can request the PCR values, verify them against expected hashes, and trust the system only if they match. Secure Boot prevents bad code from running; Measured Boot proves to a third party what code actually ran.
Senior follow-up: If a kernel is signed but vulnerable, does Secure Boot help? No. Secure Boot validates signatures, not code correctness. A signed kernel with a privilege escalation bug runs just fine until the bug is exploited. The defense in depth here is “kernel hardening” (CFI, KASLR, page table isolation) on top of Secure Boot, not Secure Boot alone.
Senior follow-up: Can I use Secure Boot with a custom kernel? Yes, via the MOK enrollment path: generate your own key with
openssl, sign your kernel with sbsign, enroll the public key into the MOK database with mokutil --import. From the next boot, your custom kernel passes verification. The catch is that mokutil requires a physically-present user to confirm enrollment via the MOK Manager UI — this is anti-evil-maid by design, but it makes automated provisioning harder.- “Secure Boot prevents all malware.” No. It prevents pre-OS malware. Anything running after the kernel loads is unaffected.
- “You should disable Secure Boot if you run Linux.” Linux supports Secure Boot via shim. There is no reason to disable it. Disabling it removes a security layer for no benefit.
- UEFI Specification 2.10, chapter 32 (Secure Boot and Driver Signing).
- Matthew Garrett’s blog series on shim and Secure Boot for Linux (mjg59.dreamwidth.org).
- Eclypsium’s BlackLotus analysis (2023) — a practical bootkit bypass and how dbx responded.
A production server hangs at boot with no console output. Walk me through diagnosis when you cannot get console access.
A production server hangs at boot with no console output. Walk me through diagnosis when you cannot get console access.
Strong Answer Framework:Common Wrong Answers:
- Confirm hardware is alive. Power LED on, fans spinning, drive activity? If not, this is below the boot stack — power supply, motherboard, or PSU. If yes, hardware is at least running through POST.
- Get serial console access if you do not have it. On servers, IPMI/iDRAC/iLO almost always has a virtual serial console. On cloud instances, AWS has “EC2 Serial Console” (must be enabled per region), GCP has Serial Console API, Azure has Boot Diagnostics. If none of these are available, the next step is reseating drives in another machine to read logs from disk after the fact.
- Read POST codes if available. Server motherboards have a 2-digit hex display showing the POST code — the firmware’s exact stage. Decode against the OEM’s POST code table. “Memory training failed” looks very different from “no bootable device” at this layer.
- Check for storage failure. If POST passes but the firmware reports “no bootable device,” either the disk is dead, the partition table is corrupted, or the bootloader was overwritten. Boot from rescue media (USB, network), check
lsblk,fdisk -l, runsmartctl -aon the disk. - If the kernel starts but does not finish. Boot with
nomodeset earlyprintk=serial,ttyS0,115200 console=ttyS0,115200n8 debugon the kernel command line. This gives early console output to serial, which IPMI captures. The output reveals which subsystem is failing: hung onudev, hung onmounting root, panic oninit. - If everything looks fine but no login prompt. Boot with
init=/bin/bashto bypass systemd entirely. If you get a shell, the bug is in user space (probably a service hanging on something like a missing network mount). Boot withsystemd.unit=rescue.targetfor a more controlled diagnostic. - If the bug is reproducible in QEMU. This is the gold scenario. Reproduce locally, attach
gdbto QEMU’s stub (-s -S), and step through the boot. Most production boot bugs are reproducible this way.
intel_iommu=off to confirm the IOMMU was at fault, then a coordinated rollback. Total outage was 27 minutes; without IPMI access it would have been hours.Senior follow-up: What if you cannot get serial console access at all? Boot a rescue image, mount the failing disk, and read
/var/log/journal/*/system.journal using journalctl -D /mnt/var/log/journal --boot=-1 to see logs from the previous boot attempt. Many systemd failures leave breadcrumbs even when the machine never came up.Senior follow-up: How do you avoid this entire class of problem in a fleet? Staged kernel rollouts. Update one host, validate it boots cleanly, then 1 percent, then 10 percent, then 100. Capture serial logs from every host as it boots. Anomaly-detect on boot duration — if a kernel suddenly takes 60 seconds to boot when it usually takes 15, that is a regression worth catching before fleet-wide deployment.
Senior follow-up: What boot debug technique do most engineers not know?
kdb (Kernel Debugger) or kgdb (Kernel GDB). Compiled into many distro kernels but disabled by default. Enable with kgdboc=ttyS0,115200 and kgdbwait on the kernel command line; the kernel will halt at the very start of start_kernel() waiting for a gdb connection over serial. From there you can step through any subsystem init.- “Just reinstall the OS.” Diagnostic abandonment. You will reproduce the same bug next week. Find the root cause.
- “Roll back to the previous kernel.” Right tactical move for an incident, wrong long-term answer. After the rollback, you must reproduce in a lab and find the actual fix, or you will be stuck on the old kernel forever.
- Linux kernel
Documentation/admin-guide/serial-console.rst. - Brendan Gregg, “Linux boot bottlenecks” (brendangregg.com).
- The Linux Programming Interface, Michael Kerrisk, chapter 25 (Process Termination) and chapter 37 (Daemons) for systemd-adjacent debug techniques.
Interview Deep-Dive
Your production server is stuck at the GRUB rescue prompt after a kernel upgrade. Walk me through how you would diagnose and recover it.
Your production server is stuck at the GRUB rescue prompt after a kernel upgrade. Walk me through how you would diagnose and recover it.
Strong Answer:
- The GRUB rescue prompt means the bootloader cannot find its configuration or the kernel image. The most common cause after a kernel upgrade is that the /boot partition ran out of space, the grub.cfg was not regenerated, or the symlinks to vmlinuz/initramfs point to a version that was never installed properly.
- My first step would be to boot from a live USB or rescue image. Then I mount the root and boot partitions and check whether /boot actually has the new kernel and initramfs files. I would look at
ls /boot/and compare against what grub.cfg references. - If the files are there, I chroot into the system, re-run
grub-mkconfig -o /boot/grub/grub.cfgandgrub-install /dev/sda(or the appropriate disk). Then reboot. - If /boot is full, I remove old kernels first. On Debian-based systems,
apt autoremovehandles this; on RHEL,dnf removewith the old kernel package names. - A subtlety people miss: on UEFI systems, the EFI System Partition (ESP) is separate from /boot. You need to check whether the .efi file on the ESP is intact and whether the UEFI boot entry still references the correct path.
efibootmgr -vfrom the rescue environment is the tool for that.
grub-install from a rescue environment, while Stage 2 or config issues may only need grub-mkconfig. On UEFI, the chain is simpler since shim.efi loads grubx64.efi directly from the ESP filesystem, so you can often just copy a known-good .efi binary back onto the ESP.Explain Secure Boot from a trust-chain perspective. If I compile my own custom kernel, what breaks, and how do I fix it?
Explain Secure Boot from a trust-chain perspective. If I compile my own custom kernel, what breaks, and how do I fix it?
Strong Answer:
- Secure Boot enforces a chain of trust: the UEFI firmware validates each binary it loads against a signature database (db) stored in NVRAM. The typical chain is UEFI firmware verifies shim.efi (signed by Microsoft), shim verifies grubx64.efi (signed by the distro’s Machine Owner Key), GRUB verifies the kernel (also MOK-signed), and optionally the kernel verifies its own modules.
- If I compile a custom kernel, it will not be signed by any key in the db or MOK databases. Secure Boot will refuse to load it and the system will not boot.
- The fix is to enroll my own MOK using
mokutil --import my_key.der, reboot into the MOK Manager (a shim-provided UI), accept the key, then sign my kernel withsbsign --key my_key.key --cert my_key.crt --output vmlinuz-signed vmlinuz. From that point, shim will accept my signed kernel. - In a production fleet, the better approach is to disable Secure Boot only on dev machines, and for production, use a CI pipeline that signs kernels with a hardware security module (HSM) holding the organization’s MOK private key.
- The gotcha most people miss: even if the kernel loads, unsigned kernel modules (like out-of-tree drivers from DKMS) will be rejected at module load time if the kernel enforces module signature verification. You need to sign those too, or add
module.sig_enforce=0to the kernel command line (which weakens the trust chain).
What is initramfs, and why can't the kernel just mount the root filesystem directly? When would you need to customize it?
What is initramfs, and why can't the kernel just mount the root filesystem directly? When would you need to customize it?
Strong Answer:
- The kernel, even with all drivers compiled in, cannot always mount the root filesystem directly because the root device might require drivers that are not compiled into the kernel — they are loaded as modules. Think of NVMe drivers, RAID controller drivers, LVM userspace tools, or LUKS encryption decryption. All of these need to run before the real root is accessible.
- initramfs (initial RAM filesystem) is a small CPIO archive loaded into memory by the bootloader alongside the kernel. The kernel mounts it as a temporary root filesystem and executes /init from it. That init script loads the necessary drivers, assembles RAID arrays, decrypts LUKS volumes, activates LVM, and then calls
switch_rootto pivot to the real root filesystem. - You would customize initramfs when: you add a new storage driver the default initramfs does not include, you use full-disk encryption with a non-standard key derivation (like a network-fetched key or a hardware token), you need to add custom network configuration for NFS root or iSCSI boot, or you want to add a custom pre-boot recovery shell.
- On Debian/Ubuntu,
update-initramfs -uregenerates it; on Fedora/RHEL,dracut -fdoes the same. A common production incident is forgetting to regenerate initramfs after installing a new kernel module that the root device depends on — the system boots fine on the old kernel but panics on the new one because the module is missing from the new initramfs.
During boot, the CPU transitions from 16-bit Real Mode to 64-bit Long Mode. Why can't it jump directly, and what would happen if you skipped the intermediate Protected Mode step?
During boot, the CPU transitions from 16-bit Real Mode to 64-bit Long Mode. Why can't it jump directly, and what would happen if you skipped the intermediate Protected Mode step?
Strong Answer:
- The x86 architecture requires a specific sequence of mode transitions because each mode depends on data structures that the previous mode sets up. You cannot jump from Real Mode to Long Mode because Long Mode requires paging to be enabled, and paging requires a valid page table hierarchy to be set up in memory. But in Real Mode, you cannot address enough memory to set up those tables conveniently, and the GDT (Global Descriptor Table) that defines the code and data segments for protected execution has not been loaded yet.
- The sequence is: Real Mode (16-bit, 1MB addressable) then enable A20 gate, load GDT, set CR0.PE bit, far jump to flush pipeline, now in 32-bit Protected Mode, set up initial page tables (identity mapping), enable PAE (CR4.PAE), set EFER.LME (Long Mode Enable), enable paging (CR0.PG), far jump to 64-bit code segment, now in Long Mode.
- If you tried to set EFER.LME and CR0.PG from Real Mode, the CPU would triple-fault and reset because the required structures (GDT with valid 64-bit code segment, valid page tables loaded in CR3) would not exist. The CPU would attempt to fetch instructions through paging with no valid page tables and immediately fault, then fault again trying to handle the fault (double fault), then fault again (triple fault), triggering a reset.
- The practical consequence: if you are writing a bootloader or early kernel code, you must follow this exact sequence. UEFI firmware abstracts this away for you — by the time your .efi application runs, the CPU is already in 64-bit mode with paging enabled. This is one of the biggest advantages of UEFI over legacy BIOS boot.