Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Virtual File System (VFS)

The File System is the primary abstraction for long-term storage. Think of VFS as a universal power adapter: your laptop (user-space application) has one plug (open, read, write), and the adapter (VFS) lets it work with any outlet in the world (EXT4, XFS, NFS, procfs, tmpfs). Without VFS, every application would need to know the specifics of every filesystem format — an impossibility. A senior understanding requires knowing how the kernel provides this unified interface across hundreds of different disk formats and how it optimizes the expensive process of finding a file on disk.

​
0. Files, Inodes, and Paths: The Basics

Before diving into VFS internals, make sure these fundamentals are crystal clear.

​
What is a File?

A file is an abstraction the OS provides over raw disk blocks. To the user, a file is:
  • A name (like report.txt).
  • A sequence of bytes (the content).
  • Some metadata (size, permissions, timestamps).
To the kernel, a file is not the name—the name is just a pointer.

​
What is an Inode?

An Inode (Index Node) is the kernel’s internal representation of a file. It contains:
  • Metadata: size, owner (UID/GID), permissions, timestamps (atime, mtime, ctime).
  • Block pointers: Where on the disk the file’s data actually lives.
  • No filename: The inode does not store the file’s name.
Think of an inode as a library catalog card: it tells you everything about the book (author, subject, shelf location) but not what the book is called. The book’s “name” is written on the shelf label (the directory entry), not on the catalog card (the inode). This separation is why you can have multiple names (hard links) pointing to the same file — multiple shelf labels pointing to the same catalog card.

​
What is a Directory?

A directory is a special file that maps names → inode numbers. When you ls a directory, the kernel:
  1. Reads the directory’s data blocks.
  2. Lists all (name, inode) pairs inside.
This is why renaming a file within the same filesystem is instant: only the directory entry changes, not the file’s data or inode.
  • Hard link: Another name pointing to the same inode. Both names are equal; deleting one doesn’t delete the data until the link count reaches zero.
  • Soft link (symlink): A file whose content is the path to another file. It can break if the target is deleted.

​
Path Resolution

When you open("/home/user/file.txt", ...):
  1. Kernel starts at the root inode (/).
  2. Looks up home in /’s directory → gets inode for /home.
  3. Looks up user in /home → gets inode for /home/user.
  4. Looks up file.txt in /home/user → gets inode for the file.
  5. Returns a file descriptor referencing that inode.
This is the path walk, and optimizing it (via the dentry cache) is one of the kernel’s most critical jobs.

​
1. The VFS Architecture

The Virtual File System (VFS) is a software layer in the kernel that provides the standard file-related system calls (open, read, write) to user-space, regardless of the underlying filesystem (EXT4, XFS, NFS, ProcFS).

​
The Four Primary Objects

  1. Superblock: Represents an entire mounted filesystem. It contains metadata like the block size, total number of inodes, and the “magic number.”
  2. Inode (Index Node): Represents a specific file (or directory) on disk. It contains all metadata (size, permissions, timestamps, block pointers) but not the filename.
  3. Dentry (Directory Entry): Connects an Inode to a filename. Dentries are transient objects created in memory to speed up path lookups.
  4. File: Represents an open file in a specific process. It stores the current “cursor” position (f_pos) and the access mode (Read/Write).

​
2. Path Lookup: The “Walk”

Converting a string like /home/user/code/main.c into an Inode is one of the most performance-critical paths in the kernel.

​
The Lookup Process

  1. Split the path: Divide by /.
  2. Dentry Cache (dcache) Lookup: The kernel first checks the memory-resident Dcache. If the dentry for home is found, it moves to the next part.
  3. Directory Traversal: If not in cache, the kernel must read the directory’s data blocks from disk to find the mapping from the filename to an Inode number.
  4. Inode Cache: Once the Inode number is found, the kernel checks the Inode cache or reads it from the Inode table on disk.

​
Optimization: RCU-Walk

To handle thousands of concurrent lookups (e.g., on a web server serving static files), modern Linux uses RCU-walk. It performs the path walk without taking any locks, assuming the directory structure will not change. If a change is detected mid-walk, it falls back to the slower “Ref-walk” with proper locking. Think of it like walking through a building with all doors open (RCU-walk). If you find a door closed mid-walk (directory structure changed), you go back and get the key ring (take locks). The optimistic path is dramatically faster because lock acquisition and contention are the primary bottlenecks in high-throughput filesystem workloads. Practical tip: If your application opens thousands of files per second (e.g., npm install, container startup), the dentry cache hit rate is your most important filesystem metric. Check it with cat /proc/sys/fs/dentry-state. A cold dentry cache after a reboot is often the real reason “the first request after deploy is slow.”

​
3. On-Disk Layout & Allocation

How data is actually stored on the physical platters or NAND cells.

​
Block Allocation

  • Extents: Instead of a list of block numbers (which is inefficient for large files), modern filesystems (EXT4, XFS) use Extents—a starting block number and a length (e.g., “Blocks 1000 to 2000”).
  • Delayed Allocation: The kernel buffers writes in memory and waits as long as possible before choosing physical blocks on disk. This allows the allocator to find a single contiguous extent for the entire file, reducing fragmentation.

​
Journaling: Atomic Operations

Writing to a file involves multiple steps: updating the bitmap, the Inode, and the data blocks. If power is lost mid-way, the filesystem becomes inconsistent — like a bank transfer that debited one account but crashed before crediting the other.
  • The Journal: A dedicated circular buffer on disk that acts as a “transaction log.” The kernel first writes the intended changes to the journal (the “intent”). Once the journal write is “committed,” it updates the actual filesystem. On reboot, the kernel simply “replays” the journal to restore consistency — completing any transactions that were committed but not yet applied, and discarding any that were not committed.
Practical tip: EXT4 journals metadata by default (data=ordered), which means file content can be lost on a crash but the filesystem structure stays consistent. If you need both metadata and data journaled, mount with data=journal — but this roughly halves write throughput because every byte is written twice (once to journal, once to final location). Databases bypass this tradeoff entirely by managing their own WAL (Write-Ahead Log) and using O_DIRECT + fsync.

​
4. Log-Structured File Systems (LFS)

Used in Flash/SSDs (e.g., F2FS, and conceptually similar to how SSDs work internally with their Flash Translation Layer).
  • Philosophy: Never overwrite data. Treat the entire disk as a circular log. New writes always go to the end, and old versions of data become garbage.
  • Benefit: Converts random writes into sequential writes, which is significantly faster for NAND flash (where erasing a block before overwriting is expensive) and reduces wear leveling overhead.
  • Trade-off: Requires a Garbage Collector to reclaim space from “dead” versions of files. Under heavy write load, the GC can compete with application I/O for bandwidth — a phenomenon called write amplification. This is the same tradeoff that SSDs face internally with their FTL.
Practical tip: If you are running a database on an SSD, the database’s own log-structured writes plus the SSD’s internal log-structured FTL can cause “double write amplification.” This is why database engineers obsess over aligning write sizes to the SSD’s erase block size and minimizing random small writes.

​
5. Page Cache & Writeback

The filesystem does not talk directly to the disk for every write(). This would be devastatingly slow — a disk write takes milliseconds, but a memory copy takes microseconds. Instead, the kernel absorbs writes into RAM and flushes them lazily.
  1. Write to Page Cache: The write() syscall simply copies data into kernel memory (Pages). The page is marked as Dirty. From the application’s perspective, the write “completed” in microseconds.
  2. Flushing (Writeback): A background kernel thread (bdi_writeback) periodically writes dirty pages to disk. The default dirty ratio is ~20% of RAM — only when dirty pages exceed this threshold does the kernel start aggressively flushing. This is why a sudden burst of writes can feel fast initially but then stall: you have filled the dirty page budget and the kernel is now synchronously writing to disk.
  3. Direct I/O: Applications (like Databases) can use O_DIRECT to bypass the page cache and manage their own buffers. This avoids “Double Buffering” (data sitting in both the application’s buffer pool and the kernel’s page cache) and gives the application precise control over when data hits disk via fsync().
Practical tip: If you see high I/O wait (wa in top) during write bursts, check dirty_ratio and dirty_background_ratio in /proc/sys/vm/. Tuning these values is one of the highest-impact knobs for write-heavy workloads. Lower the background ratio to start flushing earlier and avoid the “cliff” where synchronous writeback kicks in.

​
6. Virtual Filesystems (procfs, sysfs)

Not all filesystems represent disks.
  • ProcFS (/proc): Exposes kernel data structures (processes, memory stats) as files.
  • SysFS (/sys): Exposes the hardware device tree and driver configurations.
  • TmpFS: A filesystem that exists entirely in RAM.

​
7. Choosing a Filesystem in Production

Selecting the right filesystem for your workload is a key architectural decision. Here’s a practical guide:

​
Decision Flowchart

┌─────────────────────────────────────────────────────────────────────┐
│                  FILESYSTEM SELECTION GUIDE                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  What's your primary concern?                                       │
│                                                                     │
│  ┌─────────────────┐     ┌─────────────────┐    ┌────────────────┐ │
│  │  DATA INTEGRITY │     │   PERFORMANCE   │    │   FLEXIBILITY  │ │
│  └────────┬────────┘     └────────┬────────┘    └───────┬────────┘ │
│           │                       │                      │          │
│           ▼                       ▼                      ▼          │
│  ┌─────────────────┐     ┌─────────────────┐    ┌────────────────┐ │
│  │      ZFS        │     │   XFS or EXT4   │    │     Btrfs      │ │
│  │  • Checksums    │     │   • Low latency │    │  • Snapshots   │ │
│  │  • Scrubbing    │     │   • High IOPS   │    │  • Compression │ │
│  │  • RAID-Z       │     │   • Scalability │    │  • Subvolumes  │ │
│  └─────────────────┘     └─────────────────┘    └────────────────┘ │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

​
Comparison Table

FilesystemBest ForAvoid WhenJournalingMax File Size
EXT4General Linux, boot partitionsVery large files (>16TB)Metadata or Data16 TB
XFSLarge files, high parallelismLots of small files, shrinkingMetadata8 EB
BtrfsSnapshots, compression, devProduction databases (still maturing)Copy-on-Write16 EB
ZFSData integrity, NAS, backupsLow-memory systems (less than 8GB)Copy-on-Write16 EB
F2FSSSDs, flash storageHDDsLog-structured16 TB

​
Production Checklist

  1. Database workloads (MySQL, PostgreSQL): XFS or EXT4 with noatime, data=ordered.
  2. Container hosts (Docker, Kubernetes): XFS for overlay2 driver, or Btrfs for native driver.
  3. Backup servers: ZFS (checksums detect silent corruption) or Btrfs (snapshots).
  4. High-throughput analytics: XFS (better large file and parallel write performance).
  5. Embedded/Flash: F2FS (reduces write amplification).

​
Quick Commands

# Check current filesystem
df -T /

# Create XFS filesystem (recommended for servers)
mkfs.xfs -f /dev/sdb1

# Mount with performance options
mount -o noatime,nodiratime,discard /dev/sdb1 /data

# Check filesystem-specific features
tune2fs -l /dev/sda1 | grep features   # EXT4
xfs_info /dev/sdb1                      # XFS
btrfs filesystem show /data             # Btrfs
zpool status                            # ZFS

​
7.5 Caveats and Common Pitfalls

Filesystem traps that bite even senior engineers:
  1. ext4 has metadata journaling, but data is NOT journaled by default. The default mount option is data=ordered: the journal protects metadata (inodes, bitmaps, directory entries) but the file contents are written separately and can be lost on crash, even though the filesystem will be consistent. If you read “ext4 is journaled” and assumed your data is safe, you misread the contract. Only data=journal (the slow, double-write mode) journals data. Most production deployments accept the risk and rely on the application to handle crash consistency via fsync.
  2. Atomic rename only works within the same filesystem. rename(2) is atomic on POSIX — but only if source and destination are on the same filesystem. Cross-filesystem mv becomes copy+delete: not atomic, can leave partial files, can fail halfway with the source gone and the destination incomplete. The infamous bug: a script does mv /tmp/file /var/data/file where /tmp is tmpfs and /var/data is ext4. Looks like a rename. Is actually copy+delete with no rollback.
  3. Symlink races (TOCTOU). Time-of-check to time-of-use bugs: you lstat a path, see it is a regular file, then open it — but between those calls, an attacker swapped it for a symlink to /etc/shadow. Classic privilege escalation. Defenses: O_NOFOLLOW to refuse symlinks, openat2 with RESOLVE_NO_SYMLINKS, opening the directory fd once and using *at syscalls thereafter. Most CTF-style local-privilege-escalation exploits are some flavor of this.
  4. inotify watch limit (default 8192) bites large fleets. fs.inotify.max_user_watches defaults to 8192 on many distros (newer kernels raise it). VS Code, webpack, file-syncing daemons, and Kubernetes kubelets all consume watches. On a node running 50 containers each watching /etc/resolv.conf plus a code editor, you blow past the limit and get cryptic ENOSPC errors that look like disk-full but are not. Bump max_user_watches to 524288 or more on dev machines and worker nodes.
Solutions and patterns:
  • For data durability, use fsync (or fdatasync) after every write you cannot afford to lose, plus fsync on the parent directory after creating or renaming a file. ext4 with data=ordered is fine if you use fsync correctly; it is dangerous if you assume the filesystem will save you.
  • Keep critical paths on a single filesystem and use rename for atomic publish-then-swap patterns. Write to foo.tmp, fsync it, then rename(foo.tmp, foo) — now any reader sees either the old version or the new, never a half-written one. Verify with stat that source and dest are on the same filesystem (same st_dev).
  • Use openat2 with RESOLVE_NO_SYMLINKS or RESOLVE_BENEATH for path-handling code that touches privileged data. Older code can use O_NOFOLLOW plus directory-fd-based access. Avoid string manipulation of paths in code that runs with elevated privileges.
  • Raise fs.inotify.max_user_watches and max_user_instances proactively on developer and worker machines. Tools like entr, watchman, fswatch, and modern bundlers chew through watches. The default 8192 was set decades ago and has not aged well.
  • Pick the filesystem from the workload, not the brand loyalty. XFS for large files and high parallelism. ext4 for general-purpose and boot. ZFS or btrfs when you need checksums, snapshots, or send/receive replication. F2FS for raw flash. Each has well-known failure modes — test them in your environment before committing.

​
Summary for Senior Engineers

  • Filenames are just pointers: A file can have multiple names (Hard Links) pointing to the same Inode.
  • Dentry Cache is the bottleneck for cold-start performance (e.g., npm install).
  • Extents and Delayed Allocation are the primary defenses against disk fragmentation.
  • Journaling protects the filesystem metadata, but not necessarily your data (unless data=journal mode is used).
Next: I/O Systems & Modern I/O →

​
Interview Deep-Dive

Strong Answer Framework:
  1. Frame each on the trinity: integrity, performance, flexibility. No filesystem wins all three; the choice is workload-driven.
  2. ext4 is the boring, bulletproof default. Metadata journaling, extents, delayed allocation. Great for boot disks, mail spools, general Linux servers. Weak spots: no checksums (silent corruption invisible), max file size 16TB, scaling beyond ~50TB starts to feel sluggish.
  3. XFS is the high-throughput parallel-IO workhorse. Allocation groups give you parallel writers without lock contention. Great for large files (analytics, video, scientific data), backup repositories, and database data files. Weak spots: shrinking is unsupported (you can grow but never shrink), small-file workloads are competitive but not dominant, no native snapshots.
  4. btrfs is the Swiss army knife: copy-on-write, native snapshots, send/receive replication, integrated RAID, transparent compression. Excellent for development machines, container hosts, snapshot-based backups. Weak spots: RAID 5/6 is still buggy in 2025 (“do not use” in the kernel docs), performance can degrade with heavy fsync (Postgres), and large filesystems need careful balance management.
  5. ZFS is the gold standard for data integrity: end-to-end checksums, RAIDZ (no write hole), compression, encryption, dedup, send/receive. Used by Netflix, iXsystems, dozens of NAS vendors. Weak spots: not in mainline Linux (CDDL license), eats RAM (rule of thumb: 1GB per TB plus extra for ARC), L2ARC and SLOG add operational complexity.
  6. Make a recommendation per workload. Postgres on bare metal: ext4 or XFS with noatime,data=ordered. Backup target with corruption detection: ZFS RAIDZ2. Container host with snapshot rollback: btrfs subvolumes (or ZFS, on FreeBSD). Petabyte-scale media archive: XFS with allocation groups tuned to NUMA.
Real-World Example: Facebook published in 2019 that they had migrated significant fleet capacity from XFS to btrfs because btrfs’s CoW and compression saved double-digit storage. They contributed many btrfs fixes upstream. Conversely, in 2021 the Fedora btrfs default decision provoked debate because consumer hardware (especially flash with poor flush behavior) exposed btrfs latency tails. Both can be true: btrfs is the right choice for some teams and the wrong choice for others. Pick from your workload, not from your peer group.
Senior Follow-up 1: “Why is btrfs RAID 5/6 still considered unstable?”The “write hole” — if power is lost between writing data and parity blocks, parity becomes inconsistent. Combined with btrfs’s CoW data layout and metadata pinning bugs surfacing under recovery, you can lose data on reconstruct. ZFS RAIDZ avoids this with variable-stripe writes; mdadm RAID5 papers over it with a write-intent bitmap or a journal device. btrfs has had patches in flight for years; the kernel docs explicitly warn against RAID5/6 for important data.
Senior Follow-up 2: “Why does Postgres performance suffer on btrfs?”Postgres calls fsync frequently (every WAL commit). On CoW filesystems, fsync forces a metadata transaction even when data did not change semantically — because the block locations of the data changed. This amplifies write traffic and journal commits. Workarounds: nodatacow on the data directory (loses btrfs’s data checksums), or use ext4/XFS for Postgres data and btrfs for everything else.
Senior Follow-up 3: “How do you decide between native ZFS RAID and hardware RAID?”Hardware RAID hides the disks behind a single block device, defeating ZFS’s ability to detect and repair corruption. Always give ZFS raw access (HBA mode / IT mode), let it manage redundancy. The argument “but hardware RAID has battery-backed cache” is solved by SLOG (a small fast SSD as ZFS’s intent log device). Same for mdadm: if you are running ZFS, do not stack it on top of md.
Common Wrong Answers:
  1. “ZFS is always best because of checksums.” True only when you can pay the RAM, complexity, and licensing cost. For a stateless container host with ephemeral storage, ext4 is fine.
  2. “btrfs is unstable; never use it.” The non-RAID-5/6 features are stable and used at major-vendor scale (SUSE, Synology, Facebook). Learn what is stable vs experimental.
  3. “XFS does not support journaling.” XFS has metadata journaling. It does not journal data, but neither does ext4 by default.
Further Reading:
  • “ZFS: The Last Word in Filesystems” — Jeff Bonwick’s original Sun design papers
  • “Btrfs: The Linux B-tree Filesystem” — Mason et al., ACM TOS
  • Dave Chinner’s XFS articles on lwn.net
Strong Answer Framework:
  1. Define the problem journaling solves. A filesystem operation typically updates multiple on-disk structures: inode, block bitmap, directory entry, data blocks. A crash mid-update leaves the filesystem inconsistent. Pre-journal recovery (fsck on ext2) was a full scan — minutes to hours on multi-TB volumes.
  2. Describe the journal as a write-ahead log. Before mutating real on-disk structures, the filesystem writes the intended changes (the “transaction”) to a circular journal area. The journal write completes, then the actual structures are updated, then the journal entry is marked committed.
  3. Walk through crash recovery. On mount, the FS examines the journal. Entries marked committed but not fully applied are replayed (re-applied to the main structures). Entries not committed are discarded (the in-progress transaction never happened from the FS’s view). Either way, the FS is consistent within seconds rather than hours.
  4. Distinguish modes. ext4 has data=writeback (journal metadata only, no ordering between data and metadata — fastest, can show stale data after crash), data=ordered (default; journal metadata, write data first then commit metadata — consistent but data is not journaled), data=journal (journal both — 2x write traffic but full crash protection). Most production picks ordered.
  5. State the limit. Journaling protects the filesystem’s internal consistency. It does NOT protect application data unless the app uses fsync correctly. A database that writes to a file without fsync, then crashes, can lose data even on a journaled filesystem.
  6. Note the alternatives. Copy-on-write filesystems (btrfs, ZFS) do not need a separate journal — they update by writing new blocks and atomically swinging a pointer. Crash recovery is implicit: either the new pointer is durable (new state) or it is not (old state).
Real-World Example: Stephen Tweedie’s original ext3 paper (2000) makes the case for journaling in detail. The XFS team at SGI shipped journaling in production in 1994 for IRIX; their experience informed every Linux journaled FS that came after. A vivid case study: pre-journal Linux servers in the 90s would commonly take 30+ minutes to fsck a 100GB filesystem after a crash; ext3 cut that to seconds. Recoveries that used to require operator attention became boot-time invisible.
Senior Follow-up 1: “What does data=ordered actually order?”Data writes for a transaction must complete before the transaction’s metadata commit hits the journal. Without that ordering, the journal could record “block 12345 belongs to inode 99” before block 12345’s content was written — a crash leaves inode 99 pointing at stale or random data. Ordered mode prevents this by issuing data writes first and waiting for them, then committing metadata.
Senior Follow-up 2: “What is the journal commit interval and why does it matter?”ext4 commits the journal every commit=N seconds (default 5). A crash within that window can lose recently written metadata even though the FS will be consistent. For latency-sensitive workloads, lower commit intervals reduce the loss window at the cost of more journal writes. For throughput-bound batch jobs, larger intervals amortize journal overhead.
Senior Follow-up 3: “How does a journal interact with barriers and write caches?”The journal entry must be durable on disk before the corresponding metadata update is durable, otherwise crash recovery sees a half-written transaction. ext4 issues REQ_PREFLUSH | REQ_FUA around journal commits to force the device to flush volatile cache. If you mount with barrier=0 (you should not, in production), you trade durability for throughput; on a power failure you can corrupt the FS.
Common Wrong Answers:
  1. “Journaling protects my data.” It protects the filesystem’s metadata consistency. For data, you still need fsync.
  2. “Replay just runs every transaction.” It runs only committed transactions. Non-committed (in-flight at crash) entries are discarded.
  3. “CoW filesystems still use a journal.” btrfs has a small log tree for fast fsync, but the main mechanism is CoW + atomic superblock pointer swing. ZFS uses a ZIL (intent log) for synchronous writes but relies on uberblock atomic update for consistency.
Further Reading:
  • Stephen Tweedie, “EXT3, Journaling Filesystem” — the canonical paper
  • “The Linux ext3 filesystem — Recovery and journaling” — LWN
  • Dave Chinner, “XFS Algorithms and Data Structures” (xfs.org)
Strong Answer Framework:
  1. Diagnose the symptom precisely. df shows used space high, du on the directory shows nothing. Classic signature of unlinked-but-open files.
  2. Explain the mechanism. When you unlink a file, the dentry is removed. The inode and data blocks remain allocated as long as any fd references them. Disk space returns only when the last fd is closed AND the link count is zero.
  3. Show how to confirm. lsof +L1 lists open files with link count zero. The output points at the holding process(es). On modern systems, lsof -nP +L1 | grep deleted is your friend.
  4. Explain the common culprit. Logging daemons (rsyslogd, journald, application loggers) hold the file open for writes. Naive rm of /var/log/big.log followed by waiting — forever. The process keeps writing to the now-orphan inode.
  5. Provide multiple fixes.
    • Restart the process holding the file. Kills the fd, releases the inode.
    • Truncate via /proc/<PID>/fd/<FD> — : > /proc/12345/fd/3 zeros the file in place without the process noticing.
    • Send SIGHUP to a daemon that handles log reopen on signal.
    • Use logrotate’s copytruncate for processes that cannot be signaled.
  6. Explain prevention. Configure logrotate properly. Use truncate(2) rather than unlink+create if you need to keep the daemon writing. Monitor for unlinked-but-open files in your alerting (Datadog has a check, Prometheus exporters expose it).
Real-World Example: In 2017, GitLab’s famous database incident included a moment where engineers rm -rf’d a wrong directory and disk usage looked like it had not changed. It was actually fine — the inodes were freed — but during incident response, several engineers mistook still-open log files for “the rm did not work” and tried to redo the operation on a different host. The post-mortem explicitly called out misunderstanding of unlink semantics under fd pressure as a cognitive failure during the incident.
Senior Follow-up 1: “Why does logrotate’s copytruncate exist?”For daemons that cannot be signaled to reopen log files (or for which reopening is unsafe). copytruncate copies the existing log to a backup, then truncates the original to zero in place. The daemon’s fd still points at the same inode — now empty — and continues writing without disruption. Trade-off: a brief race window during which writes between the copy and truncate are lost.
Senior Follow-up 2: “What is the dentry cache, and why is it critical for performance?”The dcache is the kernel’s cache of recent name-to-inode lookups. Without it, every path component requires reading directory blocks. For /var/log/nginx/access.log, that is 5 lookups (root, var, log, nginx, access.log). A cold dcache after reboot is the real reason “the first request after deploy is slow” on many systems. Hit rate is typically 99 percent in steady state. Inspect with /proc/sys/fs/dentry-state.
Senior Follow-up 3: “How would you find unlinked-but-open files at scale across a fleet?”Run lsof +L1 periodically and ship to your observability stack. Or use eBPF: hook do_unlinkat and track inodes with non-zero open counts. The Datadog/Prometheus node-exporter node_filesystem_files_free versus _files ratio also surfaces inode pressure. Alerts on disk-vs-inode divergence catch this class of bug early.
Common Wrong Answers:
  1. “rm did not actually delete it; try sudo rm -f.” The unlink succeeded; understanding why disk did not free is the actual lesson.
  2. “Reboot to fix it.” Works but masks the underlying logging configuration issue.
  3. “Use shred to scrub the file first.” Solves nothing about open fds. shred is for secure deletion of contents, not for freeing inodes.
Further Reading:
  • “How Linux handles file deletion” — LWN
  • The lsof man page (especially +L)
  • GitLab post-mortem of the 2017 database incident (a long read, worth it)