Deadlocks
A deadlock is a situation where two or more threads are waiting indefinitely for resources held by each other. Think of two cars meeting on a narrow one-lane bridge from opposite sides — neither can move forward until the other backs up, but neither is willing to back up. In software, “backing up” means releasing a lock you already hold, which most programs are not designed to do. Understanding deadlocks is essential for senior engineers — they can bring entire systems down, and they are notoriously difficult to reproduce because they depend on precise timing between threads.Key Topics: Four conditions, Banker’s algorithm, prevention strategies
Time to Master: 6-8 hours
Deadlock Fundamentals
Classic Deadlock Scenario
The Four Necessary Conditions
1. Mutual Exclusion
2. Hold and Wait
3. No Preemption
4. Circular Wait
Resource Allocation Graph
Visualize resource allocation to detect deadlocks:- Single instance per resource: Cycle = Deadlock
- Multiple instances: Cycle is necessary but not sufficient
Deadlock Prevention
Break one of the four conditions:1. Break Mutual Exclusion
2. Break Hold and Wait
3. Break No Preemption
4. Break Circular Wait (Lock Ordering)
lockdep runtime validator to detect ordering violations during development. If you only remember one deadlock prevention technique, make it this one.
Practical tip: Document your lock ordering in a comment at the top of each subsystem. Future maintainers will thank you. The Linux kernel documents its lock ordering in files like Documentation/locking/lockdep-design.rst.
Deadlock Avoidance
Prevention is static — you design the rules upfront. Avoidance is dynamic: the system evaluates each resource request at runtime and denies it if granting it could lead to deadlock in the future. Think of it like a bank that checks your credit before approving a loan — even if it has the money right now, it will not lend if doing so could leave it unable to cover other obligations.Safe State
A state is safe if there exists a sequence where all processes can complete:Banker’s Algorithm
Named after the way a banker decides whether to approve a loan: “If I give this customer what they ask for, can I still guarantee that every other customer can eventually get their maximum loan?” If the answer is no, the request is denied even though the bank has the cash right now.Deadlock Detection
Instead of preventing or avoiding deadlocks upfront, let them happen and then detect and recover. This is the approach taken by most databases (PostgreSQL, MySQL/InnoDB) because it offers the best resource utilization — you do not waste resources on conservative avoidance, and deadlocks are rare enough that the occasional recovery cost is acceptable.Detection Algorithm
When to Run Detection
Deadlock Recovery
Once detected, how to break the deadlock:1. Process Termination
2. Resource Preemption
- Rolling back may lose work
- Same process may be selected repeatedly (starvation)
- Need checkpointing for rollback
Locking Design Playbook
This section is a practical checklist for designing locking in a new subsystem so you avoid deadlocks up front.Step 1: Identify Resources and Contention Points
- List all shared data structures and external resources (sockets, files, devices) that multiple threads touch.
- For each, decide:
- Is it mostly read or write heavy?
- How large is the critical section?
- Is ordering important (e.g., parent → child, global → local)?
Step 2: Define a Global Lock Ordering
- Assign each lock a total order (e.g.,
GLOBAL<DB<CACHE<LOG). - Enforce the rule: always acquire locks in increasing order.
- Document this in comments and design docs so future contributors follow it.
Step 3: Choose the Right Primitive
- Coarse mutex for simple systems with low contention.
- Fine-grained locks where contention hotspots are identified.
- Reader-writer locks for read-mostly data.
- Lock-free / RCU only when the performance benefit justifies the complexity.
Step 4: Add Timeouts and Diagnostics
- Prefer
trylock+ backoff in non-critical paths where waiting forever is unacceptable. - Add lock acquisition logging at debug level for tricky subsystems.
- In the kernel, enable tools like lockdep; in user space, integrate watch-dog threads.
Step 5: Simulate and Test
- Write stress tests that:
- Start many threads acquiring locks in randomized orders (still following your documented rules).
- Intentionally break the ordering in a test-only build to ensure your detection mechanisms fire.
- Use tools like
helgrind,TSan, or kernel lock validators where applicable.
Related Problems
Livelock
Livelock is the polite version of deadlock: two people meet in a hallway and keep stepping aside in the same direction, forever. The threads are not blocked — they are actively executing — but they make no progress.Starvation
A thread never gets the resources it needs, even though the system is not deadlocked. Think of a shy person at a buffet who keeps stepping back whenever someone more aggressive reaches for food — there is plenty of food, but they never eat.Priority Inversion
Low-priority thread blocks high-priority thread. This is the classic “intern blocks the CEO” problem — the intern holds the only conference room key, but a medium-priority manager keeps preempting the intern before they can unlock the door, so the CEO (who needs the room) waits indefinitely.- Priority inheritance: L temporarily gets H’s priority
- Priority ceiling: Lock has priority, holder gets it
Interview Deep Dive Questions
Q1: Design a system that's deadlock-free
Q1: Design a system that's deadlock-free
Q2: Walk through Banker's algorithm with an example
Q2: Walk through Banker's algorithm with an example
Q3: How does a database handle deadlocks?
Q3: How does a database handle deadlocks?
- Wait-for graph: Track which transaction waits for which
-
Deadlock detection: Periodically (every
deadlock_timeout, default 1s) check for cycles -
Victim selection: Abort transaction that:
- Has done least work (youngest)
- Holds fewest locks
- Is easiest to rollback
Q4: Explain priority inversion with a real-world example
Q4: Explain priority inversion with a real-world example
- Priority Inheritance:
- Priority Ceiling Protocol:
Q5: Implement a deadlock detection system
Q5: Implement a deadlock detection system
Practice Exercises
Banker's Simulator
Deadlock Detector
Dining Philosophers
Priority Inheritance
Key Takeaways
Four Conditions
Lock Ordering Works
Detection + Recovery
Databases Handle It
Next: Inter-Process Communication →
Interview Deep-Dive
How would you detect a deadlock at runtime in a production service? What signals do you watch?
How would you detect a deadlock at runtime in a production service? What signals do you watch?
- Define detection vs. diagnosis. Detection answers “is something stuck?”; diagnosis answers “why?” Both are needed but the tooling differs.
- Detection signals in order of usefulness:
- Latency p99 cliff — requests that normally take 50ms suddenly take 30 seconds. Easy to alert on, often the first signal.
- Thread pool saturation — monitor “active threads / max threads” per pool. When it pegs at 100% and stays there while RPS drops, threads are blocked, not busy.
- Lock wait time — expose
pthread_mutexwait time as a metric (Linux:perf lock contention; PostgreSQL:pg_stat_activity.wait_event). A growing wait queue with stable acquisition rate means contention; a frozen wait queue means deadlock. - CPU drop with steady RPS — a deadlocked process holds threads but does no work. You see CPU collapse while request rate is stable — a paradoxical signature.
- Diagnosis steps once detected:
- Capture stacks from every thread (
gdb thread apply all bt,jstack,py-spy dump). - Look for the cycle: thread A in
pthread_mutex_lockfor mutex M1, where M1 is held by thread B who is inpthread_mutex_lockfor M2. - For Java, JVM auto-detects and prints “Found one Java-level deadlock”; for Linux, lockdep does this in debug kernels.
- Capture stacks from every thread (
- Automate the response: a watchdog that detects “no requests completed in 60s but threads remain” should dump stacks and either restart or page on-call. Netflix’s Hystrix / resilience4j uses thread pool saturation as a circuit breaker trigger for exactly this reason.
pg_stat_activity showing two backends each waiting on the other’s transaction. After this, they added a Prometheus exporter for pg_locks and alert when wait time exceeds 1 second. PostgreSQL has built-in deadlock detection (default deadlock_timeout = 1s) that aborts one transaction with ERROR: deadlock detected — in databases, you usually do not need custom detection; you need to alert on the count of these errors.Lock wait events (deadlock candidate) from IO waits (slow disk) from Client waits (idle in transaction). A pure deadlock has zero CPU usage and no I/O activity on the blocked thread. A slow query is on-CPU or doing I/O.flock/fcntl/futex waiters and can build the wait-for graph. PostgreSQL detects deadlocks across backend processes because the lock manager is in shared memory. Across hosts (distributed locks via Redis/Zookeeper), you need an external coordinator with timeouts; there is no kernel-level visibility. Most distributed systems use timeouts + lease-based locks rather than true detection.- “Just set a timeout on every operation” — treats the symptom; the deadlock still happens, you just get faster errors instead of fixing the root cause.
- “Restart the service if requests are slow” — masks the bug and conflates deadlock with general slowness.
- “Use a deadlock detection algorithm in the application” — usually overkill; the OS / DB already does this. Reach for it only when you have your own custom resource manager.
- PostgreSQL docs, “Lock Management” section, especially
deadlock_timeoutandpg_locks— the model implementation. - Linux kernel
Documentation/locking/lockdep-design.rst— lockdep internals, free. - Brendan Gregg, “Systems Performance” Chapter 5 — how to diagnose lock contention with
perf.
Walk through the Dining Philosophers problem and explain at least three solutions. Which one scales?
Walk through the Dining Philosophers problem and explain at least three solutions. Which one scales?
- State the problem precisely: 5 philosophers, 5 forks shared between adjacent pairs, each needs both adjacent forks to eat. Naive “pick up left then right” deadlocks if all five act simultaneously.
- Solution 1: Resource ordering (lock by global order). Number forks 0-4. Each philosopher acquires
min(left, right)first,max(left, right)second. Breaks circular wait. Scales O(1) per attempt, no central coordination. - Solution 2: Asymmetric breaking. Philosopher 4 picks up right then left; everyone else picks up left then right. One asymmetric actor is enough to break the cycle. Scales identically to ordering, sometimes simpler to retrofit.
- Solution 3: Waiter (global mutex / arbiter). A central waiter grants permission to pick up forks. Breaks hold-and-wait. Does NOT scale — the waiter serializes all eating, throughput drops to 1 philosopher at a time.
- Solution 4: Chandy-Misra (token-based). Distributed protocol with dirty/clean fork states. No central coordinator, scales to thousands of philosophers across machines. Best for distributed systems but high implementation complexity.
- Solution 5: Try-lock with backoff. Each philosopher tries non-blocking acquire of both forks; if either fails, release and back off. Avoids deadlock but creates livelock risk without randomized backoff.
- Recommend resource ordering for production: simplest, scales, and matches what real systems (Linux kernel lock hierarchy, database lock managers) actually do.
Documentation/filesystems/directory-locking.rst documents the exact ordering rules: parent-before-child for create/delete, ancestor-first for renames, and a global rename-lock for cross-directory moves to prevent ordering cycles.deadlock_timeout (default 1s). When it finds a cycle, it aborts the transaction with the youngest XID. The user sees ERROR: deadlock detected and is expected to retry.- “Just use the waiter, it is the simplest” — correct that it is simple, wrong that it scales. Ignores the throughput cost.
- “Try-lock without backoff fixes it” — creates livelock; threads keep retrying in lockstep.
- “Acquire both forks atomically” — not actually possible in standard pthreads without a higher-level coordinator (which is the waiter solution in disguise).
- Edsger Dijkstra, “Hierarchical Ordering of Sequential Processes” (1971) — where the dining philosophers were first stated.
- Chandy & Misra, “The Drinking Philosophers Problem” (1984) — generalization with the token algorithm.
- Linux kernel
Documentation/filesystems/directory-locking.rst— production-scale lock ordering.
A production server is wedged. Stack traces show one thread blocked in pthread_mutex_lock and another in a kernel semaphore inside an ioctl. How do you debug this?
A production server is wedged. Stack traces show one thread blocked in pthread_mutex_lock and another in a kernel semaphore inside an ioctl. How do you debug this?
- First, characterize the wait. Userspace mutex shows up as
__lll_lock_waitorfutex_waitin the stack. Kernel semaphore shows up with the syscall in the upper frames — the thread is in kernel space, blocked ondown()ordown_interruptible(). - Identify the resource each is waiting on.
- For the userspace mutex: in gdb,
print *((pthread_mutex_t*)0xADDRESS)to see the owner field. That tells you which thread holds it. - For the kernel semaphore: this is harder from userspace. Use
cat /proc/PID/stackto see the in-kernel call chain, andcat /proc/PID/wchanto see what kernel function the thread is sleeping in.
- For the userspace mutex: in gdb,
- Build the wait-for graph manually. Thread A waits on userspace mutex M held by Thread B. Thread B is in ioctl waiting on kernel semaphore S. What does S protect? Often a device driver’s internal state. If the holder of S is also waiting on M (e.g., a driver callback that calls back into userspace via a signal handler that takes M), you have a deadlock spanning userspace and kernel space.
- Use kernel tools:
echo l > /proc/sysrq-triggerdumps all CPU stack traces.echo t > /proc/sysrq-triggerdumps all task states.cat /proc/lockdep_chainsif lockdep is on — shows kernel lock dependency graph.bpftraceto observe semaphore acquisitions in real time: tracedownandupcalls.
- Common root causes:
- Driver bug: ioctl handler holds a kernel lock and triggers a signal that calls back into userspace, where the signal handler takes a userspace lock that is held by another thread waiting on the ioctl. Classic kernel-userspace inversion.
mmap-backed I/O: writing to mmap’d memory triggers a page fault, which takesmmap_sem, which conflicts with another thread’smmapsyscall. This is the famousmmap_semcontention problem that drove the kernel’s switch tommap_lock(rwsem) in 2020.
- Fix patterns: never call signal handlers or callbacks while holding a kernel lock you might re-enter; never hold a userspace lock while making syscalls that could block on the same resource the lock protects.
mmap_sem was the source of dozens of deadlock bugs over a decade. A classic case: a thread calls mmap, which takes mmap_sem as writer. The new VMA triggers a page fault on first access; the fault handler also wants mmap_sem as reader. Recursive acquisition deadlock. The fix landed in 5.8 (2020): mmap_sem was renamed to mmap_lock and given proper read/write semantics with annotations enforced by lockdep. Greg Kroah-Hartman cited this as one of the longest-running multi-kernel-version refactors./proc/PID/stack. Kernel tools (ftrace, bpftrace) cannot easily decode userspace symbols without debug info. You usually need both: gdb for userspace state, cat /proc/PID/stack plus crash (kernel core dump tool) for the kernel side. perf can bridge them with --call-graph dwarf but the merged stacks are still imperfect.D state and how does it relate to deadlock?
D state (TASK_UNINTERRUPTIBLE) means the thread is sleeping in the kernel and cannot be killed — not even by SIGKILL. It is used for short, non-interruptible waits (e.g., disk I/O) that the kernel guarantees will complete. A thread stuck in D state for minutes usually indicates a driver bug or storage hang — effectively a kernel deadlock. You see it in top as the D state column.stress-ng for kernel pressure (--vm, --io, --futex), inject usleep in userspace handlers to widen the race window, and run on a slow VM where timing differences are amplified. For real reproducibility, use rr (Mozilla’s record-and-replay debugger) which captures the exact thread interleaving so you can replay the deadlock deterministically.- “Just kill -9 the process” — D state ignores SIGKILL; you cannot kill a thread blocked in the kernel.
- “Add a timeout to the ioctl” — often impossible (drivers may not honor timeouts) and treats the symptom.
- “Restart the host” — works but loses the diagnostic state. Capture stacks first.
- Brendan Gregg, “BPF Performance Tools” — has a chapter on tracing kernel locks with bpftrace.
- Linux kernel
Documentation/admin-guide/sysrq.rst— the SysRq interface for emergency diagnostics. crashutility documentation — post-mortem kernel debugger; essential for analyzing kernel deadlocks from a vmcore.
You have a distributed microservices system where Service A calls Service B while holding a database lock, and Service B calls Service A while holding a different lock. Is this a deadlock? How would you detect and fix it?
You have a distributed microservices system where Service A calls Service B while holding a database lock, and Service B calls Service A while holding a different lock. Is this a deadlock? How would you detect and fix it?
- This is a distributed deadlock, and it is much harder to detect than a local deadlock because no single node has a complete picture of the wait-for graph. In a local deadlock, the OS or database can inspect all threads and locks. In a distributed deadlock, the “locks” span multiple services, databases, and network connections.
- The wait cycle here is: Service A holds Lock-X, calls Service B over the network (blocking), Service B holds Lock-Y, calls Service A over the network (blocking). If A’s handler for B’s request also needs Lock-X, we have a classic circular wait distributed across two machines.
- Detection approaches: (1) Timeouts — the most practical approach. Every RPC call should have a deadline. If Service A’s call to B times out, it releases Lock-X and retries or fails. (2) Distributed deadlock detection using a centralized coordinator that collects wait-for edges from all services and runs cycle detection. (3) Wound-wait or wait-die schemes where the younger transaction is always aborted to break cycles.
- The fix is architectural: avoid holding locks across network calls. Use a saga pattern where each service does its local work and commits, then sends a message to the next service. If a later step fails, compensating transactions undo earlier steps.
The Linux kernel has a tool called lockdep that detects potential deadlocks at runtime without them actually occurring. How does this work?
The Linux kernel has a tool called lockdep that detects potential deadlocks at runtime without them actually occurring. How does this work?
- lockdep tracks the order in which locks are acquired across all code paths and builds a global lock ordering graph at runtime. Every time a thread acquires a lock, lockdep records the pair (previously held lock, newly acquired lock) as a directed edge. If it ever detects a cycle in this graph, it reports a potential deadlock — even if the actual deadlock has not occurred yet.
- The key insight is that lockdep operates on lock classes, not lock instances. All instances of the same lock type (e.g., all inode mutexes) are treated as one class. If code path 1 acquires class A then B, and code path 2 acquires B then A, lockdep reports a violation even if the actual instances are different. This catches deadlocks that might only manifest under rare timing.
- lockdep also tracks locks across interrupt context boundaries. If you acquire a lock in process context and the same lock class is also acquired in interrupt context, lockdep warns you because an interrupt could fire while the process-context code holds the lock.
- The trade-off is performance: lockdep adds overhead to every lock acquisition, so it is typically enabled in debug kernels and disabled in production. It has found hundreds of real locking bugs in the Linux kernel.
lock_set_subclass, lock_nested) to inform lockdep about these intentional ordering variations.Walk me through the Dining Philosophers problem and three distinct solutions. Which would you use in production?
Walk me through the Dining Philosophers problem and three distinct solutions. Which would you use in production?
- The problem: five philosophers sit around a table, each needs two forks to eat. If each picks up their left fork simultaneously, all hold one fork and wait for the other — deadlock.
- Solution 1 — Resource ordering: Number the forks 0-4. Each philosopher always picks up the lower-numbered fork first. This breaks the circular wait condition. This is the most practical production solution: simple, zero overhead, and never wastes CPU.
- Solution 2 — Global arbitrator: A single mutex protects the “pick up forks” operation. Breaks hold-and-wait but creates a bottleneck — only one philosopher can attempt to eat at a time.
- Solution 3 — Chandy-Misra: A distributed protocol using dirty/clean fork states. Fully distributed, no central coordinator, designed for systems where global ordering is impractical. But complex to implement correctly.
- In production, I use resource ordering every time unless the system is truly distributed. It is easy to reason about, verifiable by lockdep, zero runtime overhead, and scales perfectly.