Control Groups (cgroups)
Control groups are the foundation of container resource management. Understanding cgroups deeply is essential for debugging container issues and designing resource-aware systems.Interview Focus: cgroups v2, memory limits, CPU throttling, debugging
Companies: All container/cloud companies heavily test this
cgroups Overview
cgroups v1 vs v2
- v1 (Legacy)
- v2 (Unified)
- Multiple hierarchies (one per controller)
- Controllers can be mounted independently
- More flexible but complex
- Still used in many production systems
- Inconsistent APIs across controllers
- Race conditions between hierarchies
- No unified resource management
CPU Controller Deep Dive
CPU Bandwidth Limiting
CPU Throttling
CPU Throttling Visualization
Memory Controller Deep Dive
Memory Limits and Protection
Memory Accounting Details
Memory Limit vs OOM
I/O Controller
I/O Bandwidth Limiting
I/O Latency Control (io.latency)
PID Controller
cpuset Controller
NUMA and cpuset
Practical cgroups Operations
Creating and Managing cgroups
Container Runtime cgroup Operations
Delegation and Nesting
Cgroup Delegation
Delegation Requirements
Debugging cgroups Issues
Common Issues and Solutions
Container OOM but host has memory
Container OOM but host has memory
- Increase memory limit
- Reduce application memory usage
- Add swap (for burst tolerance)
CPU throttling at low usage
CPU throttling at low usage
- Increase CPU limit
- Use
cpu.burst(cgroups v2) for burst allowance - Spread work more evenly
Can't write to cgroup files
Can't write to cgroup files
echo: write error: Device or resource busyDiagnosis:IO limits not working
IO limits not working
- Wrong device major:minor
- IO through page cache (buffered writes)
- Controller not enabled
Interview Questions
Q: What happens when a container exceeds memory.high?
Q: What happens when a container exceeds memory.high?
memory.high:- Reclaim pressure increases: Kernel aggressively reclaims memory from this cgroup
- Throttling may occur: Memory allocation requests may be delayed
- No OOM: The process is NOT killed (unlike exceeding
memory.max) - Performance impact: Application may slow down due to reclaim
- Soft limits with burst allowance
- Preventing one container from consuming all cache
- Graceful degradation instead of hard OOM
Q: Explain the difference between cgroups v1 and v2
Q: Explain the difference between cgroups v1 and v2
- Simpler mental model
- Consistent behavior
- Better pressure metrics (PSI)
- Proper thread-level controls
- Cleaner delegation model
Q: How would you debug high latency in a container?
Q: How would you debug high latency in a container?
-
Check CPU throttling:
-
Check memory pressure:
-
Check I/O latency:
-
Check for noisy neighbors:
- Look at sibling cgroups’ usage
- Check parent cgroup limits
-
Use tracing:
PSI (Pressure Stall Information)
New in cgroups v2 - provides pressure metrics:some: At least one task stalledfull: All tasks stalledavg10/60/300: 10s/60s/300s moving averages (%)total: Total stall time in microseconds
Interview Deep-Dive
A Kubernetes pod is showing 30% average CPU usage but requests are timing out. Users report the service is 'slow.' How do you diagnose this using cgroup internals, and what is the most likely cause?
A Kubernetes pod is showing 30% average CPU usage but requests are timing out. Users report the service is 'slow.' How do you diagnose this using cgroup internals, and what is the most likely cause?
- The most likely cause is CFS bandwidth throttling. Average CPU usage can be misleading because CFS enforces CPU limits per-period (typically 100ms). A service might use only 30% average CPU, but if requests arrive in bursts, the container could exhaust its entire quota in the first 30ms of a period and be throttled for the remaining 70ms. During throttling, all threads in the cgroup are descheduled regardless of available CPU on the host.
- To diagnose, I would read
cat /sys/fs/cgroup/<path>/cpu.statand compute the throttle percentage:nr_throttled / nr_periods * 100. If this exceeds 5%, throttling is significant. I would also checkthrottled_usecto see total time lost to throttling. - The kernel mechanism is the CFS bandwidth controller in
kernel/sched/fair.c. Each cgroup has acfs_bandwidthstructure tracking quota and period. When a task in the cgroup runs, its runtime is charged against the quota. When quota reaches zero, the scheduler removes all tasks in the cgroup from their runqueues until the next period boundary replenishes the quota. - Solutions in order of preference: increase the CPU limit in the Kubernetes resource spec, enable
cpu.burstin cgroups v2 (allows temporary burst above quota by borrowing from future periods), spread work more evenly across time using request queuing, or switch to CPU shares (cpu.weight) instead of hard limits if the node is not oversubscribed.
- CPU requests map to
cpu.weight(shares) which provide proportional fairness: if a container requests 1 CPU and another requests 2 CPUs, the second gets twice the CPU time when both are contending. But shares only work when there is contention. Without limits, a runaway container could consume all available CPU during low-load periods, then cause latency for other containers when load increases. Limits provide a hard ceiling viacpu.maxthat caps CPU usage regardless of host utilization. The trade-off is that limits cause throttling even when CPU is idle, which is wasteful. Many teams now advocate for setting requests but not limits for CPU (Google’s best practice), accepting the risk of noisy neighbors in exchange for eliminating throttling-induced latency.
Explain the cgroups v2 'no internal processes' rule and why it was introduced. How does this affect container runtime design?
Explain the cgroups v2 'no internal processes' rule and why it was introduced. How does this affect container runtime design?
- In cgroups v2, a cgroup that has controllers enabled in its
cgroup.subtree_controlcannot have processes directly incgroup.procs— processes must be in leaf cgroups only. This means if/sys/fs/cgroup/mygroup/enables CPU and memory controllers for its children, all processes must be in child cgroups like/sys/fs/cgroup/mygroup/child1/, not in/sys/fs/cgroup/mygroup/itself. - This rule was introduced to solve a fundamental ambiguity in cgroups v1: if a parent cgroup has processes and also has child cgroups, how should the parent’s resource share be divided between its direct processes and its children? In v1, this was handled inconsistently across controllers and led to confusing behavior where resource distribution depended on the tree structure in unintuitive ways.
- For container runtimes, this means the cgroup hierarchy must be designed carefully. A container runtime cannot simply create
/sys/fs/cgroup/containers/with controllers enabled and dump container processes there. Instead, it must create/sys/fs/cgroup/containers/container-abc/for each container and place processes in the leaf. Systemd handles this naturally with its slice/scope/service hierarchy:system.slice -> docker-abc123.scopewhere the scope is the leaf containing the container’s processes. - The practical impact is that management processes (the container runtime’s shim) must be in a separate cgroup from the container’s processes. runc places the shim in the parent’s sibling cgroup and the container in its own leaf.
- Cgroups v2 introduced
cgroup.type = "threaded"which allows individual threads of a process to be in different cgroups within a threaded subtree. This is useful when you want to give different threads different CPU priorities within the same process. For example, a database might put its query processing threads in one cgroup with higher CPU weight and its background compaction threads in another cgroup with lower weight. The root of the threaded subtree is a “domain threaded” cgroup, and its children are “threaded” cgroups. Not all controllers support threaded mode — currently onlycpuandcpusetare thread-aware.
Design a monitoring system that detects containers approaching resource exhaustion before they get OOM-killed or CPU-throttled. What cgroup metrics would you monitor, and what are the thresholds?
Design a monitoring system that detects containers approaching resource exhaustion before they get OOM-killed or CPU-throttled. What cgroup metrics would you monitor, and what are the thresholds?
- For memory, I would monitor three signals. First,
memory.current / memory.maxas a utilization ratio — alert at 80%. Second,memory.pressurefrom PSI (Pressure Stall Information):some avg10 > 10means at least one task is stalling 10% of the time waiting for memory, which indicates reclaim pressure before OOM. Third, theoomcounter inmemory.eventsto detect actual OOM kills. - For CPU, I would compute throttle percentage from
cpu.stat:nr_throttled / nr_periods * 100. Alert at 5% throttle rate. I would also monitorcpu.pressurewheresome avg10 > 25indicates meaningful CPU contention. The distinction betweensome(at least one task stalled) andfull(all tasks stalled) is important:someindicates contention,fullindicates complete blockage. - For I/O, monitor
io.pressurefor bothsomeandfullstall percentages. Also checkio.statfor per-device throughput to detect I/O bottlenecks. - For PID limits, monitor
pids.current / pids.maxand alert at 80%. A fork bomb or thread leak will hit this before OOM. - Implementation-wise, I would use a polling agent reading these pseudo-files every 5-10 seconds. PSI metrics are particularly efficient because they are pre-computed moving averages, so reading them is a single file read returning a single line. For production, I would expose these as Prometheus metrics and set up Grafana alerts.
- PSI tracks the actual time tasks spend stalled waiting for resources, not just how much of a resource is being used. The kernel maintains per-CPU counters that are updated on every task state transition. When a task transitions from runnable to blocked-on-memory-reclaim, the kernel increments the memory stall counter for that CPU. PSI then computes exponentially weighted moving averages over 10s, 60s, and 300s windows. This is fundamentally better than utilization because utilization does not capture demand: a container at 90% memory utilization might be fine (large but stable working set) or about to OOM (growing leak). PSI tells you whether tasks are actually waiting, which directly correlates with user-visible performance degradation.
Next: Filesystem & VFS →