┌─────────────────────────────────────────────────────────────────┐
│                    DEBUGGING METHODOLOGY                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   1. OBSERVE                                                     │
│      • What is the symptom?                                     │
│      • When did it start?                                       │
│      • What changed recently?                                   │
│                                                                  │
│   2. HYPOTHESIZE                                                 │
│      • What could cause this?                                   │
│      • Rank by likelihood                                       │
│                                                                  │
│   3. MEASURE                                                     │
│      • Use appropriate tools                                    │
│      • Gather data to confirm/refute hypothesis                 │
│                                                                  │
│   4. DIAGNOSE                                                    │
│      • Analyze data                                             │
│      • Narrow down root cause                                   │
│                                                                  │
│   5. FIX & VERIFY                                               │
│      • Implement fix                                            │
│      • Confirm symptom is resolved                              │
│      • Ensure no regression                                     │
│                                                                  │
│   USE Method (Brendan Gregg):                                   │
│   • Utilization: How busy is the resource?                      │
│   • Saturation: Is work queuing?                                │
│   • Errors: Are there error conditions?                         │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

# Basic usage
strace ls -la

# Trace running process
strace -p <PID>

# Follow forks
strace -f ./program

# Trace specific syscalls
strace -e open,read,write ./program
strace -e trace=file ./program      # All file-related syscalls
strace -e trace=network ./program   # All network syscalls
strace -e trace=memory ./program    # Memory syscalls

# Timing information
strace -T ./program          # Time spent in each syscall
strace -tt ./program         # Absolute timestamps
strace -r ./program          # Relative timestamps

# Summary statistics
strace -c ./program          # Syscall summary at end
strace -c -S time ./program  # Sort by time

# Output to file
strace -o trace.log ./program

# Show string arguments fully
strace -s 1000 ./program     # 1000 chars instead of default 32

# Example output:
open("/etc/passwd", O_RDONLY)           = 3
│         │              │                │
│         │              │                └── Return value (fd or -1)
│         │              └── Flags
│         └── File path
└── System call name

# Error case:
open("/nonexistent", O_RDONLY)          = -1 ENOENT (No such file)
                                              │
                                              └── Error code and description

# Common patterns to look for:
# - ENOENT: File not found
# - EACCES: Permission denied
# - EAGAIN: Resource temporarily unavailable
# - EINTR: Interrupted by signal
# - Slow syscalls (with -T)
# - Unexpected syscalls (process doing more than expected)

# Trace library calls
ltrace ./program

# With syscalls too
ltrace -S ./program

# Specific libraries
ltrace -e malloc+free ./program
ltrace -e '*@libssl*' ./program

# CPU profiling
perf record -g ./program     # Record with call graphs
perf report                   # View results

# System-wide profiling
perf record -a -g -- sleep 10

# Specific events
perf stat ./program          # Basic performance counters
perf stat -e cycles,instructions,cache-misses ./program

# Live top-like view
perf top                     # System-wide
perf top -p <PID>            # Specific process

# Flame graph generation
perf record -g ./program
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

# List available events
perf list

# Hardware events
perf stat -e cycles,instructions,cache-references,cache-misses ./program

# Software events
perf stat -e context-switches,cpu-migrations,page-faults ./program

# Tracepoints
perf stat -e 'sched:*' ./program     # Scheduler events
perf stat -e 'block:*' ./program     # Block I/O events

# Dynamic tracing
perf probe --add 'do_sys_open filename:string'
perf record -e probe:do_sys_open -a
perf probe --del do_sys_open

# perf stat output example:
 Performance counter stats for './program':

       1,234,567,890      cycles                    #    2.500 GHz
         987,654,321      instructions              #    0.80  insn per cycle
          12,345,678      cache-references
           1,234,567      cache-misses              #   10.00% of all cache refs
               1,234      page-faults
                  12      context-switches

       0.987654321 seconds time elapsed

# Key metrics:
# - Instructions per cycle (IPC): Higher is better, >1 is good
# - Cache miss rate: Lower is better, >10% is concerning
# - Page faults: Should be low for running programs

# Memory leak detection
valgrind --leak-check=full ./program

# Show all errors
valgrind --leak-check=full --show-leak-kinds=all ./program

# Track origins of uninitialized values
valgrind --track-origins=yes ./program

# Memory profiler (heap usage)
valgrind --tool=massif ./program
ms_print massif.out.*

# Cache profiler
valgrind --tool=cachegrind ./program
cg_annotate cachegrind.out.*

# Thread error detector
valgrind --tool=helgrind ./program

// Compile with ASan
// gcc -fsanitize=address -g program.c -o program

// Example: Buffer overflow
int main() {
    int arr[10];
    arr[10] = 1;  // ASan will catch this!
    return 0;
}

// ASan output:
// ==12345==ERROR: AddressSanitizer: stack-buffer-overflow
// WRITE of size 4 at 0x7ffc12345678
// ... stack trace ...

# Different sanitizers
gcc -fsanitize=address   # Memory errors
gcc -fsanitize=thread    # Data races
gcc -fsanitize=undefined # Undefined behavior
gcc -fsanitize=leak      # Memory leaks only

# Process memory map
pmap -x <PID>

# Detailed /proc view
cat /proc/<PID>/maps          # Memory mappings
cat /proc/<PID>/smaps         # Detailed memory info
cat /proc/<PID>/status        # Memory summary

# System memory
free -h
vmstat 1                      # Virtual memory stats
cat /proc/meminfo

# OOM killer
dmesg | grep -i oom
cat /proc/<PID>/oom_score     # OOM score (higher = more likely to kill)

# Memory pressure
cat /proc/pressure/memory

# Trace file opens
opensnoop

# Trace exec calls
execsnoop

# TCP connection tracing
tcpconnect          # Outgoing connections
tcpaccept           # Incoming connections
tcpretrans          # Retransmissions

# Disk I/O latency
biolatency

# Process resource usage
pidstat 1

# Syscall latency
syscount -L         # Count with latency

# Count syscalls by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Trace file opens
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Disk I/O latency histogram
bpftrace -e 'tracepoint:block:block_rq_complete { @us = hist(args->nr_sector); }'

# Function latency
bpftrace -e 'kprobe:do_sys_open { @start[tid] = nsecs; }
             kretprobe:do_sys_open /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'

# One-liners for common tasks
bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'  # CPU stack sampling
bpftrace -e 'tracepoint:sched:sched_switch { @[args->prev_comm] = count(); }'  # Context switches

// Example: Count syscalls by process
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);
    __type(value, u64);
} syscall_count SEC(".maps");

SEC("tracepoint/raw_syscalls/sys_enter")
int count_syscalls(void *ctx) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *count = bpf_map_lookup_elem(&syscall_count, &pid);
    
    if (count) {
        (*count)++;
    } else {
        u64 init = 1;
        bpf_map_update_elem(&syscall_count, &pid, &init, BPF_ANY);
    }
    
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

# Real-time CPU usage
top
htop

# Per-CPU usage
mpstat -P ALL 1

# Load average explained
uptime
# 3 numbers: 1min, 5min, 15min
# Represents average # of processes in runnable + uninterruptible states
# > number of CPUs = overloaded

# CPU pressure
cat /proc/pressure/cpu
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0

# Per-process CPU
pidstat -u 1
ps aux --sort=-%cpu | head

# What's using CPU
perf top

# Time processes wait to run
bpftrace -e 'tracepoint:sched:sched_wakeup { @[comm] = hist(nsecs); }'

# BCC tool
runqlat

# CPU scheduler stats
cat /proc/schedstat
cat /proc/<PID>/schedstat

# Real-time I/O stats
iostat -xz 1

# Key metrics:
# r/s, w/s: reads/writes per second
# rkB/s, wkB/s: throughput
# await: average I/O latency (ms)
# %util: utilization (100% = saturated)

# Per-process I/O
iotop
pidstat -d 1

# Block device queue
cat /sys/block/sda/queue/nr_requests
cat /sys/block/sda/stat

# I/O latency distribution (BCC)
biolatency
biosnoop

# Trace I/O operations
blktrace -d /dev/sda -o trace
blkparse -i trace.blktrace.0

# File system stats
df -h
df -i                         # Inodes

# Cache stats
cat /proc/meminfo | grep -E 'Cached|Buffers'

# Page cache hit rate
cachestat                     # BCC tool

# File system latency (BCC)
ext4slower
xfsslower

# Trace file operations
bpftrace -e 'kprobe:vfs_read { @reads[comm] = count(); }'

# Start debugging
gdb ./program
gdb --args ./program arg1 arg2

# Attach to running process
gdb -p <PID>

# Common commands
(gdb) run                     # Start program
(gdb) break main              # Set breakpoint
(gdb) break file.c:42         # Breakpoint at line
(gdb) continue                # Continue execution
(gdb) step                    # Step into
(gdb) next                    # Step over
(gdb) print variable          # Print value
(gdb) backtrace               # Show call stack
(gdb) info threads            # List threads
(gdb) thread 2                # Switch to thread 2

# Watchpoints
(gdb) watch variable          # Break on write
(gdb) rwatch variable         # Break on read

# Core dump analysis
gdb ./program core
(gdb) backtrace

# Enable core dumps
ulimit -c unlimited

# Set core dump pattern
echo '/tmp/core.%e.%p' > /proc/sys/kernel/core_pattern

# Use systemd-coredump
coredumpctl list
coredumpctl debug <PID>

# Analyze with gdb
gdb program corefile
(gdb) bt full                 # Full backtrace with locals

# Python profiling
python -m cProfile script.py
python -m cProfile -o profile.out script.py
# Visualize with snakeviz

# Go profiling
go tool pprof http://localhost:6060/debug/pprof/profile

# Java profiling
jstack <PID>                  # Thread dump
jmap -heap <PID>              # Heap summary
async-profiler                # CPU/allocation profiling

# Node.js
node --inspect program.js     # Chrome DevTools debugging
node --prof program.js        # V8 profiler

# journalctl (systemd)
journalctl -f                 # Follow logs
journalctl -u nginx           # Specific service
journalctl --since "1 hour ago"
journalctl -p err             # Only errors
journalctl -k                 # Kernel messages
journalctl --disk-usage       # Log disk usage

# Traditional logs
tail -f /var/log/syslog
tail -f /var/log/messages
dmesg                         # Kernel ring buffer
dmesg -T                      # Human-readable timestamps
dmesg -w                      # Follow

# Log analysis
grep -r "error" /var/log/
journalctl | grep -i error | wc -l

┌─────────────────────────────────────────────────────────────────┐
│                    LINUX TRACING HIERARCHY                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Frontends (User Tools)                                        │
│   ┌────────────────────────────────────────────────────────┐    │
│   │ perf    ftrace   bpftrace   BCC   SystemTap   LTTng   │    │
│   └────────────────────────────────────────────────────────┘    │
│                               │                                  │
│                               ▼                                  │
│   Tracers                                                        │
│   ┌────────────────────────────────────────────────────────┐    │
│   │        eBPF        │       ftrace       │   perf      │    │
│   └────────────────────────────────────────────────────────┘    │
│                               │                                  │
│                               ▼                                  │
│   Data Sources                                                   │
│   ┌────────────────────────────────────────────────────────┐    │
│   │ kprobes  uprobes  tracepoints  PMCs  software events  │    │
│   └────────────────────────────────────────────────────────┘    │
│                               │                                  │
│                               ▼                                  │
│   ┌────────────────────────────────────────────────────────┐    │
│   │                    Linux Kernel                        │    │
│   └────────────────────────────────────────────────────────┘    │
│                                                                  │
│   Legend:                                                        │
│   • kprobes: Dynamic kernel function tracing                    │
│   • uprobes: Dynamic userspace function tracing                 │
│   • tracepoints: Static kernel tracing points                   │
│   • PMCs: Performance Monitoring Counters (hardware)            │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                    PERFORMANCE CHECKLIST                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Symptom              │ Check                 │ Tool            │
│   ─────────────────────┼───────────────────────┼─────────────   │
│   High CPU             │ What's using CPU?     │ top, perf top   │
│   High latency         │ Where's time spent?   │ perf, strace -T │
│   Slow disk            │ I/O wait, throughput  │ iostat, iotop   │
│   Memory issues        │ OOM, swapping         │ free, vmstat    │
│   Network slow         │ Bandwidth, latency    │ ss, tcpdump     │
│   Lock contention      │ Mutex wait time       │ perf, futex     │
│   Frequent page faults │ Memory pressure       │ perf stat       │
│   Context switching    │ Thread thrashing      │ vmstat, pidstat │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

// 1. Polling instead of blocking
// Bad:
while (!data_ready) {
    usleep(1000);  // Wastes CPU
}

// Good:
pthread_cond_wait(&cond, &mutex);

// 2. Excessive system calls
// Bad:
for (int i = 0; i < 1000000; i++) {
    write(fd, &byte, 1);  // 1M syscalls
}

// Good:
write(fd, buffer, 1000000);  // 1 syscall

// 3. False sharing
// Bad:
struct {
    int counter1;  // Same cache line
    int counter2;  // Threads contending
} shared;

// Good:
struct {
    int counter1;
    char padding[64];  // Separate cache lines
    int counter2;
} shared;

// 4. Lock convoy
// Bad: Taking a lock for too long
// Good: Minimize critical section, use read-write locks

┌─────────────────────────────────────────────────────────────────┐
│                    DEBUGGING QUICK REFERENCE                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  What                     │ Command                              │
│  ─────────────────────────┼─────────────────────────────────────│
│  System calls             │ strace -p <PID>                      │
│  Library calls            │ ltrace ./program                     │
│  CPU profiling            │ perf record -g ./program             │
│  CPU flame graph          │ perf script | flamegraph.pl          │
│  Memory leaks             │ valgrind --leak-check=full           │
│  Memory errors            │ gcc -fsanitize=address               │
│  Data races               │ gcc -fsanitize=thread                │
│  File operations          │ opensnoop (BCC)                      │
│  Network connections      │ tcpconnect, tcpaccept                │
│  Disk I/O latency         │ biolatency (BCC)                     │
│  Process resource usage   │ pidstat 1                            │
│  Syscall count            │ strace -c ./program                  │
│  Debug core dump          │ gdb program core                     │
│  Kernel messages          │ dmesg -T                             │
│  Service logs             │ journalctl -u service                │
│  Custom tracing           │ bpftrace -e 'kprobe:... { ... }'    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘