Skip to main content

Hands-on Projects

Theory is essential, but hands-on practice is what separates good candidates from great ones. These projects will help you build practical skills that interviewers look for.
Skill Level: Intermediate to Advanced
Time Investment: 20-40 hours total
Outcome: Portfolio pieces for interviews + deep understanding

Project 1: Build a Container from Scratch

Difficulty: ⭐⭐⭐
Time: 4-6 hours
Skills: Namespaces, cgroups, syscalls

Goal

Build a minimal container runtime in C or Go that demonstrates your understanding of Linux isolation primitives.

Requirements

1

Create isolated namespaces

// Clone with new namespaces
int flags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | 
            CLONE_NEWUTS | CLONE_NEWIPC;
pid_t pid = clone(child_func, stack_top, flags | SIGCHLD, NULL);
2

Set up cgroup resource limits

// Create cgroup
mkdir("/sys/fs/cgroup/mycontainer", 0755);

// Set memory limit
write_file("/sys/fs/cgroup/mycontainer/memory.max", "100M");

// Set CPU limit (50% of 1 CPU)
write_file("/sys/fs/cgroup/mycontainer/cpu.max", "50000 100000");

// Add process to cgroup
write_file("/sys/fs/cgroup/mycontainer/cgroup.procs", pid_str);
3

Set up root filesystem with pivot_root

// Mount new root
mount(rootfs, rootfs, NULL, MS_BIND | MS_REC, NULL);

// Create old_root mountpoint
mkdir(old_root, 0755);

// Pivot root
syscall(SYS_pivot_root, rootfs, old_root);

// Unmount old root
chdir("/");
umount2("/old_root", MNT_DETACH);
rmdir("/old_root");
4

Execute user command

// Set hostname
sethostname("container", 9);

// Execute command
execve("/bin/sh", argv, envp);

Bonus Features

  • Network namespace with veth pair
  • User namespace for rootless containers
  • Seccomp filtering
  • Capability dropping

What You’ll Learn

  • How Docker/containerd actually work
  • Practical namespace manipulation
  • Cgroup setup and limits
  • Filesystem isolation with pivot_root

Project 2: Syscall Tracer with eBPF

Difficulty: ⭐⭐⭐⭐
Time: 6-8 hours
Skills: eBPF, kernel tracing, data structures

Goal

Build a strace-like tool using eBPF that can trace syscalls with minimal overhead.

Requirements

1

Set up BPF program to trace syscalls

// syscall_tracer.bpf.c
SEC("tracepoint/raw_syscalls/sys_enter")
int trace_enter(struct trace_event_raw_sys_enter *ctx)
{
    u64 id = bpf_get_current_pid_tgid();
    u32 pid = id >> 32;
    
    if (target_pid && pid != target_pid)
        return 0;
    
    struct syscall_event *event;
    event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
    if (!event)
        return 0;
    
    event->pid = pid;
    event->tid = id;
    event->syscall_nr = ctx->id;
    event->timestamp = bpf_ktime_get_ns();
    bpf_get_current_comm(&event->comm, sizeof(event->comm));
    
    // Capture first 6 arguments
    event->args[0] = ctx->args[0];
    event->args[1] = ctx->args[1];
    event->args[2] = ctx->args[2];
    event->args[3] = ctx->args[3];
    event->args[4] = ctx->args[4];
    event->args[5] = ctx->args[5];
    
    bpf_ringbuf_submit(event, 0);
    return 0;
}
2

Capture return values

SEC("tracepoint/raw_syscalls/sys_exit")
int trace_exit(struct trace_event_raw_sys_exit *ctx)
{
    u64 id = bpf_get_current_pid_tgid();
    
    struct exit_event *event;
    event = bpf_ringbuf_reserve(&exits, sizeof(*event), 0);
    if (!event)
        return 0;
    
    event->tid = id;
    event->ret = ctx->ret;
    event->timestamp = bpf_ktime_get_ns();
    
    bpf_ringbuf_submit(event, 0);
    return 0;
}
3

Build user-space consumer

// User-space: Read from ring buffer
static int handle_event(void *ctx, void *data, size_t len)
{
    struct syscall_event *e = data;
    
    printf("[%s:%d] %s(",
           e->comm, e->pid, syscall_name(e->syscall_nr));
    
    // Format arguments based on syscall type
    format_syscall_args(e->syscall_nr, e->args);
    
    printf(")\n");
    return 0;
}
4

Add filtering capabilities

  • Filter by PID
  • Filter by syscall type (file, network, memory)
  • Show only slow syscalls (> threshold)

Expected Output

$ ./syscall_tracer -p 1234
[myapp:1234] openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 3
[myapp:1234] fstat(3, {...}) = 0
[myapp:1234] read(3, "root:x:0:0:..."..., 4096) = 2847
[myapp:1234] close(3) = 0

Bonus Features

  • Latency histogram per syscall
  • Argument decoding (file paths, flags)
  • Export to JSON format
  • Integration with flame graphs

Project 3: Memory Leak Detector

Difficulty: ⭐⭐⭐⭐
Time: 6-8 hours
Skills: eBPF, memory management, stack traces

Goal

Build a tool that tracks memory allocations and identifies leaks in running processes.

Approach

// Track allocations
SEC("uprobe/libc.so:malloc")
int trace_malloc(struct pt_regs *ctx)
{
    u64 size = PT_REGS_PARM1(ctx);
    u64 id = bpf_get_current_pid_tgid();
    
    // Store size for ret probe
    bpf_map_update_elem(&alloc_sizes, &id, &size, BPF_ANY);
    return 0;
}

SEC("uretprobe/libc.so:malloc")
int trace_malloc_ret(struct pt_regs *ctx)
{
    u64 addr = PT_REGS_RC(ctx);
    u64 id = bpf_get_current_pid_tgid();
    
    u64 *size = bpf_map_lookup_elem(&alloc_sizes, &id);
    if (!size)
        return 0;
    
    struct alloc_info info = {
        .size = *size,
        .timestamp = bpf_ktime_get_ns(),
    };
    bpf_get_stack(ctx, &info.stack, sizeof(info.stack), BPF_F_USER_STACK);
    
    bpf_map_update_elem(&allocations, &addr, &info, BPF_ANY);
    bpf_map_delete_elem(&alloc_sizes, &id);
    
    return 0;
}

// Track deallocations
SEC("uprobe/libc.so:free")
int trace_free(struct pt_regs *ctx)
{
    u64 addr = PT_REGS_PARM1(ctx);
    
    // Remove from allocations map
    bpf_map_delete_elem(&allocations, &addr);
    
    return 0;
}

Output Format

$ ./memleak -p 1234 30
Attaching to process 1234...
Tracing for 30 seconds...

[08:23:15] Top outstanding allocations:

24576 bytes in 12 allocations from:
    malloc+0x0
    json_parse+0x123
    handle_request+0x456
    main+0x789

8192 bytes in 4 allocations from:
    malloc+0x0
    create_buffer+0x55
    process_data+0x123
    worker_thread+0x456

Project 4: Production CPU Profiler

Difficulty: ⭐⭐⭐⭐⭐
Time: 8-10 hours
Skills: Perf events, stack unwinding, visualization

Goal

Build a sampling CPU profiler that generates flame graphs, suitable for production use.

Components

  1. Sampler: Use perf_event_open() for low-overhead sampling
  2. Stack Walker: Capture kernel and user stacks
  3. Aggregator: Collapse and count stacks
  4. Visualizer: Generate flame graph SVG

Key Implementation

// Set up perf event
struct perf_event_attr attr = {
    .type = PERF_TYPE_SOFTWARE,
    .config = PERF_COUNT_SW_CPU_CLOCK,
    .sample_period = 10000000,  // ~100 Hz
    .sample_type = PERF_SAMPLE_CALLCHAIN | PERF_SAMPLE_TID,
    .exclude_kernel = 0,
    .exclude_user = 0,
};

int fd = perf_event_open(&attr, pid, -1, -1, 0);

// Memory map for reading samples
void *mmap_base = mmap(NULL, page_size * (1 + 2^n),
                       PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

// Process samples
struct perf_event_header *header;
while ((header = get_next_sample(mmap_base)) != NULL) {
    if (header->type == PERF_RECORD_SAMPLE) {
        process_sample((struct sample_event *)header);
    }
}

Flame Graph Generation

# Collapse stacks
stacks = {}
for sample in samples:
    stack_str = ";".join(reversed(sample.callchain))
    stacks[stack_str] = stacks.get(stack_str, 0) + 1

# Output folded format
for stack, count in stacks.items():
    print(f"{stack} {count}")

Project 5: Network Connection Tracker

Difficulty: ⭐⭐⭐
Time: 4-6 hours
Skills: eBPF, networking, state machines

Goal

Build a tool that tracks all TCP connections with latency metrics.

Features

  • Track connection establishment latency
  • Track connection duration
  • Group by remote IP/port
  • Show retransmission rates

BPF Program

SEC("kprobe/tcp_v4_connect")
int trace_connect(struct pt_regs *ctx)
{
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    u64 ts = bpf_ktime_get_ns();
    
    bpf_map_update_elem(&connect_start, &sk, &ts, BPF_ANY);
    return 0;
}

SEC("kprobe/tcp_rcv_state_process")
int trace_state_change(struct pt_regs *ctx)
{
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    int state = BPF_CORE_READ(sk, __sk_common.skc_state);
    
    if (state == TCP_ESTABLISHED) {
        u64 *start_ts = bpf_map_lookup_elem(&connect_start, &sk);
        if (start_ts) {
            u64 latency = bpf_ktime_get_ns() - *start_ts;
            emit_event(sk, latency);
            bpf_map_delete_elem(&connect_start, &sk);
        }
    }
    return 0;
}

Study Plan Integration

Week 1-2: Foundation

  • Complete Project 1 (Container from Scratch)
  • Understand namespaces and cgroups deeply

Week 3-4: Tracing

  • Complete Project 2 (Syscall Tracer)
  • Master eBPF basics

Week 5-6: Memory

  • Complete Project 3 (Memory Leak Detector)
  • Understand memory allocation internals

Week 7-8: Production

  • Complete Project 4 or 5
  • Focus on production debugging skills

Interview Discussion Points

When discussing these projects in interviews:
  1. Design decisions: Why did you choose this approach?
  2. Trade-offs: What are the limitations?
  3. Production readiness: How would you make it production-safe?
  4. Extensions: How would you add feature X?
  5. Debugging: How did you debug issues while building it?
Important: Don’t just copy code - understand every line. Interviewers will ask follow-up questions to verify your understanding.

Resources


Next: Interview Questions →