> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Docker Fundamentals
> Core Docker concepts and commands
# Docker Fundamentals
Master the core concepts of Docker containerization and understand how containers revolutionized software deployment.
***
## What is Docker?
Docker is a platform for developing, shipping, and running applications in **containers**. Containers package your application with all its dependencies, ensuring it runs consistently across different environments.
Containers share the host OS kernel, using MBs instead of GBs
Run anywhere - laptop, server, cloud
Start in seconds, not minutes
Each container runs independently
***
## Images vs Containers
Understanding the difference is crucial -- confusing these two is the most common beginner mistake.
### Docker Images
* **Blueprint** for containers (like a class in OOP, or a recipe in cooking)
* Read-only template -- once built, an image never changes
* Contains application code, runtime, libraries, dependencies -- everything needed to run
* Stored in registries (Docker Hub, private registries like AWS ECR or Google GCR)
* Built from a Dockerfile using `docker build`
### Docker Containers
* **Running instance** of an image (like an object in OOP, or a cake baked from a recipe)
* Writable layer on top of the read-only image layers
* Isolated process with its own filesystem, network, and process tree
* Can be started, stopped, restarted, and deleted
* You can run multiple containers from the same image -- each is independent
```bash theme={null}
# Analogy
Image = Recipe # Defined once, used many times, never changes
Container = Cake # Each cake is independent; eating one doesn't affect others
# You can bake many cakes (containers) from one recipe (image)
# Each cake can have different frosting (environment variables, volumes)
```
**A senior engineer would say**: "Images are your build artifacts -- they should be immutable and versioned. Containers are your runtime instances -- they should be ephemeral and replaceable. If you need to fix something, rebuild the image; never SSH into a container and patch it."
***
## Docker Architecture
Docker uses a client-server architecture:
```
┌─────────────┐ ┌──────────────────┐ ┌─────────────┐
│ Docker │ REST │ Docker Daemon │ │ Docker │
│ Client │────────▶│ (dockerd) │◀────────│ Registry │
│ (CLI) │ API │ │ Pull/ │ (Docker Hub)│
└─────────────┘ └──────────────────┘ Push └─────────────┘
│
│ Manages
▼
┌──────────────┐
│ Containers │
│ Images │
│ Networks │
│ Volumes │
└──────────────┘
```
### Components
**Docker Client (`docker`)**:
* Command-line interface
* Sends commands to Docker daemon
* Can communicate with remote daemons
**Docker Daemon (`dockerd`)**:
* Background service
* Builds, runs, manages containers
* Listens for Docker API requests
**Docker Registry**:
* Stores Docker images
* Docker Hub (public)
* Private registries (AWS ECR, Google GCR, Azure ACR)
***
## Installing Docker
```bash theme={null}
# Download Docker Desktop from docker.com
# Or use winget
winget install Docker.DockerDesktop
# Verify installation
docker --version
docker run hello-world
```
```bash theme={null}
# Download Docker Desktop from docker.com
# Or use Homebrew
brew install --cask docker
# Verify
docker --version
docker run hello-world
```
```bash theme={null}
# Update package index
sudo apt update
# Install dependencies
sudo apt install apt-transport-https ca-certificates curl software-properties-common
# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io
# Add user to docker group (avoid sudo)
sudo usermod -aG docker $USER
# Verify
docker --version
docker run hello-world
```
***
## Essential Docker Commands
### Working with Images
```bash theme={null}
# Search for images
docker search nginx
# Pull image from Docker Hub
docker pull nginx
docker pull nginx:1.21 # Specific version
# List local images
docker images
docker image ls
# Inspect image
docker inspect nginx
# Remove image
docker rmi nginx
docker image rm nginx
# Remove unused images
docker image prune
docker image prune -a # Remove all unused
```
### Working with Containers
```bash theme={null}
# Run container
docker run nginx
docker run -d nginx # Detached mode (background)
docker run -d --name my-nginx nginx # With name
docker run -d -p 8080:80 nginx # Port mapping (host:container)
# List containers
docker ps # Running only
docker ps -a # All (including stopped)
# Stop container
docker stop my-nginx
docker stop $(docker ps -q) # Stop all running
# Start stopped container
docker start my-nginx
# Restart container
docker restart my-nginx
# Remove container
docker rm my-nginx
docker rm -f my-nginx # Force remove (even if running)
# Remove all stopped containers
docker container prune
```
### Container Interaction
```bash theme={null}
# View logs
docker logs my-nginx
docker logs -f my-nginx # Follow (live)
docker logs --tail 100 my-nginx # Last 100 lines
# Execute command in running container
docker exec my-nginx ls /usr/share/nginx/html
docker exec -it my-nginx bash # Interactive shell
# Copy files
docker cp my-nginx:/etc/nginx/nginx.conf ./nginx.conf
docker cp ./index.html my-nginx:/usr/share/nginx/html/
# View container stats
docker stats
docker stats my-nginx
# Inspect container
docker inspect my-nginx
```
***
## Practical Examples
### Example 1: Running Nginx Web Server
```bash theme={null}
# Pull and run nginx
docker run -d \
--name web-server \
-p 8080:80 \
nginx:alpine
# Verify it's running
curl http://localhost:8080
# View logs
docker logs web-server
# Stop and remove
docker stop web-server
docker rm web-server
```
### Example 2: Running a Database
```bash theme={null}
# Run PostgreSQL with environment variables for initial setup.
# -e sets environment variables that Postgres uses on first boot
# to create the default database and set the password.
docker run -d \
--name postgres-db \
-e POSTGRES_PASSWORD=mysecret \
-e POSTGRES_DB=myapp \
-p 5432:5432 \
postgres:14
# Connect to the database from inside the container
docker exec -it postgres-db psql -U postgres -d myapp
# Stop and remove (WARNING: data is lost without a volume!)
docker stop postgres-db
docker rm postgres-db
```
**Production gotcha**: Without a volume, all database data lives in the container's ephemeral write layer. When the container is removed, the data is gone forever. Always mount a volume for databases: `-v pgdata:/var/lib/postgresql/data`. This is the single most common "oh no" moment for Docker beginners.
### Example 3: Interactive Container
```bash theme={null}
# Run Ubuntu container interactively
docker run -it ubuntu:22.04 bash
# Inside container:
apt update
apt install curl
curl https://example.com
exit
# Container stops when you exit
```
***
## Container Lifecycle
```
┌─────────┐
│ Created │ docker create
└────┬────┘
│
▼
┌─────────┐
│ Running │ docker start / docker run
└────┬────┘
│
├──▶ Paused (docker pause)
│ │
│ └──▶ Running (docker unpause)
│
▼
┌─────────┐
│ Stopped │ docker stop
└────┬────┘
│
▼
┌─────────┐
│ Removed │ docker rm
└─────────┘
```
***
## Common Flags and Options
```bash theme={null}
# -d, --detach Run in background
# -p, --publish Port mapping (host:container)
# --name Assign name to container
# -e, --env Set environment variables
# -v, --volume Mount volume
# --rm Remove container when stopped
# -it Interactive terminal
# --network Connect to network
# --restart Restart policy
# Example with multiple flags
docker run -d \
--name my-app \
-p 3000:3000 \
-e NODE_ENV=production \
-v $(pwd):/app \
--restart unless-stopped \
node:18
```
***
## Environment Variables
Environment variables are the standard way to pass configuration into containers. This follows the [12-Factor App methodology](https://12factor.net/config) -- configuration lives outside the code, making the same image deployable to dev, staging, and production.
```bash theme={null}
# Set single variable -- useful for quick one-offs
docker run -e DATABASE_URL=postgres://localhost:5432/db myapp
# Set multiple variables -- common in production where you have many configs.
# Notice "db" instead of "localhost" -- container names resolve via Docker DNS.
docker run \
-e NODE_ENV=production \
-e PORT=3000 \
-e DATABASE_URL=postgres://db:5432/mydb \
myapp
# From file -- best practice for managing many variables.
# The .env file should NEVER be committed to version control.
docker run --env-file .env myapp
```
**Common mistake**: Passing secrets (API keys, database passwords) as `-e` flags in the command line. These show up in `docker inspect` output and shell history. For production secrets, use Docker secrets (Swarm), Kubernetes secrets, or a secrets manager like HashiCorp Vault. For local development, `--env-file .env` with `.env` in `.gitignore` is acceptable.
***
## Docker System Commands
Docker accumulates a surprising amount of disk space over time -- stopped containers, unused images, build cache layers. On a developer machine it is not unusual to reclaim 20-50GB after a cleanup.
```bash theme={null}
# View Docker disk usage -- shows images, containers, volumes, and build cache
docker system df
# Clean up stopped containers, unused networks, and dangling (untagged) images
docker system prune
# More aggressive: also remove ALL images not used by a running container
docker system prune -a
# Nuclear option: also remove unused volumes (WARNING: database data!)
docker system prune -a --volumes
# View Docker info (storage driver, OS, kernel version, registry config)
docker info
# View Docker client and server versions
docker version
```
**Practical tip**: Run `docker system df` weekly. If "Build Cache" is over 10GB, run `docker builder prune`. If "Images" is huge, run `docker image prune -a`. Avoid the `--volumes` flag unless you are certain no named volume contains data you need.
***
## Best Practices
```bash theme={null}
# Bad
docker pull nginx
# Good
docker pull nginx:1.21-alpine
```
Avoid `latest` tag in production - it's unpredictable.
```bash theme={null}
# Bad
docker run -d nginx
# Good
docker run -d --name web-server nginx
```
Makes management easier.
```bash theme={null}
# Remove stopped containers
docker container prune
# Remove unused images
docker image prune
# Remove everything unused
docker system prune -a
```
```bash theme={null}
# Container auto-removes when stopped
docker run --rm -it ubuntu bash
```
***
## Troubleshooting
### Container Won't Start
```bash theme={null}
# Check logs
docker logs container-name
# Inspect container
docker inspect container-name
# Check if port is already in use
netstat -an | grep 8080 # Linux/Mac
netstat -an | findstr 8080 # Windows
```
### Permission Denied
```bash theme={null}
# Linux: Add user to docker group
sudo usermod -aG docker $USER
# Log out and back in
# Or run with sudo
sudo docker ps
```
### Container Keeps Restarting
```bash theme={null}
# Check logs
docker logs container-name
# Run without restart policy
docker run -d --name test nginx
# Debug the issue
```
***
## Key Takeaways
* Docker containers are lightweight, portable, and fast
* Images are blueprints, containers are running instances
* Use `docker run` to create and start containers
* Use `docker ps` to list containers
* Use `docker logs` to debug
* Clean up regularly with `docker system prune`
***
## Container Internals (Critical for Interviews!)
Understanding what makes containers work under the hood is essential for senior roles.
### Linux Namespaces
Namespaces provide **isolation**. Each container gets its own view of:
| Namespace | Isolates |
| --------- | -------------------------------------------- |
| **PID** | Process IDs (container sees itself as PID 1) |
| **NET** | Network stack (own IP, ports, routing) |
| **MNT** | Filesystem mounts |
| **UTS** | Hostname and domain name |
| **IPC** | Inter-process communication |
| **USER** | User and group IDs |
### Control Groups (cgroups)
Cgroups provide **resource limiting**:
```bash theme={null}
# Limit memory to 512MB
docker run -m 512m nginx
# Limit CPU to 0.5 cores
docker run --cpus 0.5 nginx
# cgroups enforce these limits at kernel level
```
### Union Filesystem (OverlayFS)
Docker images use **layered filesystems**:
```
┌─────────────────────────┐
│ Container Layer (RW) │ ← Writable layer
├─────────────────────────┤
│ Application Layer │ ← COPY . .
├─────────────────────────┤
│ Dependencies Layer │ ← npm install
├─────────────────────────┤
│ Base Image Layer │ ← FROM node:18
└─────────────────────────┘
```
**Interview Insight**: Containers are NOT lightweight VMs. They're isolated processes sharing the host kernel, using namespaces for isolation and cgroups for resource limits.
***
## OCI (Open Container Initiative)
The industry standard for container formats and runtimes.
### OCI Standards
| Standard | Description |
| --------------------- | ---------------------------------------------- |
| **Image Spec** | How container images are built and distributed |
| **Runtime Spec** | How containers are executed |
| **Distribution Spec** | How images are pushed/pulled from registries |
### Container Runtimes
| Runtime | Description | Use Case |
| ------------------- | ------------------------------------- | ----------------- |
| **containerd** | Industry standard, used by Docker/K8s | Production |
| **CRI-O** | Lightweight, Kubernetes-native | Kubernetes |
| **runc** | Low-level runtime (OCI reference) | Building runtimes |
| **gVisor** | Sandboxed (user-space kernel) | Security-critical |
| **Kata Containers** | VM-based isolation | Multi-tenancy |
***
## Docker Security Best Practices
### 1. Run as Non-Root
```dockerfile theme={null}
# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
```
### 2. Read-Only Filesystem
```bash theme={null}
docker run --read-only nginx
```
### 3. Drop Capabilities
```bash theme={null}
# Run with minimal capabilities
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE nginx
```
### 4. Security Scanning
```bash theme={null}
# Scan image for vulnerabilities
docker scout cves nginx:latest
# Using Trivy
trivy image nginx:latest
```
***
## Interview Questions & Answers
| Aspect | Container | Virtual Machine |
| ------------------ | -------------------------- | --------------------------- |
| **Isolation** | Process-level (namespaces) | Hardware-level (hypervisor) |
| **Kernel** | Shares host kernel | Has own kernel |
| **Size** | MBs | GBs |
| **Startup** | Seconds | Minutes |
| **Resource Usage** | Low overhead | High overhead |
| **Security** | Weaker isolation | Stronger isolation |
**Namespaces**: Provide isolation (what a container can see)
* PID namespace: Own process tree
* NET namespace: Own network stack
* MNT namespace: Own filesystem view
**cgroups**: Provide resource limits (what a container can use)
* CPU limits
* Memory limits
* I/O bandwidth
1. Docker CLI sends request to Docker daemon
2. Daemon checks if image exists locally
3. If not, pulls from registry (Docker Hub)
4. Creates a new container from image layers
5. Allocates a read-write layer on top
6. Creates network interface and assigns IP
7. Starts the container process with namespaces and cgroups
8. Executes the CMD/ENTRYPOINT
| Instruction | Behavior | Overridable |
| -------------- | ------------------------- | ------------------------- |
| **CMD** | Default command/arguments | Yes, with docker run args |
| **ENTRYPOINT** | Main executable | Only with --entrypoint |
**Best Practice**: Use ENTRYPOINT for the executable, CMD for default arguments:
```dockerfile theme={null}
ENTRYPOINT ["python", "app.py"]
CMD ["--port", "8080"]
```
1. **Check logs**: `docker logs container_name`
2. **Inspect**: `docker inspect container_name`
3. **Run interactively**: `docker run -it image sh`
4. **Override entrypoint**: `docker run --entrypoint sh image`
5. **Check events**: `docker events`
6. **Resource issues**: `docker stats`
BuildKit is the modern builder for Docker images:
* **Parallel builds**: Build independent stages concurrently
* **Better caching**: Cache mounts for package managers
* **Secret mounting**: Don't bake secrets into layers
* **SSH forwarding**: Clone private repos during build
Enable: `DOCKER_BUILDKIT=1 docker build .`
***
## Common Pitfalls
**1. Using `latest` Tag**: Unpredictable. Always use specific version tags.
**2. Running as Root**: Security risk. Create and use a non-root user.
**3. Large Images**: Slow pulls and more attack surface. Use Alpine or distroless.
**4. Not Cleaning Up**: Disk fills up fast. Run `docker system prune` regularly.
**5. Storing Data in Containers**: Data is lost when container dies. Use volumes!
**6. Ignoring Resource Limits**: Containers can consume all host resources without limits.
***
## Interview Deep-Dive
**Strong Answer:**
* First, I check the application logs: `docker logs --tail 200 container_name`. Most issues surface here -- unhandled exceptions, configuration errors, or "listening on wrong address" messages. I look for any error or panic in the last few hundred lines.
* If logs look normal, I check resource constraints: `docker stats container_name`. The container might be CPU-throttled (showing high CPU percentage near its limit) or memory-constrained. I also check `docker inspect --format='{{.State.OOMKilled}}' container_name` -- if the application was OOM-killed and restarted, it might be in a crash loop where it starts, allocates memory, gets killed, restarts, and the cycle repeats.
* Next, I try to exec into the container: `docker exec -it container_name sh`. If the shell opens, I check if the process is actually running (`ps aux`), what ports it is listening on (`netstat -tlnp` or `ss -tlnp`), and whether it can reach its dependencies (e.g., `nc -zv db 5432` for a database).
* If the process is listening on `127.0.0.1` instead of `0.0.0.0`, that is the bug -- the application is only accepting connections from inside the container, not from the Docker bridge network. This is extremely common with Node.js, Python Flask, and Go HTTP servers where the default bind address is localhost.
* If I cannot exec in (distroless image, no shell), I use a debug sidecar: `docker run --net container:target_container nicolaka/netshoot` gives me network tools in the target container's network namespace.
* Finally, I check if the port mapping is correct: `docker port container_name` should show the expected host-to-container mapping. A mismatch here means traffic is not reaching the container at all.
**Follow-up: The application binds to 0.0.0.0:3000 and the port mapping is correct, but requests from the host still time out. What else could it be?**
I would check if a host-level firewall (iptables, nftables, Windows Firewall) is blocking the port. On Linux, Docker manipulates iptables rules for port forwarding, and custom firewall rules can conflict. I would run `sudo iptables -t nat -L -n | grep 3000` to verify the DNAT rule exists. On macOS/Windows, Docker Desktop runs inside a VM, so "host" means the VM, not the laptop -- and Docker Desktop's network layer can occasionally need a restart. I would also check if another process on the host already has that port bound: `lsof -i :3000` (Linux/macOS) or `netstat -an | findstr 3000` (Windows).
**Strong Answer:**
* The Docker CLI sends a POST to the Docker daemon at `/var/run/docker.sock`. The daemon delegates to containerd via gRPC, which in turn invokes runc -- the low-level OCI runtime that does the actual kernel work.
* **Namespace creation**: runc creates a new set of Linux namespaces using the `clone()` syscall with flags like `CLONE_NEWPID` (process isolation), `CLONE_NEWNET` (network isolation), `CLONE_NEWNS` (mount isolation), `CLONE_NEWUTS` (hostname isolation), and `CLONE_NEWIPC` (IPC isolation). The nginx process believes it is PID 1 in its own world.
* **Cgroup setup**: runc writes to `/sys/fs/cgroup/memory/docker//memory.limit_in_bytes` to set the 512MB memory limit. It also configures the memory cgroup's OOM killer behavior. If nginx exceeds 512MB, the kernel OOM killer terminates the process -- no graceful shutdown, just gone.
* **Filesystem setup**: runc mounts the image layers using OverlayFS. The read-only image layers become the "lower" directories, and a fresh writable layer becomes the "upper" directory. The unified mount point becomes the container's root filesystem. This is copy-on-write: reads come from lower layers, writes go to the upper layer.
* **Networking**: Docker creates a virtual ethernet pair (veth). One end goes into the container's network namespace (visible as `eth0` inside the container), the other connects to the `docker0` bridge on the host. Docker then inserts an iptables DNAT rule that forwards TCP traffic arriving at the host on port 8080 to the container's IP on port 80.
* **Process execution**: runc executes the nginx entrypoint. runc itself exits after setup -- it is not a long-running daemon. containerd monitors the running process via its shim.
**Follow-up: Why does runc exit after starting the container? What monitors the process afterward?**
This is a deliberate architectural decision. runc is a "fire and forget" binary -- it sets up the namespaces, cgroups, and mounts, starts the process, and exits. A lightweight shim process (containerd-shim) takes over as the container's parent process. This design means that containerd (and even dockerd) can be restarted without affecting running containers. In earlier Docker versions, restarting the Docker daemon killed all containers. The shim architecture solved this -- containers are now decoupled from the daemon lifecycle.
**Strong Answer:**
* `latest` is not a version -- it is a moving pointer. Docker Hub maintainers update the `latest` tag whenever they push a new image. There is no guarantee about what `latest` pointed to yesterday versus today.
* Concrete scenario: On Monday, your CI/CD pipeline builds and deploys successfully. `node:latest` resolves to Node 18.19.0. On Wednesday, the Node.js team pushes Node 20.11.0 and updates the `latest` tag. On Thursday, your pipeline runs again, pulls Node 20, and now your application breaks because a dependency is incompatible with the new V8 engine. You have changed zero lines of code, yet your build is broken.
* Worse: this failure is non-deterministic. The exact same Dockerfile produces different images on different days. You cannot reproduce the Monday build because `latest` no longer points to Node 18. Your Docker layer cache may mask this locally (you still have the old image cached), but CI runners typically start fresh.
* The fix: always pin to a specific version and variant: `node:18.19.0-alpine3.19`. This guarantees byte-identical builds across time and environments. Update the version intentionally, not accidentally, by changing the tag in the Dockerfile and testing the upgrade.
* For extra safety in regulated environments, pin to the image digest: `node@sha256:abc123...`. This is immutable -- even if someone retags the image, the digest never changes.
**Follow-up: How do you balance pinning versions with keeping dependencies up to date?**
I use automated dependency update tools like Dependabot or Renovate. They submit pull requests when new base image versions are available. The PR triggers the CI pipeline, which catches any incompatibilities before they reach production. This gives you the reproducibility of pinned versions with the currency of automated updates. The key is that updates are intentional (a PR you review and merge) rather than implicit (a tag that shifts under you silently).
***
Next: [Building Docker Images →](/courses/devops-tools/docker-images)