> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Kubernetes Configuration
> Managing application configuration with ConfigMaps and Secrets
# Kubernetes Configuration
Decouple configuration artifacts from image content to keep containerized applications portable. Think of it this way: your container image is the recipe, and ConfigMaps/Secrets are the ingredients list that changes depending on whether you are cooking for a dinner party (production) or a quick lunch (development). You never bake the ingredients into the recipe book itself.
***
## 1. ConfigMaps
Used to store non-confidential data in key-value pairs.
* Environment variables
* Command-line arguments
* Configuration files
### Creating ConfigMaps
**Imperative (CLI):**
```bash theme={null}
kubectl create configmap app-config --from-literal=LOG_LEVEL=info
```
**Declarative (YAML):**
```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
# Key-value pair
database_host: "postgres-service"
log_level: "debug"
# File content
nginx.conf: |
server {
listen 80;
server_name localhost;
}
```
### Using ConfigMaps in Pods
**As Environment Variables:**
```yaml theme={null}
spec:
containers:
- name: app
env:
# Single variable
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: app-config
key: log_level
# All variables from ConfigMap
envFrom:
- configMapRef:
name: app-config
```
**As a Volume (Mounting files):**
```yaml theme={null}
spec:
containers:
- name: nginx
volumeMounts:
- name: config-volume
mountPath: /etc/nginx/conf.d
volumes:
- name: config-volume
configMap:
name: app-config
items:
- key: nginx.conf
path: default.conf
```
***
## 2. Secrets
Used to store sensitive information, such as passwords, OAuth tokens, and SSH keys. Secrets are structurally identical to ConfigMaps -- the key difference is how Kubernetes handles them: Secrets can be encrypted at rest in etcd, RBAC can restrict access more tightly, and kubelet stores them in tmpfs (memory, not disk) when mounted into pods.
* Stored in `etcd` (encrypted at rest if configured -- and you should always configure this in production).
* Mounted into pods as files or environment variables.
* **Base64 encoded** (not encrypted by default in YAML!). This is the single biggest misconception about Kubernetes Secrets. Base64 is an encoding, not encryption -- anyone with `base64 -d` can read your "secret."
### Creating Secrets
**Imperative:**
```bash theme={null}
kubectl create secret generic db-pass --from-literal=password=supersecret123
```
**Declarative:**
```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
name: db-pass
type: Opaque
data:
# Must be Base64 encoded! This is NOT encryption.
# echo -n "supersecret123" | base64 => c3VwZXJzZWNyZXQxMjM=
# echo "c3VwZXJzZWNyZXQxMjM=" | base64 -d => supersecret123
# Anyone who can read this YAML can decode the password instantly.
password: c3VwZXJzZWNyZXQxMjM=
# Alternative: use stringData to avoid base64 encoding manually.
# Kubernetes will encode it for you. Convenient, but still not secure in Git.
# stringData:
# password: supersecret123
```
### Using Secrets in Pods
**As Environment Variables:**
```yaml theme={null}
spec:
containers:
- name: app
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-pass
key: password
```
**As a Volume:**
```yaml theme={null}
spec:
containers:
- name: app
volumeMounts:
- name: secret-volume
mountPath: /etc/certs
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: ssl-certs
```
***
## Best Practices
Never commit YAML files containing Base64 encoded secrets. Use tools like **Sealed Secrets**, **External Secrets Operator**, or **Vault**.
Set `immutable: true` for ConfigMaps/Secrets to prevent accidental updates and improve performance.
If you mount a ConfigMap as a volume, updates propagate to the file automatically (eventually). Apps need to watch the file for changes to reload without restart.
***
## Resource Quotas
Limit resource consumption per namespace. In a shared cluster, ResourceQuotas are your firewall against the "noisy neighbor" problem -- without them, one team's runaway deployment can consume all cluster resources and starve everyone else. Think of it as a budget: each namespace gets an allocation, and Kubernetes rejects requests that would exceed it.
```yaml theme={null}
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: development
spec:
hard:
requests.cpu: "10" # Total CPU requests across all pods in this namespace
requests.memory: 20Gi # Total memory requests
limits.cpu: "20" # Total CPU limits
limits.memory: 40Gi # Total memory limits
pods: "50" # Maximum number of pods (prevents runaway scaling)
services: "10" # Maximum services (prevents port exhaustion)
secrets: "20" # Maximum secrets (prevents unbounded growth)
configmaps: "20" # Maximum configmaps
```
**Interview Tip**: ResourceQuotas are enforced at the namespace level. When a quota is set, all pods must specify resource requests/limits.
***
## LimitRanges
Set default and max/min resource constraints for pods/containers in a namespace. If ResourceQuotas are the budget, LimitRanges are the spending policies -- they set guardrails on individual pods rather than the total namespace allocation. Critically, LimitRanges inject defaults: if a developer forgets to set resource requests/limits, the LimitRange fills them in automatically.
```yaml theme={null}
apiVersion: v1
kind: LimitRange
metadata:
name: resource-limits
namespace: development
spec:
limits:
- type: Container
default: # Default limits (applied if none specified in pod spec)
cpu: "500m"
memory: "256Mi"
defaultRequest: # Default requests (used by scheduler for placement)
cpu: "100m"
memory: "128Mi"
max: # Maximum allowed -- pods exceeding this are REJECTED
cpu: "2"
memory: "2Gi"
min: # Minimum required -- prevents tiny pods that waste scheduling overhead
cpu: "50m"
memory: "64Mi"
```
**Production gotcha**: When a ResourceQuota is set on a namespace, Kubernetes requires EVERY pod to specify resource requests and limits. If a pod does not specify them, it is rejected with "must specify requests/limits." LimitRanges with defaults solve this by auto-injecting values, preventing broken deployments when quotas are enabled.
***
## Priority Classes
Control pod scheduling and eviction priority.
```yaml theme={null}
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "Critical production workloads"
preemptionPolicy: PreemptLowerPriority
---
# Use in Pod spec
spec:
priorityClassName: high-priority
```
| Built-in Priority | Value | Use Case |
| ------------------------- | ---------- | ------------------ |
| `system-cluster-critical` | 2000000000 | kube-system pods |
| `system-node-critical` | 2000001000 | Node-critical pods |
| Custom high priority | 1000000 | Production apps |
| Default (none set) | 0 | Standard workloads |
***
## Secrets Management Best Practices
### 1. External Secrets Operator
Sync secrets from external sources (AWS Secrets Manager, HashiCorp Vault). This is the gold standard for production secrets management -- the secret value never appears in Git, never exists in a YAML file, and is fetched directly from a centralized secret store.
```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: 1h # How often to sync from the external source.
# Shorter = faster rotation, more API calls.
secretStoreRef:
name: aws-secrets-manager # Reference to a SecretStore resource
kind: SecretStore # that configures AWS credentials and region
target:
name: db-secret # The Kubernetes Secret that will be created/updated
data:
- secretKey: password # Key in the Kubernetes Secret
remoteRef:
key: prod/database # Path in AWS Secrets Manager
property: password # JSON property within the secret value
```
### 2. Sealed Secrets
Encrypt secrets that can be safely stored in Git.
```bash theme={null}
# Create SealedSecret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml
```
### 3. Enable etcd Encryption
Encrypt secrets at rest in etcd. Without this, anyone with access to etcd (or its backups) can read every secret in plain text. This is a compliance requirement for PCI DSS, HIPAA, and SOC2.
```yaml theme={null}
# /etc/kubernetes/enc/enc.yaml
# This file is passed to kube-apiserver via --encryption-provider-config.
# After enabling, you must re-encrypt existing secrets:
# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: # Generate with: head -c 32 /dev/urandom | base64
- identity: {} # Fallback: read unencrypted secrets written before encryption was enabled.
# NOTE: provider order matters. The FIRST provider is used for writes.
# identity: {} as the last entry allows reading old unencrypted secrets
# during migration. Remove it after all secrets are re-encrypted.
```
***
## Interview Questions & Answers
| Aspect | ConfigMap | Secret |
| ---------------------- | -------------------- | -------------------------- |
| **Purpose** | Non-sensitive config | Sensitive data |
| **Encoding** | Plain text | Base64 encoded |
| **At-rest encryption** | No | Optional (etcd encryption) |
| **Size limit** | 1MB | 1MB |
| **Mounting** | Env vars or volumes | Env vars or volumes |
**Not by default!**
* Secrets are only Base64 encoded, not encrypted
* Anyone with RBAC access can read them
* They're stored in etcd (unencrypted by default)
**To secure secrets:**
* Enable etcd encryption at rest
* Use RBAC to restrict access
* Use external secret management (Vault, AWS Secrets Manager)
* Use Sealed Secrets for GitOps
**Option 1: Volume mount** - ConfigMap changes propagate automatically (with delay)
* App must watch file for changes
**Option 2: Trigger rollout manually:**
```bash theme={null}
kubectl rollout restart deployment/my-app
```
**Option 3: Use Reloader** - Automatically restarts pods when ConfigMap changes:
```yaml theme={null}
annotations:
reloader.stakater.com/auto: "true"
```
ResourceQuotas prevent resource exhaustion in multi-tenant clusters by:
* Limiting total CPU/memory per namespace
* Limiting number of objects (pods, services, secrets)
* Enforcing resource requests/limits on all pods
Without quotas, one team could consume all cluster resources.
1. **External Secrets Operator**: Auto-syncs from external sources on schedule
2. **Volume-mounted secrets**: Kubernetes updates files (apps must reload)
3. **Rolling restart**: `kubectl rollout restart` to pick up new secrets
4. **Sidecar pattern**: Use a sidecar that watches for secret changes
***
## Common Pitfalls
**1. Base64 ≠ Encryption**: Base64 is encoding, not encryption. Anyone can decode it.
**2. Committing Secrets to Git**: Never commit Secret YAMLs with actual values. Use Sealed Secrets or External Secrets.
**3. Not Setting ResourceQuotas**: Without quotas, a runaway pod can consume all cluster resources.
**4. Forgetting LimitRanges**: Without defaults, pods without resource specs can consume unlimited resources.
**5. Large ConfigMaps**: ConfigMaps have a 1MB limit. For larger configs, use a sidecar to fetch from external storage.
***
## Key Takeaways
* Use **ConfigMaps** for plain text config.
* Use **Secrets** for sensitive data.
* Inject as **Environment Variables** for simple values.
* Mount as **Volumes** for config files.
* **Base64 is NOT encryption!**
* Use **ResourceQuotas** and **LimitRanges** for multi-tenant clusters.
* Implement proper **secrets management** with external tools.
***
## Interview Deep-Dive
**Strong Answer:**
* First, I would check how the ConfigMap is consumed. If the pod injects the ConfigMap as environment variables, those are set at pod startup and never update automatically. A rolling restart is the only way to pick up changes.
* If the ConfigMap is mounted as a volume, Kubernetes does eventually propagate changes, but there is a propagation delay controlled by the kubelet sync period, which defaults to roughly 60 seconds but can take longer because of caching at the API server level (the `--sync-frequency` flag and the watch cache TTL).
* I would run `kubectl describe pod ` to confirm the mount type, then `kubectl exec` into the pod and cat the mounted file to see whether the filesystem reflects the new values yet.
* Even when the file updates, the application itself needs to watch the file for changes. Most apps (nginx, for example) do not hot-reload config files without a signal or restart. Tools like Reloader by Stakater solve this by annotating the deployment and triggering a rollout automatically when a referenced ConfigMap changes.
**Follow-up: What is the risk of using `envFrom` to inject an entire ConfigMap as environment variables in a large team?**
If someone adds a key to the ConfigMap that collides with a variable the application or the container runtime already uses (like `PATH`, `HOME`, or `HOSTNAME`), it silently overrides it and breaks the container in hard-to-diagnose ways. In my experience, explicitly mapping individual keys with `valueFrom.configMapKeyRef` is safer because it makes the contract visible in the pod spec. It also means a review catches new variable names before they ship.
**Strong Answer:**
* Immediate priority is credential rotation. Base64 is not encryption -- anyone who can read the repo history can decode those credentials in one command. I would rotate the database password immediately, even before cleaning up the repo.
* Then I would remove the file from Git history using `git filter-branch` or the BFG Repo Cleaner, because a simple `git rm` only removes it from the current tree. The old commit still contains the secret.
* Going forward, I would enforce a secrets management workflow: Sealed Secrets (kubeseal encrypts secrets client-side so the ciphertext is safe to commit), External Secrets Operator (syncs from AWS Secrets Manager or HashiCorp Vault so secrets never exist in YAML at all), or SOPS with age/KMS encryption on the YAML values.
* I would also add a pre-commit hook or CI check (something like `gitleaks` or `detect-secrets`) to scan for high-entropy strings and known secret patterns before code reaches the remote.
**Follow-up: Between Sealed Secrets and External Secrets Operator, when would you choose one over the other?**
Sealed Secrets is simpler and self-contained -- the encryption key lives in the cluster, and you commit encrypted YAML to Git. It works well for teams that want a pure GitOps flow with no external dependencies. External Secrets Operator is better when you already have a centralized secrets store (Vault, AWS Secrets Manager, GCP Secret Manager) and want secrets to be managed, rotated, and audited in one place. The trade-off is operational complexity: ESO adds another controller and an external dependency, but you get centralized rotation, audit trails, and cross-cluster consistency.
**Strong Answer:**
* ResourceQuotas set the ceiling per namespace -- total CPU, memory, and object counts. They answer the question "how much can this team consume in aggregate?"
* LimitRanges set defaults and constraints per pod or container -- minimum, maximum, and default values. They answer "how big can any single workload be?" and also inject defaults so that every pod has resource requests even if the developer forgets.
* PriorityClasses control eviction order and preemption. When the cluster is under memory pressure, the kubelet evicts BestEffort pods first, then Burstable, then Guaranteed. PriorityClasses let you further rank workloads: a payment-processing pod with priority 1000000 preempts a batch analytics pod with priority 100 during resource contention.
* The interaction is layered: LimitRanges ensure every pod in the namespace has resource requests, which is actually required once a ResourceQuota is set (pods without requests are rejected). PriorityClasses then determine scheduling and eviction behavior across namespaces when the cluster is overcommitted.
* In practice, I have seen teams set ResourceQuotas per namespace for dev and staging but leave production more permissive, because you do not want quota admission rejection to block a critical production scale-up at 3 AM.
**Follow-up: What happens if a pod exceeds its memory limit versus its CPU limit? Why is the behavior different?**
Memory is incompressible -- the kernel cannot reclaim memory from a running process without killing it. So when a container exceeds its memory limit, the OOM killer terminates it immediately. CPU is compressible -- the kernel can throttle a process by giving it fewer CPU cycles. So when a container exceeds its CPU limit, it gets throttled (its `cfs_period` is capped), but it stays alive. This distinction matters for debugging: high CPU throttling causes latency spikes but the pod stays up, while memory pressure causes restarts that show as OOMKilled in `kubectl describe pod`.
**Strong Answer:**
* The key is to mount the ConfigMap as a volume rather than using environment variables, and design the application to watch the mounted file for changes. When Kubernetes updates the volume (via the kubelet sync), the application detects the change and reloads its configuration in-place without restarting.
* For applications that do not natively support config file watching, I would use a sidecar container that watches the file (using inotifywait or a simple polling loop) and sends a signal (like SIGHUP) to the main process, or calls an admin reload endpoint.
* One subtlety: Kubernetes updates volume-mounted ConfigMaps using a symlink swap. The mount point is actually a symlink to a timestamped directory, and the update atomically changes the symlink target. This means the application sees the entire new config atomically, not a partially written file.
* Another approach is to use immutable ConfigMaps and create a new ConfigMap with each config change (e.g., `app-config-v2`), then update the Deployment reference to point to the new ConfigMap name. This triggers a rolling update but gives you explicit version control over configuration and easy rollback by switching back to the previous ConfigMap name.
**Follow-up: What is the propagation delay for volume-mounted ConfigMap updates, and what controls it?**
The delay is bounded by the kubelet sync period (default 1 minute) plus the API server cache TTL (the `--watch-cache-sizes` configuration). In practice, I have measured it at 30 seconds to 2 minutes in most clusters. The kubelet periodically reconciles its mounts against the API server state. If you need faster propagation, you can reduce the kubelet `--sync-frequency`, but this increases API server load because every node polls more frequently.
***
Next: [Kubernetes Storage →](/courses/devops-tools/kubernetes-storage)