Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

​

Part XXVI — Product Thinking and Leadership

​

Chapter 33: Product Thinking

Big Word Alert: Technical Leadership. Leading through influence rather than authority. A senior/staff engineer does not manage people — they shape technical direction, make architectural decisions, mentor engineers, and bridge the gap between product goals and technical implementation. The skill that separates senior from staff: the ability to make an entire team more effective, not just yourself.

​

33.1 Business Awareness and Product Thinking

1. What problem are we solving? Not “what feature are we building” — what user pain or business need does this address? If nobody can articulate the problem clearly, the feature is not ready for engineering.
2. Who has this problem, and how badly? Is this 10,000 users hitting it daily, or 3 enterprise clients who mentioned it once? The answer shapes how much you invest.
3. What is the simplest thing that could work? Not the simplest thing that is technically elegant — the simplest thing that solves the user’s problem. Sometimes that is a manual process. Sometimes it is a spreadsheet. Sometimes it is an existing tool with a thin integration layer.
4. What would we learn by shipping V1? If the answer is “nothing — we already know this is the right solution,” you are either very confident or not asking hard enough questions.
5. What is the cost of being wrong? If this feature fails, is it a wasted sprint or a wasted quarter? This determines how much validation you need before building.

​

33.2 How Engineers Should Engage with PMs

• Participate in user research. Sit in on user interviews. Read support tickets. Watch session recordings. When you understand the user’s pain firsthand, you build better solutions — you stop building features and start solving problems.
• Challenge the spec, not the PM. When a PM says “build X,” the right response is not “okay” or “no” — it is “What problem does X solve? Have we validated that users actually want this? Is there a simpler version that tests the hypothesis?” This is not pushback — it is collaboration.
• Propose alternatives. PMs know the problem; engineers know what is cheap or expensive to build. If a PM asks for a complex recommendation engine, and you know that a simple “most popular” list would cover 80% of the value at 5% of the cost, say so. PMs cannot make good trade-offs without engineering input on feasibility and cost.
• Own the outcome, not just the output. Shipping the feature is not the finish line. Did it move the metric? Are users adopting it? What does the data say after two weeks? Engineers who care about outcomes earn a seat at the product table.
• Speak the language of impact. Instead of “I refactored the caching layer,” say “I reduced API latency by 40%, which should improve the conversion rate on the checkout page.” Connect your work to business outcomes that PMs and stakeholders care about.

​

33.3 MVP Thinking

​

Chapter 34: Ownership and Senior Behavior

​

34.1 Ownership

​

34.2 Leadership Without Authority

1. Influence through expertise. Become the person whose technical opinion people seek out. This means going deep in your domain, staying current, and being right often enough that people trust your judgment. Write thorough design docs. Give well-reasoned code reviews. When you say “this approach will cause problems at scale,” people should believe you because you have been right before.
2. Build trust incrementally. Trust is earned through consistency, not grand gestures. Deliver what you promise. Be honest about what you do not know. Admit mistakes quickly. Follow through on commitments. Over months, this compounds into genuine authority.
3. Lead by example. Write the documentation you wish existed. Set up the monitoring dashboard before the incident happens. Refactor the messy code without being asked. When you consistently model the behavior you want to see, others follow — not because you told them to, but because they see it works.
4. Create leverage through systems. One engineer writing good code improves one codebase. One engineer who creates a linting rule, a design doc template, or an architectural guideline improves every codebase. Think in terms of multipliers: what can you create once that helps the entire team forever?
5. Invest in relationships across teams. Attend cross-team meetings. Help other teams debug their issues. Understand their roadmaps and constraints. When you eventually need their cooperation on a shared project, they will remember.
6. Make decisions visible. Write ADRs (Architecture Decision Records). Send summary emails after technical discussions. Document the “why” behind choices. This makes your thinking transparent and builds confidence in your judgment across the organization.

​

34.3 How Senior Engineers Communicate

​

34.4 Handling Conflict

​

34.5 Running Effective Engineering Meetings

• The author writes a design doc (RFC) and shares it at least 24-48 hours in advance. No cold reads in the meeting — if people have not read the doc, reschedule. Reading time is preparation, not meeting time.
• The doc should cover: problem statement, proposed solution, alternatives considered (and why they were rejected), trade-offs, open questions the author wants feedback on.
• Assign 2-3 reviewers explicitly. “Everyone is invited” means “nobody prepared.”

1. 5 minutes: Author gives a 5-minute summary. Not a re-reading of the doc — a summary that highlights the key decisions and the open questions.
2. 10 minutes: Clarifying questions only. “I do not understand how X handles Y” — not “I think X should use Z instead.” Separate understanding from critique.
3. 30 minutes: Discussion of trade-offs, risks, and alternatives. Focus on the author’s open questions first — they flagged those because they genuinely need input. Then address concerns from reviewers.
4. 10 minutes: Decision and next steps. End with a clear outcome: approved, approved with changes, needs revision. Document the decision and the reasoning in the doc itself.
5. 5 minutes: Buffer for overflow.

1. Pre-groom ruthlessly. Sprint planning should not be the first time the team sees a ticket. The top 10-15 items in the backlog should already be estimated, have clear acceptance criteria, and have had technical spikes completed for anything uncertain. If the team is seeing a ticket for the first time in planning, the backlog grooming process is broken.
2. Plan to 70% capacity, not 100%. Every sprint has interruptions: production incidents, urgent bug fixes, unexpected complexity, someone getting sick. If you plan to 100%, you will miss the sprint goal every time. Planning to 70% gives you a buffer that keeps commitments reliable — and reliable commitments build trust with product.
3. Identify dependencies on day one. If a ticket depends on another team’s API, a design decision that has not been finalized, or an infrastructure change — flag it immediately. Dependencies are the number one cause of sprint failures because they are invisible until they block you.
4. Define “done” before you start. “Done” means deployed to production, monitored, and documented — not “code is written and sitting in a PR.” If the team does not agree on what done means, they will disagree on whether the sprint was successful.
5. End with a clear sprint goal. Not a list of tickets — a goal. “Users can complete the checkout flow with Apple Pay” is a sprint goal. “Finish JIRA-1234, JIRA-1235, JIRA-1236” is a task list. Goals give the team permission to make trade-offs during the sprint: if something blocks JIRA-1235 but the sprint goal is still achievable through JIRA-1234 and JIRA-1236, the team can adapt without panic.

​

34.6 Technical Roadmap Planning

1. Start with the business roadmap. What does the company need to achieve in the next 6-12 months? Revenue targets, new markets, new product lines, scale milestones. If you do not know, ask. If nobody can tell you, that is a red flag worth raising.
2. Identify the technical enablers and blockers. For each business goal, ask: “What technical capability do we need that we do not have today?” and “What technical debt or limitation will prevent us from achieving this?” Example: “The business wants to expand to Europe. That requires GDPR compliance, data residency, and multi-region deployment — none of which our current architecture supports.”
3. Prioritize using impact and urgency. Not all technical investments are equal. Use a simple 2x2: high-impact / high-urgency items go first (they enable near-term business goals). High-impact / low-urgency items go in the “Next” horizon. Low-impact items get deprioritized or cut.
4. Sequence for dependencies. Some investments unlock others. If you need multi-region deployment for Europe expansion, and multi-region requires a data replication strategy, and replication requires migrating off the legacy database — that dependency chain determines the order, regardless of what business stakeholders want first.
5. Define success metrics for each initiative. “Migrate to the new database” is not measurable. “Reduce p99 query latency from 800ms to 200ms and support 10x current write throughput” is measurable. Metrics make it possible to evaluate whether the investment paid off.
6. Communicate with a narrative, not a spreadsheet. A roadmap document should tell a story: “Here is where we are today, here is where the business needs us to be, here are the gaps, and here is how we plan to close them.” The Communication and Soft Skills chapter covers the presentation mechanics — how to structure the narrative for different audiences (executives want the 2-minute version with business impact; the engineering team wants the 15-minute version with technical depth and sequencing).

​

Part XXVII — Prioritization, Execution, and Technical Debt

​

Chapter 35: Execution

Big Word Alert: Scope Creep. The gradual, uncontrolled expansion of a project’s scope beyond its original objectives. “Can we also add…” and “While we are at it…” are the warning signs. Scope creep is the most common reason projects miss deadlines. Prevention: clearly define what is out of scope before starting, push back on additions (or explicitly accept the timeline impact), use a parking lot list for “good ideas for later.”

​

35.1 Risk-First Delivery

​

35.2 Incremental Delivery

​

35.3 Prioritization Frameworks

​

RICE Scoring

​

Portfolio Prioritization — Choosing What Not to Fund

​

Project Cancellation — When to Kill Your Darlings

1. The business case has changed. The market shifted, a competitor made it irrelevant, or the customer who requested it churned. If the “why” no longer exists, the “what” should not either.
2. Persistent execution failure. The project has missed 3 consecutive milestones and the root cause is fundamental (wrong technology choice, unclear requirements that cannot be clarified, team skill mismatch), not incidental (someone was sick, a dependency was late once).
3. Opportunity cost exceeds value. Even if the project would succeed, the engineers working on it could deliver more value elsewhere. This is the hardest signal to act on because it requires comparing a known project to hypothetical alternatives.
4. The integration cost has exploded. What was scoped as a 6-week project now requires changes to 4 other services, a database migration, and a new deployment pipeline. The project itself might be fine, but the total cost of delivery has tripled.

• Write a cancellation RFC. Document why the project is being cancelled, what was learned, and what (if anything) is being preserved. This prevents the same project from being re-proposed in 6 months by someone who was not around for the first attempt.
• Celebrate the learning, not the failure. “We learned that our customers do not actually want real-time sync — they want reliable eventual sync. That learning saved us from building and maintaining a complex system nobody would use.”
• Reassign the team immediately. Engineers in limbo — “we might restart this” — are demoralized and unproductive. Either the project is cancelled and the team moves on, or it is not cancelled and the team commits. The gray zone is poison.

​

Dependency Management Across Teams

• Asymmetric priority. Your team needs Team B’s API by week 4. Team B has their own roadmap and your request is item 12 on their backlog. No amount of planning on your side changes Team B’s priorities.
• Interface ambiguity. You agreed on an API contract in week 1. By week 6, both teams have slightly different interpretations of what “pagination support” means. Integration week reveals the mismatch.
• Hidden transitive dependencies. Your project depends on Team B, but Team B depends on Team C for a database migration. Team C’s slip cascades through two layers before it reaches you.

1. Map dependencies in week 1, not week 8. For every cross-team dependency, answer: What exactly do we need? When do we need it? Who on the other team owns it? What is their current priority and timeline? If any of these answers are “I do not know,” that dependency is a risk.
2. Negotiate commitment, not just alignment. “Team B is aware of our need” is not the same as “Team B has committed to delivering the API by April 15 and it is on their sprint board.” Awareness is not commitment. Get the dependency into the other team’s planning system with a specific date.
3. Build interfaces, not integrations, early. Agree on the contract (API schema, message format, data model) in week 1 and build a mock that your team can develop against. This decouples your team’s progress from the other team’s delivery timeline. When their real implementation is ready, you swap the mock for the real thing and run integration tests.
4. Escalate early, not late. If a dependency is slipping, raise it in week 3, not week 7. Early escalation sounds like “I want to flag a risk so we can address it proactively.” Late escalation sounds like “another team failed us and now our project is late.” Same information, completely different organizational response.
5. Have a Plan B for every critical dependency. “If Team B’s API is not ready by April 15, we will use a direct database read as a temporary workaround and migrate to the API in V2.” This is not ideal, but it means a dependency slip does not become a project-stopping event.

​

Staffing Constraints and Capacity Planning

1. Wrong allocation (easiest): You have enough engineers but they are working on the wrong things. Solution: reprioritize. This is a leadership problem, not a headcount problem.
2. Skill mismatch: You have enough engineers but not enough with the right skills. Solution: upskilling (3 to 6 months), internal transfers, or targeted hiring for specific roles.
3. Structural underinvestment: You have consistently fewer engineers than the work requires. Solution: make the business case for headcount, but also scope the work to match your actual capacity rather than your wished-for capacity.
4. Organizational overhead (hardest): You have enough engineers but they spend so much time on coordination, context-switching, and meetings that effective engineering hours are half of paid hours. Solution: reduce team surface area, batch similar work, protect focus time.

​

Platform ROI — Justifying Investment in Internal Tools and Infrastructure

​

Execution Under Constraint — Headcount, Time, and Political Pressure

1. Make constraints visible, not implicit. The moment you accept an ambitious plan without acknowledging that you do not have the headcount, the time, or the organizational alignment to execute it, you have signed up for failure. State constraints in writing. “This plan assumes 6 dedicated engineers for 12 weeks with no re-prioritization mid-quarter. If any of those assumptions change, the delivery date changes.”
2. Negotiate scope, not quality and not timelines. When constraints tighten — headcount is cut, the deadline moves up, a new initiative is added mid-quarter — the negotiation must be about what you deliver, not how well you deliver it. The phrase to use: “Given the new constraint, here are three options for what we can deliver. Which does the business value most?” Never silently absorb the constraint by cutting testing, skipping design review, or working unsustainable hours.
3. Track capacity as a first-class metric. Capacity is not “how many engineers we have” — it is “how many engineer-hours of focused work are available after on-call, meetings, cross-team support, and operational toil.” Most teams discover they have 50 to 60 percent effective capacity, not 100 percent. If you plan to 100 percent, every sprint will fail. Track actual capacity over 3 to 4 sprints, use that baseline for future planning, and present it to stakeholders so they understand why “5 engineers for 6 weeks” does not equal “30 engineer-weeks of output.”
4. Document what you are not doing. Every quarter, publish a “not doing” list alongside your roadmap. “These 7 initiatives were proposed and explicitly deferred. Here is the cost of deferral for each. We will re-evaluate at the next planning cycle.” This prevents the conversation from reopening mid-quarter and creates an audit trail that protects the team from “why did you not build X?” revisionism.
5. Manage political pressure with transparency, not heroics. When a politically powerful stakeholder demands something that conflicts with existing commitments, the worst response is to silently try to accommodate both. The staff engineer’s move is to surface the conflict: “We can do X (the VP’s request) if we defer Y (the existing commitment). Here is the impact of that swap. I recommend we make this decision together with both stakeholders in the room.” This is uncomfortable, but it is less painful than the alternative: promising everything, delivering nothing on time, and having both stakeholders blame engineering.

​

35.4 Technical Debt Management

​

Martin Fowler’s Technical Debt Quadrant

• Reckless-Deliberate debt should be challenged in code review and planning. If the team regularly ships with “we don’t have time for design,” that is a process problem, not a time problem.
• Prudent-Deliberate debt is healthy when tracked. Create a ticket immediately, tag it as tech-debt, and schedule it within 1-2 sprints.
• Reckless-Inadvertent debt signals a need for investment in the team — better onboarding, pair programming, architectural guidelines.
• Prudent-Inadvertent debt is how learning works. Refactor when you discover it. No guilt required.

​

Quantifying Technical Debt — Making It a Business Conversation

1. Time tax per sprint. Track how many engineer-days each sprint are spent on workarounds, manual processes, or fighting the debt. Example: “This tech debt costs us 2 engineer-days per sprint in workarounds. That is 10% of our capacity — equivalent to losing one full-time engineer across 5 sprints.”
2. Incident frequency. Count the incidents caused by or worsened by the debt. Example: “This legacy payment system causes 3 incidents per month, each averaging 2 hours to resolve. That is 72 engineer-hours per year of unplanned work, plus the customer impact of 45 minutes average downtime per incident.”
3. Feature velocity drag. Measure how long new features take in the debt-laden area versus a clean area. Example: “Adding a new payment method takes 3 weeks in our current system. In a clean architecture, comparable integrations take 4 days. Every new payment method we add carries a 2.5-week debt tax.”
4. Onboarding cost. Measure how long it takes new engineers to become productive in debt-heavy areas. Example: “New engineers take 6 weeks to make their first meaningful contribution to the billing service, versus 2 weeks for our notification service. That 4-week delta is pure debt cost — and we hired 4 engineers last quarter.”
5. Opportunity cost. Calculate what the team could be building if they were not fighting debt. Example: “We have 3 feature requests from enterprise clients worth an estimated $500K ARR. We cannot start any of them until we address the data model debt in the reporting module, which we have been deferring for 6 months.”

​

35.5 Estimation

• Unknown unknowns. You cannot estimate what you do not know you do not know. That third-party API might have an undocumented rate limit. That “simple” migration might uncover 3 years of data inconsistencies.
• Anchoring bias. Once someone says “this should take about a week,” every subsequent estimate gravitates toward that anchor, regardless of the actual complexity.
• Optimism bias. Engineers estimate for the happy path — everything compiles on the first try, no unexpected bugs, no context switching, no meetings. Reality includes all of those things.
• Complexity is non-linear. Two features that each take 3 days individually might take 10 days together because of integration complexity, shared state, and coordination overhead.

• Three-point estimates: “Best case 3 days, likely case 5 days, worst case 10 days.”
• Confidence levels: “I am 90% confident we can ship in 2 weeks. I am 50% confident we can ship in 1 week.” This tells the PM exactly how much risk they are taking with each timeline.
• Cone of uncertainty: Early in a project, estimates can be off by 4x. After design is complete, 2x. After coding starts, 1.5x. Communicate which phase you are in: “This is a week-1 estimate — expect it to change as we learn more.”
• Flag the unknowns explicitly: “This estimate assumes the payment API supports batch operations. If it does not, add 3-5 days.” This lets stakeholders understand what could change and why.

​

Part XXVIII — Infrastructure and Platform

​

Chapter 36: Infrastructure Basics

​

36.1 Containers and Orchestration

​

Docker Fundamentals

# BAD — copying source code before installing dependencies
# Every code change invalidates the dependency cache
COPY . /app
RUN npm install

# GOOD — install dependencies first, then copy source code
# Dependencies are only reinstalled when package.json changes
COPY package.json package-lock.json /app/
RUN npm install
COPY . /app

# Stage 1: Build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Production
FROM node:20-alpine AS production
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
USER node
EXPOSE 3000
CMD ["node", "dist/main.js"]

​

Kubernetes: The Mental Model

​

Kubernetes Operational Depth — What Signals Real Experience

# Example: A well-configured set of probes
livenessProbe:
  httpGet:
    path: /healthz        # Lightweight — checks app process, NOT dependencies
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3     # 3 consecutive failures before restart
readinessProbe:
  httpGet:
    path: /ready           # Checks that the app CAN serve traffic (DB connected, cache warm)
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 2
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30    # 30 * 10s = 5 minutes to start before liveness takes over
  periodSeconds: 10

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2          # At least 2 pods must stay running during evictions
  selector:
    matchLabels:
      app: api-server

• CPU limits are controversial. Many experienced operators set CPU requests but do NOT set CPU limits. The reasoning: CPU is compressible — if a pod needs more CPU and it is available, throttling it (which is what a limit does) wastes capacity that nobody else is using. Tim Hockin (Kubernetes co-founder) has publicly recommended this approach. Memory limits, however, are essential — memory is not compressible, and an unbounded memory consumer will get OOMKilled by the OS in unpredictable ways.
• Quality of Service (QoS) classes. Kubernetes assigns QoS based on requests and limits: Guaranteed (requests = limits for both CPU and memory), Burstable (requests set but limits are higher or absent), BestEffort (no requests or limits). During node pressure, BestEffort pods are evicted first, then Burstable, then Guaranteed. Set requests and limits on production workloads to avoid being the first pod evicted.
• Vertical Pod Autoscaler (VPA) vs. Horizontal Pod Autoscaler (HPA). HPA adds more pods. VPA adjusts resource requests on existing pods. Use HPA for stateless services that scale horizontally. Use VPA for workloads where adding replicas does not help (single-threaded batch jobs, databases). Do not use both on the same resource simultaneously — they will fight each other.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend     # Only the frontend can talk to the API
      ports:
        - port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database     # The API can only talk to the database
      ports:
        - port: 5432

​

Kubernetes Day 2 Operations — What Happens After the Initial Deployment

1. Read the changelog and deprecation notices. Every Kubernetes release removes or changes APIs. If your manifests use a deprecated API version (e.g., extensions/v1beta1 Ingress was removed in 1.22), your workloads will break. Run kubectl deprecations or use tools like pluto to scan your cluster for deprecated APIs before upgrading.
2. Upgrade the control plane first, then the worker nodes. Kubernetes supports a version skew of N-1 between control plane and kubelets, so you can upgrade the control plane one version ahead and then roll the nodes.
3. Use a rolling node upgrade strategy. Cordon a node (prevent new pods from being scheduled), drain it (gracefully evict existing pods — this is where PDBs matter), upgrade or replace it, then uncordon. Repeat for each node. On managed services (EKS, GKE, AKS), this is partially automated, but you still need to validate workload health at each step.
4. Test in a staging cluster first. Always. No exceptions. Upgrade staging, run your integration tests, let it soak for a few days, then proceed to production. The cost of a staging cluster is trivial compared to a production outage during an upgrade gone wrong.
5. Have a rollback plan. For control plane upgrades on managed services, rollback is often not possible — you move forward. This makes the staging validation step non-negotiable. For self-managed clusters, etcd backups before the upgrade are your safety net.

• Node OS patching. The underlying Linux nodes need regular security updates. Use automated tools like kured (Kubernetes Reboot Daemon) that detect when a node requires a reboot after a kernel update, cordon and drain the node, reboot it, and uncordon it — all respecting your PDBs. The OS Fundamentals chapter covers the kernel-level primitives (namespaces, cgroups) that containers rely on — a kernel vulnerability is a container escape vulnerability.
• Container image scanning. Scan images in your CI pipeline (Trivy, Snyk, Grype) and in the registry on a schedule. New CVEs are discovered daily — an image that was clean last month may have critical vulnerabilities today. Enforce policies: block deployment of images with critical or high-severity CVEs using admission controllers (OPA/Gatekeeper or Kyverno).
• Runtime security monitoring. Tools like Falco detect anomalous behavior inside running containers — unexpected process execution, suspicious network connections, file system modifications in read-only directories. This catches attacks that static scanning cannot.
• RBAC review. Kubernetes RBAC (Role-Based Access Control) should follow the principle of least privilege. Audit who has cluster-admin access — it should be a very short list. Service accounts should have the minimum permissions needed. Review RBAC policies quarterly, especially after team changes.
• Secret rotation. Secrets in Kubernetes should be rotated regularly and ideally managed through an external secret manager (HashiCorp Vault, AWS Secrets Manager) using the External Secrets Operator or similar tools, rather than stored as native Kubernetes Secrets (which are only base64-encoded, not encrypted, by default).

• Scale-up is slow. Provisioning a new node takes 2-5 minutes. If your traffic spikes happen faster than that, you need over-provisioned headroom (extra nodes kept warm) or pod-based solutions like KEDA that scale from zero faster.
• Scale-down is conservative (by design). The Cluster Autoscaler waits 10 minutes by default before removing underutilized nodes, and it will not remove a node if doing so would violate a PDB or if the node has pods with local storage. This prevents flapping but means you pay for idle capacity during gradual ramp-downs.
• HPA and VPA conflict. Do not use HPA and VPA on the same metric for the same workload — they will fight. HPA says “add more pods,” VPA says “give each pod more resources.” Pick one dimension to scale.

1. Right-size your resource requests. Most teams massively over-request resources. A pod requesting 2 CPU cores but averaging 0.3 cores wastes 85% of that allocation. Use tools like Kubecost, Goldilocks (which runs VPA in recommendation mode), or your cloud provider’s cost tools to identify over-provisioned workloads. Reducing resource requests lets you fit more pods per node, which reduces the total number of nodes you need.
2. Use spot/preemptible instances for fault-tolerant workloads. Spot instances are 60-90% cheaper than on-demand. Use them for stateless services that can tolerate interruption (your Deployment has 5+ replicas and PDBs configured). Keep critical workloads (databases, stateful services, single-replica deployments) on on-demand instances. Mix instance types to reduce the probability of simultaneous spot reclamation.
3. Implement namespace-level resource quotas. Without quotas, one team’s runaway deployment can consume the entire cluster’s capacity. ResourceQuotas limit the total CPU, memory, and pod count per namespace. LimitRanges set default requests and limits for pods that do not specify them — preventing the “no resource requests” BestEffort pods that cause unpredictable eviction behavior.
4. Schedule non-urgent workloads during off-peak hours. Batch jobs, ML training, data pipelines — anything that does not need to run at peak traffic time should be scheduled during off-peak hours when cluster utilization is lower. This smooths the demand curve and reduces peak node count.
5. Monitor and allocate costs by team/service. You cannot optimize what you cannot measure. Tag namespaces and workloads with team and service labels. Use Kubecost, CloudHealth, or native cloud cost tools to produce per-team cost reports. When teams see their actual cloud spend, optimization conversations become self-motivated.

​

36.2 Infrastructure as Code

​

IaC Tool Comparison: Terraform vs Pulumi vs CloudFormation

​

36.3 Platform Engineering and Developer Experience

• Cognitive load is the bottleneck. As infrastructure grows more complex (Kubernetes, service meshes, observability stacks, security policies), expecting every application developer to understand all of it is unrealistic. Platform engineering absorbs that complexity into a dedicated team and exposes simple interfaces.
• Consistency at scale. Without a platform, 20 teams will set up CI/CD, monitoring, and deployments in 20 different ways. Debugging becomes harder. Onboarding new engineers takes longer. Security gaps appear in the teams that did not follow best practices.
• Developer velocity. The goal is to reduce the time from “I have an idea” to “it is running in production.” If deploying a new service takes 2 weeks of infrastructure setup, that is 2 weeks of wasted engineering time that a platform team can eliminate.

​

36.4 The Infrastructure Decision Framework: Build vs Buy vs Managed

• The capability is your core competitive advantage. If you are a search company, you build your search engine. If you are a payments company, you own your payment processing logic. You never outsource what makes you unique.
• No existing product meets your requirements, and the gap is fundamental (not just a missing feature that the vendor might add).
• The build cost is justified by long-term savings — you have calculated the total cost of ownership over 3-5 years and in-house wins.
• You have the team to maintain it. Building is the easy part — maintaining, upgrading, patching security vulnerabilities, handling edge cases, and being on-call for it forever is the expensive part.

• The capability is important but not differentiating. Authentication, error tracking, analytics, email delivery — these are critical but they are not why customers choose your product.
• A commercial product covers 80%+ of your needs out of the box. The remaining 20% can be handled through the product’s extension points or workarounds.
• The vendor has deeper expertise than your team in this domain. A company that has spent 10 years building an observability platform will handle edge cases you have never considered.
• Time-to-market matters more than long-term cost optimization.

• You need infrastructure primitives (databases, caches, message queues, object storage) that are well-understood commodity capabilities.
• Your team is small and cannot afford the operational overhead of running the infrastructure themselves.
• The managed service’s pricing model works at your scale. (Watch out: managed services are often cheap at low volume and expensive at high volume. Run the numbers at your projected scale, not just today’s scale.)
• You are willing to accept the cloud provider’s constraints in exchange for not being paged at 3 AM for infrastructure issues.

Is this our core competitive advantage?
├── YES → BUILD (invest in what makes you unique)
└── NO → Does a good commercial product exist?
    ├── YES → Does it meet 80%+ of our needs?
    │   ├── YES → BUY (don't reinvent the wheel)
    │   └── NO → Can a managed cloud service cover it?
    │       ├── YES → MANAGED SERVICE
    │       └── NO → BUILD (but question whether you really need it)
    └── NO → Is there a managed cloud service?
        ├── YES → MANAGED SERVICE
        └── NO → BUILD (you have no choice)

​

Interview Deep-Dive Questions

​

1. You are a senior engineer on a team where product keeps requesting features but nobody is maintaining the systems those features run on. Deployments are slow, incidents are rising, and morale is dropping. What do you do?

​

Follow-up: The product manager agrees to 20 percent, but senior leadership is skeptical. They say “we tried a tech debt sprint last year and nothing improved.” How do you respond?

​

Going Deeper: How do you handle the situation if one or two senior engineers are burned out and have mentally checked out? They are doing the minimum and their code quality is declining.

​

2. Describe your approach to writing an RFC for a major architectural change that you know will be controversial. How do you get buy-in from engineers who disagree with you?

​

Follow-up: You have published the RFC. One highly respected staff engineer writes a scathing critique saying the entire approach is wrong and proposes a fundamentally different direction. The team is now split. How do you handle this?

​

Going Deeper: You have seen teams that produce a lot of design docs and RFCs but execution is slow. What is going wrong and how do you fix it?

​

3. Walk me through how you would decompose a large, ambiguous project into a deliverable plan. Assume the project is “migrate our monolithic application to a service-oriented architecture.”

​

Follow-up: You are in month 4. The first service extraction went well but the second one has hit a wall — two domains share a database with deeply entangled foreign key relationships. What do you do?

​

4. You are on-call and a production incident occurs. Your Kubernetes pods are crash-looping with OOMKilled errors after a routine deployment. Walk me through your incident response from the moment the alert fires.

​

Follow-up: The rollback did not help. The previous version is now also crashing with OOMKilled. What changed?

​

5. You join a new company as a senior engineer. The team has no design review process, no RFCs, and architectural decisions are made in hallway conversations with no documentation. How do you introduce structure without alienating the team?

​

Follow-up: You introduced design docs. Two months later, only you and one other engineer are writing them. The rest of the team skips them and continues making decisions informally. What now?

​

6. Explain Conway’s Law and give me a real example where you have seen organizational structure either help or hinder system architecture.

​

Follow-up: If Conway’s Law is true, does that mean you should reorganize teams before re-architecting systems?

​

7. You are tasked with improving developer experience at your company. The current state: deploying a new service takes 3 weeks of manual setup, there is no standardized monitoring, and every team has a different CI/CD pipeline. Where do you start?

​

Follow-up: A senior engineer on one team says “we do not want your golden path. Our custom setup works fine and your template does not support our specific needs.” How do you handle this?

​

8. How do you think about the trade-off between shipping fast and building things “the right way”? Give me a framework, not a platitude.

​

Follow-up: A junior engineer on your team takes this framework and starts shipping everything fast, justifying it as “high reversibility.” They are producing a lot of sloppy code. How do you course-correct?

​

Going Deeper: How do you handle the situation where the “right” architecture is clear but the team does not have the skills to build it? Do you simplify the architecture or invest in upskilling?

​

9. Tell me about the Terraform state file. Why does it exist, what problems does it cause, and how would you manage it in a team of 30 engineers?

​

Follow-up: An engineer accidentally ran terraform destroy on the production database state. The database is gone. What is your recovery plan?

​

10. Your company is growing from 5 to 50 engineers over the next year. What infrastructure and process investments would you advocate for now, before the growth happens?

​

Follow-up: Leadership pushes back and says “we cannot afford to slow down feature development for infrastructure investments. We are in a growth phase.” How do you respond?

​

Going Deeper: At what point during the growth from 5 to 50 do you introduce a dedicated platform team versus having application engineers handle infrastructure on the side?

​

11. You are evaluating whether to adopt Kubernetes for your company’s infrastructure. What questions do you ask, and what would make you recommend against it?

​

Follow-up: The CTO has already decided on Kubernetes. Your job is to roll it out. How do you structure the migration from your current ECS setup?

​

12. How would you evaluate the health of an engineering organization? You are interviewing for a VP of Engineering role and the CEO asks: “Our engineers say they are overworked but we are shipping slower than last year. What would you investigate?”

​

Follow-up: Your investigation reveals that 35 percent of engineering time is spent on incidents and operational toil, primarily caused by 3 legacy services. The teams owning those services say they have no time to fix them because they are too busy fighting fires. How do you break this cycle?

​

Going Deeper: How do you prevent this situation from recurring? What organizational mechanisms catch these problems before they become crises?

​

Advanced Interview Scenarios

​

13. Your company’s cloud bill jumped 40 percent last month with no corresponding traffic increase. The CFO is demanding answers by Friday. Walk me through your investigation.

​

Follow-up: The investigation reveals that one team’s machine learning training pipeline is responsible for 30 percent of total cloud spend. The ML team says they need those GPU instances. How do you handle this?

​

Follow-up: You reduced the bill, but six months later costs have crept back up. Why does this happen, and how do you make cost discipline stick?

​

14. You shipped a feature three weeks ago. Today, you discover it has a subtle data corruption bug that has been silently writing incorrect values to 2 percent of user records. There are no alerts because the data passes all validation checks — the values are plausible but wrong. How do you handle this?

​

Follow-up: Your remediation plan requires replaying events, but you discover the event stream only retains 7 days of data. The corruption started 21 days ago. How do you recover the older data?

​

15. The obvious “best practice” question: Your team is about to adopt microservices because “that is what companies at our scale do.” You think it is the wrong call. Make the case for staying on the monolith.

​

Follow-up: You convince the team to stay on the monolith, but six months later, the build takes 25 minutes and deploy coordination across 25 engineers is becoming painful. Was your advice wrong?

​

Follow-up: A VP joins from a FAANG company and mandates microservices for everything. How do you push back effectively?

​

16. A production incident happens at 2 AM. You are the on-call engineer and the only person who can fix it, but you have zero monitoring or observability on the affected service. No metrics, no traces, no structured logs — just raw stdout dumped to a file on disk. How do you debug it?

​

Follow-up: After the incident, you want to add monitoring to this legacy service. The team says there is no budget and no time. How do you justify the investment?

​

17. You are leading a database migration from PostgreSQL to DynamoDB because leadership believes it will reduce costs and scale better. Halfway through, you realize DynamoDB’s access patterns do not fit your query requirements and the migration is going to compromise features. What do you do?

​

Follow-up: Leadership says “we already told the board we are migrating to DynamoDB. Changing direction makes us look incompetent.” How do you navigate the political dimension?

​

18. You are running a postmortem for an incident that caused 4 hours of downtime. During the postmortem, it becomes clear that one engineer’s mistake was the direct cause. The engineering VP wants to name the individual in the postmortem document. How do you handle this?

​

Follow-up: After the blameless postmortem, the same class of incident happens two months later. How do you handle a postmortem for a repeat incident without it feeling toothless?

​

19. You discover that a “temporary” workaround deployed 18 months ago has become load-bearing infrastructure. It was never designed for its current role, has no tests, no documentation, and 6 production services depend on it. It works — most of the time. What do you do?

​

Follow-up: How do you prevent “temporary” workarounds from becoming permanent in the first place?

​

20. A competitor just shipped a feature that your CEO considers existential. The CEO wants your team to ship a comparable feature in 4 weeks. Your honest estimate is 12 weeks for a production-quality implementation. You also know that cutting corners on this specific feature creates real safety or data integrity risk. How do you navigate this?

​

Follow-up: The 5-week V1 ships on time, but customers immediately start requesting the V2 capabilities you deferred. The CEO says “see, we should have built the whole thing.” How do you respond?

​

21. You are interviewing a candidate for a senior infrastructure engineer role. They have impressive resume credentials — FAANG experience, authored Kubernetes operators, speaks at conferences. In the technical interview, they design an elegant but massively over-engineered solution to a simple problem. How do you evaluate this candidate?

​

Follow-up: You hire this candidate. Three months in, they are pushing for a complex event-driven architecture when the team is comfortable with synchronous REST APIs. Is this the over-engineering pattern you were worried about, or is this the right technical direction?

​

22. Your team runs a critical service on Kubernetes. A cluster upgrade from 1.27 to 1.28 is scheduled for Saturday. Friday afternoon at 4 PM, your staging cluster upgrade reveals that a Custom Resource Definition your team depends on has a breaking change in 1.28 that was not in the changelog. Production upgrade is in 16 hours. What is your decision process?

​

Follow-up: You chose to postpone, but your manager says “we have already communicated the maintenance window to customers and we cannot change it.” How do you balance this business constraint against the technical risk?

​

Cross-Chapter Connections