Documentation Index
Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
Use this file to discover all available pages before exploring further.
Module 7: Replication & High Availability
Production databases must survive hardware failures, network issues, and data center outages. This module teaches you how to build highly available PostgreSQL deployments. Real-world analogy: Replication is like having a live backup singer who knows every word to every song. If the lead vocalist (primary) loses their voice mid-concert (server crash), the backup (standby) can take over immediately without the audience (users) noticing more than a brief pause. The key engineering challenge is keeping the backup singer perfectly in sync — and deciding whether you are willing to slow down the lead singer to guarantee synchronization.Estimated Time: 10-12 hours
Hands-On: Set up streaming replication
Key Skill: Designing for 99.99% uptime
Hands-On: Set up streaming replication
Key Skill: Designing for 99.99% uptime
7.1 High Availability Concepts
The Need for Replication
Replication Types
- Physical (Streaming)
- Logical
Replicates WAL bytes - exact binary copy of primary.Pros:
- Complete replica (all databases, schemas)
- Simple setup
- Low overhead on primary
- Same PostgreSQL major version required
- Cannot select specific tables
- Standby is read-only
7.2 Streaming Replication Setup
Primary Server Configuration
Standby Server Setup
Monitoring Replication
Practical tip: Replication lag is not one number — it is four numbers, each measuring a different stage. Understanding the difference betweensent_lsn, write_lsn, flush_lsn, and replay_lsn is essential for diagnosing replication problems. If sent is far ahead of write, you have a network bottleneck. If write is ahead of flush, the standby’s disk is slow. If flush is ahead of replay, the standby’s CPU cannot keep up with applying changes.
Failover Process Deep Dive
- Faster recovery (seconds to minutes)
- Risk of split-brain if network partition
- Requires robust failure detection
- Slower (minutes to hours)
- Human verification prevents mistakes
- Better for planned maintenance
- ✓ Verify leader is truly dead (not just slow)
- ✓ Choose most up-to-date follower
- ✓ Promote follower to leader
- ✓ Update DNS/load balancer
- ✓ Reconfigure other followers
- ✓ Monitor for split-brain
- ✓ Investigate root cause
7.3 Synchronous vs Asynchronous
Configuration Options
7.4 Failover and Switchover
Planned Switchover
Automatic Failover with Patroni
7.5 Logical Replication
Publisher Setup
Subscriber Setup
Use Cases for Logical Replication
Cross-Version Migration
Replicate from PG 14 to PG 16, then switch over
Selective Replication
Only sync specific tables to analytics database
Multi-Master
Write to different tables on different servers
Data Consolidation
Combine data from multiple sources
7.6 Connection Pooling
The Problem
PgBouncer Setup
Pool Modes
Common pitfall: Choosing the wrong pool mode is the number one source of PgBouncer problems. Intransaction mode, session-level state (SET commands, prepared statements, temp tables, LISTEN/NOTIFY) is lost between transactions because each transaction may land on a different backend connection. If your application uses Django or Rails with prepared statements, you may need session mode — but that significantly reduces pooling efficiency.
| Mode | Description | Best For | Gotcha |
|---|---|---|---|
| session | Connection held for entire session | Session state needed (SET, temp tables) | Minimal pooling benefit — one backend per client |
| transaction | Connection held for transaction | Most applications (stateless queries) | Breaks prepared statements, SET, LISTEN, temp tables |
| statement | Connection released after each query | Simple queries, max sharing | Completely breaks multi-statement transactions |
7.7 Read Replicas and Load Balancing
Architecture
Application-Level Routing
7.8 Disaster Recovery
Backup Strategies
pg_dump (Logical Backup)
pg_dump (Logical Backup)
pg_basebackup (Physical Backup)
pg_basebackup (Physical Backup)
Continuous Archiving (PITR)
Continuous Archiving (PITR)
3-2-1 Backup Rule
7.9 Practice: Set Up Replication
Lab Exercise
- Set up two PostgreSQL instances (use Docker)
- Configure streaming replication
- Test failover
- Set up PgBouncer in front
Docker Compose Setup
Docker Compose Setup
Next Module
Module 8: Scaling Strategies
Scale from thousands to millions of users
Interview Deep-Dive
You are designing a database architecture with 99.99% uptime. Walk through your approach with PostgreSQL.
You are designing a database architecture with 99.99% uptime. Walk through your approach with PostgreSQL.
Strong Answer:
- 99.99% means max 52 minutes downtime per year, covering planned maintenance and unplanned failures. Architecture: primary + 2 synchronous standbys managed by Patroni with etcd consensus. Use
synchronous_standby_names = 'ANY 1 (standby1, standby2)'so the primary needs only one standby ACK to commit. PgBouncer in front absorbs connection storms during failover. HAProxy for read replica load balancing. - Zero-downtime maintenance: pg_upgrade via logical replication for major versions,
CREATE INDEX CONCURRENTLYfor schema changes, never VACUUM FULL during business hours. Budget: 30 min for 2 unplanned failovers/year, 22 min for planned maintenance.
When would you choose logical replication over physical streaming replication?
When would you choose logical replication over physical streaming replication?
Strong Answer:
- Cross-version upgrades: Physical requires identical major versions. Logical works across versions (PG 14 to PG 16), enabling zero-downtime upgrades.
- Selective replication: Physical copies everything. Logical can publish specific tables to specific subscribers.
- Writable subscribers: Physical standbys are read-only. Logical subscribers can have their own tables, indexes, and materialized views.
- Tradeoffs: Logical has higher overhead (WAL decoding), does not replicate DDL, and minimal conflict resolution (INSERT conflicts stop replication until manually resolved). For pure HA failover, physical is simpler and faster.
pg_replication_slots for active = false and set max_slot_wal_keep_size to cap retention.Your read replica shows 30 seconds of replication lag during peak hours. Diagnose and fix it.
Your read replica shows 30 seconds of replication lag during peak hours. Diagnose and fix it.
Strong Answer:
- Check
pg_stat_replicationon the primary and compare the four LSN values to identify the bottleneck stage: - sent_lsn >> write_lsn: Network bottleneck. Fix: faster link, enable wal_compression (30-50% WAL reduction), or co-locate replica.
- write_lsn >> flush_lsn: Replica disk cannot keep up with fsync. Fix: faster storage (NVMe).
- flush_lsn >> replay_lsn: Replica CPU cannot apply WAL fast enough. Common when primary generates WAL from many parallel backends but replay is single-threaded. Fix: reduce indexes on replica, or leverage PG 15+ parallel recovery improvements.
- Bonus: Long-running queries on replica holding back replay due to high
max_standby_streaming_delay. Fix: lower the delay (accept query cancellation) or route analytics to a dedicated replica.
synchronous_commit = remote_apply ensures the standby has fully replayed WAL before the primary acknowledges the commit. Reads on the standby are always consistent with committed primary data. The cost is increased commit latency on the primary.