Skip to main content
Replication & High Availability Concept

Module 7: Replication & High Availability

Production databases must survive hardware failures, network issues, and data center outages. This module teaches you how to build highly available PostgreSQL deployments. Real-world analogy: Replication is like having a live backup singer who knows every word to every song. If the lead vocalist (primary) loses their voice mid-concert (server crash), the backup (standby) can take over immediately without the audience (users) noticing more than a brief pause. The key engineering challenge is keeping the backup singer perfectly in sync — and deciding whether you are willing to slow down the lead singer to guarantee synchronization.
Estimated Time: 10-12 hours
Hands-On: Set up streaming replication
Key Skill: Designing for 99.99% uptime
PostgreSQL replication and high availability topologies

7.1 High Availability Concepts

The Need for Replication

Replication Types

Replicates WAL bytes - exact binary copy of primary.
Pros:
  • Complete replica (all databases, schemas)
  • Simple setup
  • Low overhead on primary
Cons:
  • Same PostgreSQL major version required
  • Cannot select specific tables
  • Standby is read-only

7.2 Streaming Replication Setup

Primary Server Configuration

Standby Server Setup

Monitoring Replication

Practical tip: Replication lag is not one number — it is four numbers, each measuring a different stage. Understanding the difference between sent_lsn, write_lsn, flush_lsn, and replay_lsn is essential for diagnosing replication problems. If sent is far ahead of write, you have a network bottleneck. If write is ahead of flush, the standby’s disk is slow. If flush is ahead of replay, the standby’s CPU cannot keep up with applying changes.

Failover Process Deep Dive

Database Failover Automatic vs Manual Failover: Automatic (PostgreSQL with Patroni/Stolon):
  • Faster recovery (seconds to minutes)
  • Risk of split-brain if network partition
  • Requires robust failure detection
Manual (Traditional):
  • Slower (minutes to hours)
  • Human verification prevents mistakes
  • Better for planned maintenance
Common pitfall — split-brain: Split-brain occurs when both the primary and the standby believe they are the leader, accepting writes independently. This creates two divergent copies of your data that are extremely difficult to reconcile. It is the single most dangerous failure mode in database replication. Patroni avoids this by using a distributed consensus store (etcd/ZooKeeper) as the source of truth for leadership — a node is only the leader if the consensus store says so. Failover Checklist:
  1. ✓ Verify leader is truly dead (not just slow)
  2. ✓ Choose most up-to-date follower
  3. ✓ Promote follower to leader
  4. ✓ Update DNS/load balancer
  5. ✓ Reconfigure other followers
  6. ✓ Monitor for split-brain
  7. ✓ Investigate root cause
Testing Failover: Use chaos engineering tools like Chaos Monkey to regularly test your failover process in production.

7.3 Synchronous vs Asynchronous

Configuration Options


7.4 Failover and Switchover

Planned Switchover

Automatic Failover with Patroni


7.5 Logical Replication

Publisher Setup

Subscriber Setup

Use Cases for Logical Replication

Cross-Version Migration

Replicate from PG 14 to PG 16, then switch over

Selective Replication

Only sync specific tables to analytics database

Multi-Master

Write to different tables on different servers

Data Consolidation

Combine data from multiple sources

7.6 Connection Pooling

The Problem

PgBouncer Setup

Pool Modes

Common pitfall: Choosing the wrong pool mode is the number one source of PgBouncer problems. In transaction mode, session-level state (SET commands, prepared statements, temp tables, LISTEN/NOTIFY) is lost between transactions because each transaction may land on a different backend connection. If your application uses Django or Rails with prepared statements, you may need session mode — but that significantly reduces pooling efficiency.

7.7 Read Replicas and Load Balancing

Architecture

Application-Level Routing


7.8 Disaster Recovery

Backup Strategies

Pros: Portable, selective, human-readable (with -Fp) Cons: Slow for large databases, point-in-time only
Pros: Fast, consistent, can be used for new replicas Cons: Same PG version only, full backup only
Enables: Recover to any point in time

3-2-1 Backup Rule


7.9 Practice: Set Up Replication

Lab Exercise

  1. Set up two PostgreSQL instances (use Docker)
  2. Configure streaming replication
  3. Test failover
  4. Set up PgBouncer in front

Next Module

Module 8: Scaling Strategies

Scale from thousands to millions of users

Interview Deep-Dive

Strong Answer:
  • 99.99% means max 52 minutes downtime per year, covering planned maintenance and unplanned failures. Architecture: primary + 2 synchronous standbys managed by Patroni with etcd consensus. Use synchronous_standby_names = 'ANY 1 (standby1, standby2)' so the primary needs only one standby ACK to commit. PgBouncer in front absorbs connection storms during failover. HAProxy for read replica load balancing.
  • Zero-downtime maintenance: pg_upgrade via logical replication for major versions, CREATE INDEX CONCURRENTLY for schema changes, never VACUUM FULL during business hours. Budget: 30 min for 2 unplanned failovers/year, 22 min for planned maintenance.
Follow-up: How do you prevent split-brain during failover?Patroni uses etcd as the source of truth for leadership. A node is only the leader if it holds the etcd leader key (requires quorum). If the old primary is partitioned, its lease expires and Patroni demotes it. The new primary can only be promoted after acquiring the leader key. This prevents both nodes from accepting writes simultaneously.
Strong Answer:
  • Cross-version upgrades: Physical requires identical major versions. Logical works across versions (PG 14 to PG 16), enabling zero-downtime upgrades.
  • Selective replication: Physical copies everything. Logical can publish specific tables to specific subscribers.
  • Writable subscribers: Physical standbys are read-only. Logical subscribers can have their own tables, indexes, and materialized views.
  • Tradeoffs: Logical has higher overhead (WAL decoding), does not replicate DDL, and minimal conflict resolution (INSERT conflicts stop replication until manually resolved). For pure HA failover, physical is simpler and faster.
Follow-up: What happens if the subscriber is down for a day?It catches up automatically via replication slots, which prevent the publisher from recycling unconsumed WAL. The danger: a long-disconnected subscriber causes unbounded WAL retention on the publisher. Monitor pg_replication_slots for active = false and set max_slot_wal_keep_size to cap retention.
Strong Answer:
  • Check pg_stat_replication on the primary and compare the four LSN values to identify the bottleneck stage:
  • sent_lsn >> write_lsn: Network bottleneck. Fix: faster link, enable wal_compression (30-50% WAL reduction), or co-locate replica.
  • write_lsn >> flush_lsn: Replica disk cannot keep up with fsync. Fix: faster storage (NVMe).
  • flush_lsn >> replay_lsn: Replica CPU cannot apply WAL fast enough. Common when primary generates WAL from many parallel backends but replay is single-threaded. Fix: reduce indexes on replica, or leverage PG 15+ parallel recovery improvements.
  • Bonus: Long-running queries on replica holding back replay due to high max_standby_streaming_delay. Fix: lower the delay (accept query cancellation) or route analytics to a dedicated replica.
Follow-up: Can you achieve zero lag while allowing reads on the replica?Yes. synchronous_commit = remote_apply ensures the standby has fully replayed WAL before the primary acknowledges the commit. Reads on the standby are always consistent with committed primary data. The cost is increased commit latency on the primary.