Learn distributed systems: books in order
This curriculum takes an intermediate learner from the core principles of distributed systems through the rigorous theory of consensus, replication, and consistency, and finally into the advanced engineering and research-level thinking required to build and reason about real-world fault-tolerant systems. Each stage builds on the vocabulary and mental models of the previous one, so the order within and across stages is deliberate.
Foundations & Mental Models
IntermediateEstablish a solid conceptual vocabulary — time, clocks, failure models, CAP, and the fundamental trade-offs — so that every later topic has a firm foundation to attach to.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of dense technical content and re-reading for comprehension)
- Logical and physical time: clocks, ordering, causality, and why wall-clock time is unreliable in distributed systems
- Failure models: crash failures, Byzantine failures, partial failures, and their implications for system design
- Consistency models: strong vs. eventual consistency, linearizability, causal consistency, and when each matters
- The CAP theorem: the fundamental trade-off between consistency, availability, and partition tolerance
- Replication and consensus: why they are hard, leader-based vs. leaderless approaches, and quorum-based decisions
- Network partitions and asynchrony: why you cannot distinguish a slow node from a dead one, and how this breaks assumptions
- Distributed algorithms foundations: message passing, state machines, and the role of determinism in correctness
- Trade-offs and system design: how theoretical constraints translate into practical architectural choices
- Why is it impossible to use wall-clock time alone to determine the order of events in a distributed system, and what alternatives exist?
- What is the difference between a crash failure and a Byzantine failure, and why does this distinction matter for system design?
- State the CAP theorem and explain why a system cannot simultaneously guarantee all three properties under network partition.
- What is linearizability, and how does it differ from eventual consistency? When would you choose one over the other?
- How do quorum-based replication and consensus algorithms (like Raft or Paxos) help solve the consistency problem?
- Explain the fundamental challenge of distinguishing a slow node from a failed node in an asynchronous network, and its consequences.
- Draw a timeline diagram of three distributed nodes with events and causality arrows; identify which orderings are possible without a global clock.
- Write pseudocode for a simple logical clock (Lamport or vector clock) and trace through a scenario with 3 nodes exchanging messages.
- Design a toy replication scheme for a key-value store: specify your failure model assumptions, consistency guarantee, and what happens during a network partition.
- Analyze a real-world system (e.g., Cassandra, DynamoDB, or Redis) and map its design choices to the CAP theorem—which property does it prioritize?
- Implement a simple state machine replication simulator: one leader, multiple followers, message loss, and node crashes; observe consistency breakdowns.
- Create a comparison table: for 4–5 consistency models (strong, eventual, causal, linearizable), list when each is appropriate and what it costs.
Next up: This stage equips you with the vocabulary and mental models needed to evaluate trade-offs in later topics—replication strategies, consensus algorithms, and distributed transactions all rest on your intuition for time, failure, and consistency, which you will now apply to concrete protocol design.
The single best bridge from practitioner to distributed-systems thinker: covers replication, partitioning, consistency, and transactions with clarity and depth. Read this first to build the vocabulary every subsequent book assumes.
Provides the classical academic framing — naming, synchronization, consistency models, fault tolerance — that gives structure to the intuitions built in Kleppmann. Fills in the 'why' behind design choices.
Consensus, Replication & Fault Tolerance
IntermediateDeeply understand how distributed systems agree on values, replicate state safely, and survive failures — the core algorithmic heart of the field.
▸ Study plan for this stage
Pace: 4–5 weeks, ~25–30 pages/day (focusing on Chapters 6–10 covering replication, state management, and resilience patterns)
- Single-leader replication: how one node coordinates writes and propagates state to followers, including handling leader failure and catch-up
- Multi-leader and leaderless replication: trade-offs between consistency, availability, and partition tolerance across different topologies
- Quorum-based consensus: how reading/writing to a majority ensures safety and liveness in the presence of failures
- State machines and deterministic execution: ensuring replicas converge to the same state through identical operation sequences
- Handling network partitions and split-brain scenarios: practical strategies to prevent conflicting decisions when nodes lose contact
- Write-ahead logging (WAL) and durability: persisting operations before applying them to guarantee no data loss on failure
- Eventual consistency vs. strong consistency: understanding the CAP theorem trade-offs and when each model is appropriate
- Failure detection and recovery: heartbeats, timeouts, and mechanisms for detecting and replacing failed nodes
- Explain the difference between single-leader and multi-leader replication. What are the consistency and availability trade-offs of each approach?
- How does quorum-based consensus ensure that a distributed system can tolerate f failures with 2f+1 nodes? Walk through a concrete example.
- What is a split-brain scenario, and how can write-ahead logging and quorum reads/writes prevent it?
- Describe the role of state machines in replication. Why must operations be deterministic and applied in the same order on all replicas?
- How does eventual consistency differ from strong consistency? Give examples of systems that use each and explain the trade-offs.
- What mechanisms does Burns describe for detecting node failures and triggering failover? How do timeouts and heartbeats work together?
- Implement a simple single-leader replication system: create a primary node that accepts writes, logs them, and sends updates to two follower nodes; simulate a follower failure and verify catch-up logic
- Build a quorum-based key-value store with 5 nodes: implement read and write operations that require quorum agreement (3/5); test that it tolerates 2 simultaneous node failures
- Simulate a network partition: create two groups of nodes and demonstrate split-brain behavior without quorum enforcement; then add quorum logic and show how it prevents conflicting writes
- Implement write-ahead logging: add a persistent log to a replicated state machine; crash and restart a node, verifying that all logged operations are replayed correctly
- Design a failure detection system: implement heartbeat-based detection with configurable timeouts; test false positives (flaky network) and true failures, measuring detection latency
- Compare consistency models: build two versions of a replicated counter (one eventual, one strong); write concurrent clients and observe the difference in behavior under network delays
Next up: This stage establishes the algorithmic foundations of agreement and resilience; the next stage will build on these primitives to explore distributed transactions, coordination services, and higher-level abstractions that depend on reliable consensus and replication.
Introduces reusable patterns (sidecars, ambassadors, scatter/gather) for building reliable distributed systems; a practical on-ramp before diving into algorithm-heavy material.
Consistency, Transactions & Time
ExpertMaster the precise definitions of consistency models (linearizability, serializability, eventual consistency) and understand how real systems implement transactions across distributed nodes.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (with 2–3 days per week for deeper dives on consistency models and transaction protocols)
- Linearizability vs. serializability: formal definitions, when each applies, and why they differ in distributed contexts
- ACID properties in distributed systems: how atomicity, consistency, isolation, and durability break down across nodes and must be rebuilt
- Isolation levels (READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SERIALIZABLE) and their real-world trade-offs in Database Internals
- Two-Phase Commit (2PC) protocol: coordination, blocking behavior, and failure modes in Transactional Information Systems
- Concurrency control mechanisms: pessimistic locking, optimistic concurrency control, and MVCC as implemented in real databases
- Transaction logs, write-ahead logging (WAL), and recovery protocols for durability guarantees
- Eventual consistency and weak consistency models: when and why systems relax strong consistency
- Conflict detection and resolution in distributed transactions: timestamp ordering, conflict graphs, and abort strategies
- Define linearizability and serializability formally. Why is a schedule linearizable but not serializable, or vice versa? Provide concrete examples.
- Explain the ACID properties in a distributed system context. How does each property degrade when nodes are separated by a network, and what mechanisms restore them?
- Describe the Two-Phase Commit protocol step-by-step. What are its failure modes, and why is it blocking? When would a system choose 2PC over eventual consistency?
- Compare pessimistic locking, optimistic concurrency control, and MVCC. For each, describe the workload characteristics where it performs best and worst.
- What is write-ahead logging, and how does it enable durability and recovery? Sketch the recovery process after a crash.
- Explain the relationship between isolation levels and consistency models. Can READ COMMITTED isolation guarantee serializability? Why or why not?
- Describe eventual consistency. What application scenarios tolerate it, and what invariants must developers maintain when using it?
- Work through the 2PC protocol on paper: draw the state machine for coordinator and participants, then trace through a failure scenario (e.g., coordinator crashes after PREPARE phase). Document what happens to in-flight transactions.
- Implement a simple in-memory key-value store with optimistic concurrency control: use version numbers or timestamps, detect conflicts on write, and abort conflicting transactions. Test with concurrent writers.
- Trace through a write-ahead logging (WAL) scenario: write a sequence of transactions to a log, simulate a crash mid-transaction, then implement and test the recovery algorithm.
- Analyze isolation levels experimentally: set up a multi-threaded test harness that executes concurrent transactions at different isolation levels (READ COMMITTED, REPEATABLE READ, SERIALIZABLE) and verify which anomalies (dirty reads, non-repeatable reads, phantom reads) occur.
- Build a conflict graph detector for distributed transactions: given a history of reads and writes across multiple nodes, construct the conflict graph and determine if the schedule is conflict-serializable.
- Compare consistency models on a toy distributed system: implement a simple replicated key-value store with strong consistency (via 2PC), eventual consistency (async replication), and causal consistency. Measure latency and throughput trade-offs.
Next up: This stage equips you with the theoretical foundations and practical understanding of how systems enforce correctness under concurrency and failure; the next stage will explore how to design and reason about distributed algorithms that operate under these consistency and transaction constraints.
Bridges storage engine internals with distributed algorithms — covers B-trees, LSM trees, and then distributed consensus and anti-entropy protocols, grounding abstract consistency models in real implementation.
The definitive deep reference on transaction theory, isolation levels, and concurrency control; read after Petrov to formalize the consistency and serializability concepts encountered throughout the curriculum.
Research Foundations & Advanced Theory
ExpertEngage with the landmark papers and theoretical results — FLP impossibility, Paxos, Raft, Dynamo, Spanner — and develop the ability to read and evaluate primary distributed systems research independently.
▸ Study plan for this stage
Pace: 12–14 weeks, ~40–50 pages/day (Lynch: weeks 1–7; Cachin: weeks 8–14). Allocate 1–2 weeks per major section for deep engagement with proofs and algorithms.
- Computational model and timing assumptions (synchronous, asynchronous, partial synchrony) and their impact on solvability
- FLP impossibility theorem: why consensus is unsolvable in asynchronous systems with one crash failure
- State machines, replication, and the role of consensus in building fault-tolerant systems
- Paxos algorithm: leader election, proposer–acceptor–learner roles, and liveness under partial synchrony
- Byzantine fault tolerance and Byzantine Generals Problem: algorithms for systems with malicious failures
- Reliable broadcast, atomic commit, and composing primitives to build higher-level abstractions
- Safety vs. liveness properties and formal verification of distributed algorithms
- Practical consensus protocols (Raft, Paxos variants) and their trade-offs in real systems
- Why is consensus impossible to solve in a fully asynchronous system with even one crash failure, and what does the FLP impossibility theorem tell us about the limits of distributed computing?
- How do different timing models (synchronous, asynchronous, partial synchrony) affect which problems are solvable, and what assumptions does Paxos rely on?
- Explain the roles of proposers, acceptors, and learners in Paxos, and how the algorithm guarantees both safety and liveness.
- What is the difference between crash failures and Byzantine failures, and why do Byzantine-fault-tolerant algorithms require stronger assumptions?
- How can you compose basic primitives like reliable broadcast and consensus to build higher-level abstractions like atomic commit?
- What are safety and liveness properties, and how do you formally verify that a distributed algorithm satisfies them?
- Work through Lynch's proofs of the FLP impossibility theorem step-by-step; write a 1–2 page summary explaining why asynchrony + crashes make consensus hard.
- Implement a simplified Paxos proposer and acceptor in pseudocode or a real language; trace through a scenario with message delays and failures.
- Prove or disprove: 'In a synchronous system, consensus is always solvable.' Justify your answer using Lynch's model.
- Read and annotate Cachin's chapters on Byzantine fault tolerance; create a comparison table of Byzantine vs. crash-fault-tolerant algorithms.
- Design a reliable broadcast protocol from scratch using only point-to-point links; verify it satisfies the required safety and liveness properties.
- Implement a state machine replication system using Paxos or a simpler consensus protocol; test it with simulated failures (dropped messages, delayed messages, crashed replicas).
Next up: This stage equips you with the theoretical foundations and formal reasoning skills to understand why landmark systems (Dynamo, Spanner, Raft) make the design choices they do, preparing you to study production systems and their trade-offs in the next stage.
The canonical theoretical reference for distributed algorithms: covers synchrony models, impossibility results (including FLP), consensus, and Byzantine fault tolerance with full mathematical rigor. The capstone of the curriculum.
Complements Lynch by presenting modern abstractions — broadcast primitives, shared memory, consensus — in a clean algorithmic style that directly maps theory to implementable protocols, including Byzantine-tolerant ones.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
