Learn Apache Kafka: the best books in order
This curriculum takes an intermediate developer from Kafka's core concepts through production-grade event streaming and stream processing. Each stage builds directly on the last — you'll establish solid Kafka fundamentals first, then master the Streams API and real-time pipelines, and finally tackle advanced architecture and operations at scale.
Kafka Foundations
IntermediateUnderstand Kafka's architecture — brokers, topics, partitions, producers, consumers, and offsets — and be able to run and interact with a Kafka cluster confidently.
▸ Study plan for this stage
Pace: 4–5 weeks, ~25–30 pages/day, with 2–3 days per week reserved for hands-on lab work
- Kafka's publish-subscribe architecture and how it differs from traditional message queues
- Brokers, topics, and partitions as the core organizational units of Kafka clusters
- Producer API: how messages are serialized, partitioned, and sent to brokers with acknowledgment strategies
- Consumer API: consumer groups, offset management, and rebalancing mechanisms
- Offset tracking and log-based storage: how Kafka maintains message ordering and durability per partition
- Replication and fault tolerance: leader/follower model and in-sync replicas (ISRs)
- Cluster configuration and operational basics: broker settings, topic creation, and monitoring
- Explain how Kafka's partitioned topic model enables both scalability and message ordering guarantees.
- What is the role of consumer groups, and how does Kafka handle rebalancing when a consumer joins or leaves?
- Describe the difference between acks=0, acks=1, and acks=all in producer configuration and their trade-offs.
- How does Kafka use offsets to allow consumers to replay messages and maintain their position in a partition?
- What is the purpose of in-sync replicas (ISRs) and how do they relate to data durability?
- Walk through the steps of setting up a multi-broker Kafka cluster and creating a replicated topic.
- Install and run a local Kafka cluster (3 brokers) using Docker or native binaries; verify broker health via JMX metrics.
- Create a multi-partition, replicated topic; use kafka-topics.sh to inspect partition leadership and replica distribution.
- Write a simple producer in your language of choice that sends 1000 messages to a topic with different partitioning strategies; measure latency under acks=0, acks=1, and acks=all.
- Write a consumer application that subscribes to a topic, processes messages, and manually commits offsets at different intervals; observe rebalancing behavior when adding/removing consumers.
- Simulate a broker failure by stopping one broker in your cluster; observe which partitions lose their leader and how the cluster recovers.
- Build a multi-consumer group scenario where two independent applications consume the same topic; verify that each group maintains separate offset positions.
Next up: This stage establishes the mental model of Kafka's distributed architecture and hands-on fluency with core APIs, preparing you to explore advanced topics like exactly-once semantics, stream processing frameworks, and production deployment strategies in subsequent stages.
Written by Kafka's co-creator, this is the canonical starting point. It covers the full mental model — topics, producers, consumers, replication, and reliability — giving you the vocabulary every subsequent book assumes.
Event-Driven Design & Messaging Patterns
IntermediateLearn how to think in events — designing event-driven systems, choosing the right messaging patterns, and understanding where Kafka fits in the broader landscape of distributed architectures.
▸ Study plan for this stage
Pace: 4–5 weeks, ~40–50 pages/day (focusing on pattern chapters; skim architectural overviews)
- Message-oriented middleware (MOM) as the foundation for asynchronous, decoupled communication
- Core messaging patterns: Point-to-Point, Publish-Subscribe, and their trade-offs
- Message routing patterns (Content-Based Router, Message Filter, Splitter, Aggregator) for intelligent message flow
- Message transformation patterns (Message Translator, Envelope Wrapper) to handle format mismatches across systems
- Idempotence and exactly-once semantics: why they matter and how to design for them
- Dead Letter Channels and error handling strategies in asynchronous systems
- Event-driven thinking: designing systems where components react to state changes rather than direct calls
- What is the fundamental difference between Point-to-Point and Publish-Subscribe messaging, and when would you choose each for a given business problem?
- How do Content-Based Routers and Message Filters differ, and why might you use both in the same system?
- Why is idempotence critical in message-driven systems, and what are two strategies to achieve it?
- What is a Dead Letter Channel, and how does it help you build resilient asynchronous systems?
- How would you use Message Translator and Envelope Wrapper patterns to integrate two systems with incompatible message formats?
- Describe an event-driven architecture you could build using Publish-Subscribe: what events would flow, and how would subscribers react?
- Map a real business process (e.g., order fulfillment, user registration) to messaging patterns: identify which channels would be Point-to-Point vs. Publish-Subscribe, and justify each choice.
- Design a message schema for a domain event (e.g., 'OrderPlaced' or 'PaymentProcessed') including headers, payload, and metadata; then design a Message Translator to convert it to a legacy system's format.
- Implement a simple Content-Based Router in pseudocode or a lightweight framework (e.g., Spring Integration, Apache Camel): route incoming orders to different fulfillment channels based on order type.
- Sketch an error-handling flow using Dead Letter Channels: define what triggers a message to be sent to the DLQ, how it's monitored, and how it's replayed.
- Build a small Publish-Subscribe prototype (using a message broker like RabbitMQ or a local event bus): publish domain events and wire up multiple subscribers that react independently.
- Analyze a monolithic application and propose how to decompose it into event-driven microservices using patterns from the book; document the events, channels, and subscribers.
Next up: Understanding these foundational messaging patterns and event-driven thinking prepares you to see Kafka not as a generic message broker, but as a specific implementation choice optimized for high-throughput, event-log-based architectures—and to recognize which patterns it excels at and which it doesn't.
The foundational reference for messaging patterns (queues, topics, routers, filters) that underpin all event streaming work; reading it here ensures you recognize these patterns when they appear in Streams and pipeline design.
Kafka Streams & Stream Processing
IntermediateMaster the Kafka Streams API — stateless and stateful transformations, windowing, joins, and interactive queries — and understand the stream-table duality at the heart of real-time processing.
▸ Study plan for this stage
Pace: 6–8 weeks, ~40–50 pages/day (mix of reading and hands-on coding)
- Kafka Streams topology: sources, processors, sinks, and the DAG model of stream processing
- Stateless transformations (map, filter, flatMap, branch) and when to use each
- Stateful operations: aggregations, reductions, and local state stores with changelog topics
- Stream-table duality: how streams and tables are dual representations of the same data, and KStream vs. KTable vs. GlobalKTable
- Windowing strategies (tumbling, hopping, session, grace period) and their trade-offs
- Stream joins (KStream-KStream, KStream-KTable, KTable-KTable) and co-partitioning requirements
- Interactive queries: querying local state stores in real-time without leaving the topology
- KsqlDB as a declarative SQL layer over Kafka Streams for rapid prototyping and operational queries
- What is the difference between a KStream, KTable, and GlobalKTable, and when would you use each in a topology?
- How does the stream-table duality explain why an aggregation produces a changelog stream, and what role does the state store play?
- Design a topology for a real-time e-commerce scenario: ingest orders (KStream), join with customer profiles (KTable), aggregate revenue by region in 1-minute windows, and explain co-partitioning requirements.
- What are the trade-offs between tumbling, hopping, and session windows, and how does grace period affect late-arriving data?
- How would you implement an interactive query to fetch the current balance of a customer from a state store without stopping the topology?
- Compare KStream-KTable joins vs. KTable-KTable joins: what are the semantics, and when would you choose one over the other?
- Build a stateless word-count topology using map and filter; run it against a Kafka topic and verify output.
- Implement a stateful aggregation (e.g., sum of transaction amounts per user) with a local state store; query the store interactively and observe changelog topic updates.
- Create a topology that joins a KStream of orders with a KTable of product prices; verify co-partitioning and test with out-of-order joins.
- Design and implement a session-windowed aggregation (e.g., user sessions with 5-minute inactivity gap); test with late-arriving events and observe grace period behavior.
- Build an interactive query application that exposes a REST endpoint to query a state store; load test it with concurrent queries.
- Replicate a real-time dashboard scenario using KsqlDB: create a stream from a Kafka topic, define a table with aggregations, and run continuous queries.
Next up: This stage equips you with the operational and API-level mastery of Kafka Streams; the next stage will deepen your understanding of production concerns—error handling, exactly-once semantics, scaling, monitoring, and architectural patterns—to move from working topologies to resilient, observable systems.
The most thorough hands-on guide to the Kafka Streams API; it walks through the DSL and Processor API with real examples, making it the natural next step after understanding Kafka's core.
Extends stream processing into ksqlDB, covering stateful operations, event-time semantics, and SQL-based streaming — deepening your toolkit and preparing you for real-time pipeline design.
Real-Time Pipelines & the Streaming Ecosystem
ExpertBuild end-to-end real-time data pipelines, integrate Kafka with the broader ecosystem (Schema Registry, Kafka Connect, Flink), and apply stream processing theory rigorously.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of dense theory and worked examples; allow extra time for re-reading complex sections on windowing, state management, and distributed systems concepts)
- Unbounded data: event time vs. processing time vs. ingestion time, and why the distinction matters for correctness
- Windowing strategies (tumbling, sliding, session windows) and their role in aggregating infinite streams
- Stateful stream processing: managing state, fault tolerance, and exactly-once semantics
- Watermarks and triggers: controlling when results are emitted and how late data is handled
- Distributed systems foundations: consistency models (strong vs. eventual), consensus, and replication
- Kafka as a durable event log: partitioning, replication, and offset management in the context of streaming
- Stream processing frameworks (Flink, Kafka Streams) as implementations of streaming system theory
- End-to-end pipeline design: schema evolution, exactly-once delivery, and monitoring in production
- What is the difference between event time, processing time, and ingestion time, and why does each matter for a real-time pipeline?
- How do windowing strategies (tumbling, sliding, session) differ, and when would you choose each for a specific use case?
- Explain the role of watermarks and triggers in stream processing. How do they control correctness and latency trade-offs?
- What does 'exactly-once semantics' mean, and what mechanisms (idempotence, deduplication, distributed snapshots) enable it in Kafka-based pipelines?
- How do consistency models (strong vs. eventual) and consensus algorithms (Raft, Paxos) underpin distributed streaming systems?
- Design an end-to-end pipeline: given a business requirement (e.g., real-time fraud detection), describe the role of Kafka, schema registry, stream processor, and state store.
- Read and annotate Chapters 1–3 of 'Streaming Systems' (What, Where, When, How framework); create a one-page summary of the four questions and how they relate to windowing and triggers.
- Implement a tumbling-window aggregation in Kafka Streams (or Flink): ingest events with timestamps, compute sum/count per 5-minute window, and verify results with late-arriving data.
- Read Chapter 9 ('Consistency and Consensus') in 'Designing Data-Intensive Applications'; map consensus algorithms (Raft, Paxos) to how Kafka brokers replicate and achieve durability.
- Build a stateful stream processor that tracks user sessions: use a session window, emit session-end events, and test fault tolerance by simulating a crash and restart.
- Design a schema evolution scenario: start with a simple Avro schema, evolve it (add a field), and verify that both old and new consumers can read the data via Schema Registry.
- Implement an end-to-end pipeline: Kafka source → Kafka Streams processor (with state store) → Kafka sink; add monitoring (lag, throughput) and test exactly-once semantics under failure.
Next up: This stage establishes the theoretical and practical foundations of streaming systems and distributed data architecture; the next stage will apply these principles to specialized domains (e.g., real-time ML, event sourcing, or operational analytics) and production hardening (scaling, multi-cluster, disaster recovery).
The definitive theoretical treatment of streaming — watermarks, triggers, windowing, and exactly-once semantics — giving you the rigorous mental model needed to design correct, production-grade pipelines.
Ties everything together: replication, partitioning, stream vs. batch, and the role of logs in distributed systems. Reading it here lets you place Kafka and your pipelines in the full landscape of data engineering.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
