Data engineering: books for building reliable data pipelines
This curriculum takes an intermediate learner from solid data engineering fundamentals through to the distributed systems and streaming architectures that power modern data platforms. Each stage builds directly on the last: you first internalize the core patterns and tools, then master pipeline orchestration and warehouse design, then tackle real-time streaming, and finally understand the distributed systems principles that underpin everything at scale.
Core Foundations of Data Engineering
IntermediateEstablish a shared vocabulary and mental model for the full data engineering lifecycle — ingestion, storage, transformation, and serving — and understand how modern data stacks are architected.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of reading and note-taking). Week 1–5: "Fundamentals of Data Engineering" (~400 pages); Week 6–10: "The Data Warehouse Toolkit" (~500 pages). Allocate 1–2 days per week for review and exercises.
- The data engineering lifecycle: ingestion, storage, transformation, and serving as interconnected stages with feedback loops
- Data sources, ingestion patterns (batch vs. streaming), and the tradeoffs between latency, throughput, and complexity
- Storage architectures: data lakes, data warehouses, and lakehouses; schema-on-read vs. schema-on-write
- Dimensional modeling and the star schema as a foundation for organizing data for analytics and reporting
- Slowly Changing Dimensions (SCDs) and how to handle historical data in warehouse designs
- ETL vs. ELT paradigms and when to apply each in a modern data stack
- Data quality, governance, and metadata as cross-cutting concerns throughout the lifecycle
- Serving layer patterns: how data is exposed to end users, BI tools, and downstream systems
- Describe the four stages of the data engineering lifecycle and explain how they interact with feedback loops
- What are the key differences between a data lake, data warehouse, and lakehouse architecture, and when would you choose each?
- Explain the star schema and why dimensional modeling is effective for analytical queries and business intelligence
- What are Slowly Changing Dimensions (SCDs), and how would you implement Type 1, Type 2, and Type 3 SCD patterns?
- Compare ETL and ELT approaches: what are the tradeoffs, and how does your choice affect data quality and transformation logic?
- How do data quality, governance, and metadata management support the entire data engineering lifecycle?
- Map out the data engineering lifecycle for a real or hypothetical company: identify data sources, ingestion methods, storage choices, transformations, and serving patterns
- Design a star schema for a sample business domain (e.g., e-commerce, SaaS, retail) with fact and dimension tables; document grain, measures, and attributes
- Implement a Slowly Changing Dimension: write pseudocode or SQL for a Type 2 SCD that tracks historical changes to a customer dimension
- Build a small ETL pipeline (using Python, SQL, or a tool like dbt) that ingests sample data, applies transformations, and loads it into a dimensional model
- Compare two data stack architectures (e.g., traditional data warehouse vs. modern lakehouse): create a decision matrix weighing cost, latency, flexibility, and governance
- Document a data quality framework for a hypothetical dataset: define metrics, validation rules, and how you'd monitor them across the pipeline
Next up: This stage establishes the mental models and vocabulary needed to evaluate, design, and build specific data systems; the next stage will dive into hands-on implementation of these architectures using modern tools and cloud platforms.
The single most comprehensive map of the data engineering landscape today. Read this first to anchor every subsequent book to a coherent framework of the data lifecycle.
Kimball's dimensional modeling patterns are the lingua franca of analytical storage. Reading it second gives you the warehouse design vocabulary you'll need before going deeper into modern cloud warehouses and pipelines.
Pipelines, Orchestration & Transformation
IntermediateDesign, build, and orchestrate reliable batch data pipelines; understand workflow scheduling, dependency management, and the transformation layer (ELT/ETL) in depth.
▸ Study plan for this stage
Pace: 10–12 weeks, ~40–50 pages/day (mix of reading and hands-on work). Week 1–3: Data Pipelines Pocket Reference (foundational concepts); Week 4–7: Data Pipelines with Apache Airflow (deep dive into orchestration); Week 8–12: Analytics Engineering with SQL and DBT (transformation layer and production wor
- Pipeline architecture: sources, transformations, sinks, and the flow of data through each stage
- Batch vs. streaming trade-offs and when to use each paradigm
- Workflow orchestration: DAGs, task dependencies, scheduling, and failure handling in Airflow
- ELT vs. ETL: the shift toward loading raw data first and transforming in the warehouse
- Data quality and validation: ensuring correctness, completeness, and consistency at each pipeline stage
- Idempotency and replayability: designing pipelines that can be safely re-run without side effects
- dbt fundamentals: models, tests, documentation, and the analytics engineering workflow
- Incremental loading and state management: handling large datasets efficiently with minimal re-processing
- What are the core components of a data pipeline, and how do they differ from a traditional ETL system?
- How does Apache Airflow use DAGs to manage task dependencies, and what are the benefits of this approach over cron jobs?
- What is the ELT paradigm, and why has it become preferred over traditional ETL in modern data warehouses?
- How do you design an idempotent pipeline, and why is idempotency critical for reliable batch processing?
- What role does dbt play in the transformation layer, and how does it integrate with your data warehouse?
- How do you implement data quality checks and tests in a production pipeline, and what should you validate?
- Build a simple batch pipeline using Apache Airflow with 3–5 tasks (extract, validate, transform, load) and define explicit dependencies between them.
- Design and implement an idempotent transformation: write SQL or dbt code that can be safely re-run on the same data without duplicates or errors.
- Create a dbt project with at least 3 models (staging, intermediate, mart layers), add tests (not null, unique, referential integrity), and generate documentation.
- Implement incremental loading in dbt using dbt_utils or custom SQL: load only new/changed records since the last run.
- Set up error handling and alerting in an Airflow DAG: use try-except blocks, SLAs, and email notifications for failures.
- Write data quality checks (row counts, null checks, schema validation) and integrate them as Airflow tasks before downstream processing.
Next up: This stage equips you with the ability to design and operate reliable, scalable batch pipelines and understand the modern ELT paradigm; the next stage will likely deepen your expertise in real-time streaming, advanced monitoring, and handling complex multi-source scenarios at enterprise scale.
A practical, pattern-focused guide to building pipelines that bridges theory and implementation. Its concise scope makes it the right warm-up before tackling Airflow in depth.
Airflow is the dominant orchestration tool in the industry; this book covers DAG design, operators, and production best practices — essential skills for any data engineer.
dbt has become the standard transformation layer in the modern data stack. Reading it after Airflow clarifies the boundary between orchestration and transformation.
Streaming & Real-Time Data
IntermediateUnderstand event-driven architectures, stream processing semantics (time, state, exactly-once), and how to build low-latency pipelines with Kafka and Flink.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of dense technical content and hands-on labs)
- Kafka architecture: brokers, topics, partitions, replicas, and consumer groups as the foundation for distributed event streaming
- Kafka replication and fault tolerance: in-sync replicas (ISR), leader election, and durability guarantees
- Stream processing semantics: at-most-once, at-least-once, and exactly-once delivery guarantees and their trade-offs
- Time in streaming systems: event time vs. processing time vs. ingestion time, and why it matters for correctness
- Stateful stream processing: windowing (tumbling, sliding, session), state management, and handling late data
- Watermarks and triggers: mechanisms for deciding when to emit results and handling out-of-order events
- Building low-latency pipelines: backpressure, batching trade-offs, and end-to-end latency optimization
- Exactly-once semantics in practice: idempotent writes, transactional guarantees, and practical implementation patterns
- How does Kafka's partition-based architecture enable both scalability and ordering guarantees within a partition?
- What is the difference between event time, processing time, and ingestion time, and why is this distinction critical for streaming applications?
- Explain the trade-offs between at-most-once, at-least-once, and exactly-once semantics in the context of a real-world use case.
- How do watermarks and triggers work together to handle late-arriving data in a streaming pipeline?
- What are the key challenges in implementing exactly-once semantics, and how can idempotent writes and transactional systems help?
- How would you design a low-latency pipeline that processes events from Kafka while maintaining correctness under failures?
- Set up a local Kafka cluster (3 brokers) and create a multi-partition topic; produce and consume messages while observing partition assignment and rebalancing behavior.
- Write a Kafka producer that sends events with different timestamps and a consumer that tracks lag; measure end-to-end latency.
- Implement a simple stream processor (using Kafka Streams or Flink) that counts events in tumbling windows and verify exactly-once semantics by replaying the same input.
- Build a stateful pipeline that tracks user sessions using session windows; test behavior with out-of-order and late-arriving events.
- Design and implement an idempotent sink that writes stream results to a database (e.g., PostgreSQL) with deduplication logic; verify no duplicates after failures.
- Create a multi-stage pipeline (Kafka → processor → Kafka → sink) and introduce failures (broker crashes, consumer crashes); measure recovery time and data loss.
Next up: This stage establishes the foundational patterns and semantics of event-driven systems, positioning you to tackle advanced topics like distributed tracing, complex CEP (Complex Event Processing), and production hardening in the next stage.
Kafka is the backbone of virtually every real-time data platform. This canonical O'Reilly guide covers producers, consumers, streams, and operations — the necessary entry point to streaming.
Goes deep on the theory of streaming: watermarks, triggers, windowing, and the Dataflow model. Read after Kafka to graduate from tool knowledge to principled streaming design.
Distributed Systems at Scale
ExpertInternalize the distributed systems principles — consistency, replication, fault tolerance, and storage engines — that explain why data platforms behave the way they do at scale.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day. Start with DDIA (600 pages, ~3 weeks), then Database Internals (600 pages, ~4 weeks), with 1–2 weeks for review, integration, and hands-on projects.
- Replication strategies (leader-based, leaderless, multi-leader) and their consistency guarantees—why each exists and when to use it
- Consistency models (strong, eventual, causal, read-your-writes) and how they trade off availability and latency
- Fault tolerance mechanisms: quorum reads/writes, consensus algorithms (Raft, Paxos), and handling network partitions
- Storage engines and data structures: B-trees, LSM trees, and how they optimize for different access patterns and hardware constraints
- Distributed transaction semantics: ACID properties across partitions, two-phase commit, and why it fails at scale
- Partitioning and sharding strategies: range-based, hash-based, and consistent hashing—and their impact on rebalancing and hotspots
- Write-ahead logs (WAL) and durability guarantees: how systems prevent data loss during failures
- The CAP theorem and its practical implications: why you cannot have all three properties and how real systems navigate these tradeoffs
- Explain the difference between leader-based and leaderless replication. What are the consistency and availability tradeoffs of each?
- Why do eventual consistency systems require causal consistency or read-your-writes guarantees? Give a concrete example where eventual consistency alone fails.
- Describe how Raft consensus works and why it is easier to reason about than Paxos. How does it handle leader failures?
- Compare B-trees and LSM trees: what workloads does each optimize for, and how do their write and read amplification differ?
- What is a write-ahead log and why is it essential for durability? How does it interact with replication?
- Design a partitioning strategy for a time-series database with millions of writes per second. How would you handle hotspots and rebalancing?
- Implement a simple leader-based replication system in Python or Go: a primary node that accepts writes, replicates to replicas, and handles replica failures by promoting a new leader.
- Build a quorum-based read/write system: implement logic to read from a majority of replicas and write to a majority, then verify consistency under network delays.
- Trace through a Raft consensus scenario (e.g., leader election, log replication, handling split-brain) using a visualization tool or by hand; then read the Raft paper's formal proofs.
- Implement an LSM tree from scratch (or study an existing implementation like RocksDB): understand compaction, memtable flushing, and read amplification.
- Design and simulate a partitioning strategy for a distributed key-value store: define partition keys, rebalancing logic, and measure hotspot distribution under realistic load.
- Write a distributed transaction simulator: implement two-phase commit, then deliberately introduce network failures and observe why it deadlocks or violates consistency.
Next up: This stage grounds you in the *why* behind distributed systems—the fundamental tradeoffs and mechanisms—which prepares you to evaluate and design specific data platforms (message queues, data warehouses, stream processors) in the next stage, where you'll apply these principles to real-world architectures.
The definitive book on the internals of distributed data systems. Placed last because it synthesizes everything — storage engines, replication, partitioning, stream processing — and rewards readers who already have real pipeline experience.
Digs into storage engine design, B-trees, LSM trees, and distributed consensus algorithms. Pairs with Kleppmann to give you a complete picture of how the storage layer actually works under your pipelines.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
