Reinforcement learning: books from foundations to deep RL
This curriculum builds a rigorous, end-to-end mastery of reinforcement learning, starting from the mathematical and algorithmic foundations (MDPs, dynamic programming, temporal-difference learning) and progressing through policy gradients, deep RL, and cutting-edge agent architectures. Because the learner starts at an intermediate level, the path skips introductory ML primers and dives straight into core RL theory, then layers on deep learning integration and modern research-grade techniques.
Core RL Theory & Foundations
IntermediateUnderstand Markov decision processes, Bellman equations, dynamic programming, Monte Carlo methods, and temporal-difference learning — the mathematical backbone of all RL algorithms.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (Sutton first 4–5 weeks, Szepesvari 3–5 weeks, with overlap for consolidation)
- Markov Decision Processes (MDPs): state spaces, action spaces, transition probabilities, reward functions, and the Markov property
- Bellman equations: recursive decomposition of value functions (V and Q) and their role in optimal control
- Value iteration and policy iteration: dynamic programming algorithms for solving MDPs with known models
- Monte Carlo methods: learning from complete episodes without a model, handling exploration-exploitation tradeoffs
- Temporal-difference (TD) learning: bootstrapping value estimates from incomplete trajectories, combining DP and MC
- On-policy vs. off-policy learning: SARSA, Q-learning, and importance sampling for policy evaluation and improvement
- Function approximation foundations: why tabular methods scale poorly and how linear/nonlinear approximators extend RL
- Convergence guarantees: conditions under which RL algorithms provably converge to optimal policies
- What is the Markov property, and why is it central to formulating reinforcement learning problems as MDPs?
- Derive the Bellman optimality equation for V*(s) and explain how it relates to the optimal policy.
- Compare value iteration and policy iteration: what are their computational complexities, convergence rates, and when would you use each?
- Explain the bias-variance tradeoff between Monte Carlo and temporal-difference methods, with concrete examples.
- What is the difference between on-policy (SARSA) and off-policy (Q-learning) learning, and why does Q-learning require importance sampling or epsilon-greedy exploration?
- Under what conditions do tabular RL algorithms (value iteration, Q-learning) converge, and how does function approximation affect these guarantees?
- Implement value iteration and policy iteration from scratch on a small gridworld (e.g., 5×5); verify convergence and compare iteration counts.
- Code a Monte Carlo agent that learns from complete episodes in a simple environment (e.g., blackjack or CartPole); plot learning curves.
- Implement tabular Q-learning with epsilon-greedy exploration on a discrete MDP; compare on-policy (SARSA) vs. off-policy (Q-learning) learning curves.
- Derive the Bellman equations by hand for a custom 3-state MDP; solve it analytically and verify your solution with code.
- Implement a TD(0) agent and a Monte Carlo agent on the same problem; analyze their sample efficiency and variance empirically.
- Extend one of your implementations to use linear function approximation (e.g., tile coding or polynomial features); observe how approximation error affects convergence.
Next up: Mastery of these foundational algorithms and theory equips you to understand modern deep RL methods (DQN, policy gradients, actor-critic), which combine these core principles with neural network function approximation and large-scale exploration strategies.
The canonical, comprehensive textbook for RL — it defines MDPs, value functions, Q-learning, and policy gradients with clarity and rigor. Every subsequent book in this curriculum assumes familiarity with its concepts.
A concise, mathematically precise monograph that formalizes the convergence guarantees and complexity of the core algorithms introduced in Sutton & Barto, bridging intuition with theory.
Deep Learning Prerequisites for Deep RL
IntermediateBuild a solid working understanding of neural networks, optimization, and the deep learning toolkit that underpins all modern deep RL agents.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (focusing on Parts I–III: Foundations, Modern Practical Architectures, and Regularization)
- Computational graphs and automatic differentiation: how backpropagation computes gradients through nested function compositions
- Stochastic gradient descent and variants (momentum, Adam, RMSprop): the optimization algorithms that train neural networks
- Feedforward neural networks and universal approximation: why deep networks can learn complex functions
- Convolutional and recurrent architectures: specialized network designs for images and sequences
- Regularization techniques (dropout, L1/L2, batch normalization, early stopping): preventing overfitting and improving generalization
- Loss functions and output units: matching network architecture to the learning problem (classification, regression, structured prediction)
- Vanishing/exploding gradients and architectural solutions: why deep networks are hard to train and how to fix it
- Representation learning: how hidden layers learn useful feature hierarchies automatically
- Explain how backpropagation uses the chain rule to compute gradients through a deep network, and why this is more efficient than numerical differentiation
- Compare and contrast SGD, momentum, and adaptive learning rate methods (Adam, RMSprop). When would you use each, and what are their trade-offs?
- What is the universal approximation theorem, and what does it tell us (and not tell us) about the practical power of neural networks?
- Describe the vanishing gradient problem: why it occurs in deep networks, and what architectural solutions (ReLU, residual connections, LSTM gates) help address it
- How do convolutional layers exploit spatial structure, and why are they more efficient than fully-connected layers for image data?
- Explain the purpose and mechanics of dropout, batch normalization, and L1/L2 regularization. How do they reduce overfitting?
- Implement backpropagation from scratch for a simple 2–3 layer feedforward network on MNIST or a toy dataset; verify gradients numerically
- Train a feedforward network on MNIST with different optimizers (SGD, momentum, Adam) and learning rates; plot training curves and compare convergence speed
- Build a CNN for CIFAR-10 or ImageNet-scale classification; experiment with different architectures (VGG-style, ResNet-style) and measure accuracy
- Implement dropout and batch normalization in a deep network; measure the impact on training speed, final accuracy, and generalization gap
- Train an LSTM or GRU on a sequence task (e.g., character-level language modeling or time-series prediction); observe how gating mechanisms help with long-term dependencies
- Conduct a regularization study: train the same architecture with varying L2 penalties, dropout rates, and early stopping thresholds; plot validation curves to understand the bias–variance trade-off
Next up: Mastery of these deep learning fundamentals—especially gradient-based optimization, network architectures, and generalization—provides the essential toolkit for understanding how deep RL agents learn policies and value functions through backpropagation and experience replay.
Provides the essential neural network theory — backpropagation, regularization, optimization, and representation learning — needed to understand how function approximation works inside deep RL agents.
Deep Reinforcement Learning
IntermediateLearn how to combine deep neural networks with RL algorithms — covering DQN, policy gradient methods (REINFORCE, A3C, PPO), actor-critic architectures, and model-based RL.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (with code walkthroughs and experiments interspersed)
- Deep Q-Networks (DQN): architecture, experience replay, target networks, and how they stabilize learning from high-dimensional inputs
- Policy Gradient Methods: REINFORCE algorithm, advantage estimation, and the relationship between policy optimization and gradient ascent
- Actor-Critic Architectures: dual networks for policy and value estimation, advantage functions, and how they reduce variance while maintaining low bias
- Advanced Policy Gradients: A3C (Asynchronous Advantage Actor-Critic) for parallel training and PPO (Proximal Policy Optimization) for stable, sample-efficient learning
- Model-Based Reinforcement Learning: planning with learned environment models, latent dynamics, and integration with model-free methods
- Exploration vs. Exploitation in Deep RL: ε-greedy, entropy regularization, and curiosity-driven exploration in high-dimensional spaces
- Implementation patterns: neural network architectures for RL, handling continuous vs. discrete action spaces, and debugging deep RL agents
- Practical considerations: hyperparameter tuning, reward shaping, and evaluating deep RL agents in complex environments
- Why do DQN agents use experience replay and target networks, and what problems do these mechanisms solve?
- How does the policy gradient theorem enable REINFORCE, and why is the baseline/advantage function critical for reducing variance?
- What is the key difference between on-policy and off-policy learning, and how do A3C and PPO exemplify different approaches?
- How do actor-critic methods combine the benefits of policy gradients and value-based learning, and what role does the critic play?
- What are the main challenges in model-based RL, and how can learned models be integrated with model-free algorithms?
- How would you design an exploration strategy for a deep RL agent in a sparse-reward environment, and what trade-offs exist?
- Implement a DQN agent from scratch on Atari (e.g., Pong or Breakout) using PyTorch, including experience replay and target networks; compare learning curves with and without these stabilization techniques
- Code REINFORCE with a baseline on a continuous control task (e.g., CartPole or MuJoCo), then measure how the baseline reduces gradient variance
- Build an A3C agent that trains multiple workers in parallel on a shared policy and value network; run it on a discrete action space environment and observe speedup vs. single-threaded learning
- Implement PPO (clipped objective) on a continuous control task; experiment with different clip ranges and compare sample efficiency to REINFORCE and A3C
- Develop a simple model-based RL agent: train a forward dynamics model, use it for planning (e.g., random shooting or cross-entropy method), and compare planning-only vs. planning + model-free fine-tuning
- Implement an actor-critic agent (A2C) on a discrete action space; ablate the advantage function (use returns vs. TD residuals) and measure impact on convergence speed and stability
Next up: Mastering deep RL algorithms and their implementation prepares you to tackle advanced topics like multi-agent RL, hierarchical RL, meta-learning, and real-world deployment challenges where sample efficiency, robustness, and scalability become critical.
Bridges theory and practice by implementing DQN, policy gradients, actor-critic, and more in PyTorch — ideal as the first hands-on deep RL text after mastering the fundamentals.
Uses visual intuition and step-by-step code to solidify understanding of policy gradients, advantage functions, and PPO — reinforcing algorithmic intuition before moving to research-level material.
Advanced Topics & Modern Agents
ExpertEngage with advanced RL paradigms — multi-agent RL, hierarchical RL, exploration strategies, offline RL, and the theoretical underpinnings of modern agent design.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day with 2–3 days/week for implementation exercises
- Deep Q-Networks (DQN) and extensions: Double DQN, Dueling DQN, Prioritized Experience Replay
- Policy gradient methods: REINFORCE, Actor-Critic architectures, A3C, and PPO
- Multi-agent reinforcement learning (MARL): cooperative vs. competitive settings, communication, credit assignment
- Hierarchical reinforcement learning: options framework, temporal abstraction, and skill learning
- Exploration strategies: ε-greedy, UCB, Thompson sampling, curiosity-driven learning, and intrinsic motivation
- Offline reinforcement learning: batch RL, distributional shift, conservative Q-learning, and offline policy evaluation
- Function approximation theory: convergence guarantees, stability, and overestimation bias in deep RL
- Modern agent architectures: attention mechanisms, transformers in RL, and scalable training paradigms
- What are the key limitations of vanilla DQN, and how do Double DQN and Dueling DQN address them?
- How do policy gradient methods differ from value-based methods, and when is each approach preferable?
- What are the main challenges in multi-agent RL, and how do cooperative and competitive settings differ in their solutions?
- Explain the options framework and how hierarchical RL enables temporal abstraction and skill reuse.
- What is the exploration-exploitation trade-off, and how do modern exploration strategies (curiosity, intrinsic motivation) improve upon ε-greedy?
- Why is offline RL important, and what are the key challenges (distributional shift, value overestimation) when learning from fixed datasets?
- Implement Double DQN and Dueling DQN from scratch on Atari or continuous control tasks; compare convergence and sample efficiency against vanilla DQN.
- Build a multi-agent RL environment (e.g., predator-prey or cooperative navigation) and implement both independent learners and a centralized critic approach.
- Code the options framework: define a set of options (skills) for a domain, train an option-level policy, and measure hierarchical abstraction benefits.
- Implement curiosity-driven exploration (intrinsic motivation) using prediction error and compare sample efficiency against ε-greedy baselines.
- Develop an offline RL agent using Conservative Q-Learning (CQL) on a fixed dataset; evaluate robustness to distributional shift.
- Reproduce a key result from the book (e.g., A3C on a continuous control task or PPO on a benchmark) and analyze hyperparameter sensitivity.
Next up: This stage equips you with the theoretical foundations and practical mastery of modern deep RL agents, positioning you to explore emerging frontiers such as meta-learning, world models, and language-guided RL in the next stage.
Covers the full landscape of modern deep RL algorithms (PPO, SAC, TD3, SLM) with clean implementations, serving as a bridge between applied deep RL and research-grade agent design.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
