Deep learning: a reading path from neural networks to modern architectures
This curriculum takes an intermediate learner from solid mathematical foundations through the full depth of modern deep learning — connecting theory to implementation at every step. Each stage builds directly on the last: first sharpening the math and intuition, then mastering core neural network mechanics, then scaling to advanced architectures like CNNs and Transformers that power today's state-of-the-art models.
Mathematical & Conceptual Foundations
IntermediateRefresh and solidify the linear algebra, probability, and calculus needed to follow deep learning derivations without getting lost, and build an intuitive map of the field.
▸ Study plan for this stage
Pace: 8–10 weeks, ~25–30 pages/day. Start with "Mathematics for Machine Learning" (Chapters 2–5: Linear Algebra, Analytic Geometry, Matrix Decompositions, Vector Calculus) over 5–6 weeks, then "The Hundred-Page Machine Learning Book" over 2–3 weeks for conceptual synthesis and field overview.
- Vector spaces, linear transformations, and matrix operations as the language for representing data and model parameters
- Eigendecomposition and SVD: how to understand data structure and model behavior through matrix factorization
- Gradients, Jacobians, and the chain rule as the foundation for backpropagation and optimization
- Probability distributions, Bayes' theorem, and maximum likelihood estimation as the statistical framework underlying learning
- Convexity and loss surfaces: why some optimization problems are tractable and others are not
- The bias-variance tradeoff and generalization: why models fail and how regularization helps
- The unified view of supervised learning: loss functions, empirical risk minimization, and the role of inductive bias
- Why is the chain rule essential for deep learning, and how does it connect to backpropagation?
- What does it mean geometrically when we say a neural network learns a non-linear transformation of the input space?
- How do eigenvalues and eigenvectors help us understand what a linear transformation does to data?
- What is the relationship between maximum likelihood estimation and the loss functions used in deep learning?
- Why does regularization help prevent overfitting, and what does the bias-variance tradeoff tell us about model complexity?
- How does the Jacobian matrix generalize the concept of a derivative to multivariate functions, and why does this matter for neural networks?
- Implement matrix operations (multiplication, inversion, transpose) from scratch in NumPy and verify against built-in functions; compute eigendecomposition and SVD for a sample dataset and interpret the results geometrically.
- Derive the chain rule for a 3-layer neural network by hand, then verify your result by computing gradients numerically and comparing to symbolic differentiation.
- Write code to compute the Jacobian matrix of a simple function (e.g., a 2-layer network) and visualize how it changes as you vary the weights.
- Given a dataset, compute the empirical risk (mean squared error or cross-entropy) and manually apply gradient descent for 5–10 steps; plot the loss curve and verify it decreases.
- Implement L2 regularization in a simple linear regression model and plot how test error changes with regularization strength; explain the bias-variance tradeoff in your results.
- Work through 2–3 derivations from 'Mathematics for Machine Learning' (e.g., deriving the normal equations for linear regression, or the formula for the variance of a linear transformation) without looking at the solution first.
Next up: This stage equips you with the mathematical vocabulary and intuition to understand how neural networks transform data through compositions of linear and non-linear operations, and why optimization via gradient descent works—preparing you to study the architecture and training dynamics of actual deep learning models.
Covers linear algebra, multivariate calculus, and probability in one cohesive text aimed squarely at ML practitioners — exactly the mathematical toolkit needed before diving into backpropagation and optimization.
A concise, dense survey of ML concepts that bridges classical methods and neural networks, giving the learner a mental scaffold on which all subsequent deep learning material will hang.
Core Deep Learning — Theory & Practice
IntermediateUnderstand neural networks end-to-end: forward passes, backpropagation, loss functions, regularization, and the full training loop, with both the math and working code.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (focusing on Parts I–II and selected chapters from Part III; approximately 600–700 pages total)
- Linear algebra and calculus foundations: vectors, matrices, gradients, and chain rule as the mathematical backbone of neural networks
- Forward propagation: how data flows through layers, activation functions, and why nonlinearity is essential
- Backpropagation algorithm: computing gradients via the chain rule, understanding computational graphs, and why it enables efficient learning
- Loss functions and optimization: cross-entropy, MSE, and how gradient descent (SGD, momentum, adaptive methods) updates weights
- Regularization techniques: L1/L2 penalties, dropout, batch normalization, and early stopping to prevent overfitting
- Neural network architecture design: choosing layer sizes, activation functions, and depth for different problems
- The complete training loop: data preprocessing, batching, validation, hyperparameter tuning, and debugging failed training
- Implementing networks from scratch: building forward/backward passes in code to internalize the mechanics
- Explain the chain rule and how it enables backpropagation to compute gradients efficiently through a deep network.
- What is the difference between a forward pass and a backward pass, and why are both necessary for training?
- How do different loss functions (cross-entropy vs. MSE) relate to different problem types, and what does minimizing them achieve?
- Why do we need activation functions like ReLU or sigmoid, and what happens if we stack linear layers without them?
- Describe how regularization (L2, dropout, batch norm) reduces overfitting and when you would use each technique.
- Walk through a complete training loop: data loading, forward pass, loss computation, backpropagation, and weight updates.
- Implement a 2-layer fully connected network from scratch (NumPy only) with forward pass, backpropagation, and SGD; train it on MNIST or a toy dataset.
- Derive backpropagation equations by hand for a 3-layer network; verify your math by comparing to numerical gradients (gradient checking).
- Build a neural network using PyTorch or TensorFlow; experiment with different activation functions, layer sizes, and learning rates; visualize how loss curves change.
- Implement L2 regularization and dropout from scratch; compare training/validation curves with and without regularization on an overfitting-prone problem.
- Train a network on a dataset with poor preprocessing (e.g., unscaled inputs); then preprocess correctly and observe the impact on convergence speed.
- Debug a failing training run: identify whether the problem is bad initialization, learning rate too high/low, architecture mismatch, or data issues.
Next up: This stage equips you with the foundational theory and hands-on intuition for how neural networks learn; the next stage will build on this by exploring specialized architectures (CNNs, RNNs) and advanced techniques (attention, transfer learning) for specific domains.
The canonical graduate-level textbook — rigorously derives backpropagation, optimization, and regularization from first principles, making it the definitive reference for the math behind training neural networks.
Hands-On Implementation
IntermediateTranslate theory into working models using modern frameworks, building and training CNNs, RNNs, and other architectures from scratch so the math becomes executable code.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day with 2–3 hours daily hands-on coding
- Keras/TensorFlow API fundamentals: layers, models, and the functional API for building flexible architectures
- Training loops and optimization: backpropagation, gradient descent, loss functions, and monitoring convergence
- CNNs from first principles: convolutions, pooling, feature maps, and architectural patterns (LeNet, VGG, ResNet)
- RNNs and sequence modeling: LSTM/GRU cells, vanishing gradients, and handling variable-length sequences
- Data pipelines and preprocessing: batching, normalization, augmentation, and efficient data loading
- Debugging and validation: overfitting detection, regularization techniques (dropout, L1/L2), and hyperparameter tuning
- Transfer learning and fine-tuning: leveraging pretrained models and adapting them to new tasks
- Practical workflow: from problem definition through model evaluation, using real datasets (MNIST, CIFAR-10, ImageNet subsets)
- How do you construct a CNN in Keras/TensorFlow, and what is the role of each layer type (Conv2D, MaxPooling, Dense)?
- Explain the difference between the Sequential API and the Functional API, and when would you use each?
- What causes vanishing gradients in RNNs, and how do LSTM and GRU cells address this problem?
- How do you implement a custom training loop using tf.GradientTape, and why might you need one beyond the standard model.fit()?
- Describe the complete pipeline for preparing image data: loading, normalization, augmentation, and batching.
- What are the signs of overfitting, and what regularization techniques (dropout, L1/L2, early stopping) would you apply?
- How does transfer learning work, and what are the steps to fine-tune a pretrained model like VGG or ResNet on a new dataset?
- Build a CNN from scratch in Keras to classify MNIST or CIFAR-10, experimenting with different architectures (2–3 conv blocks, pooling, dropout) and comparing accuracy.
- Implement a custom training loop using tf.GradientTape for a simple CNN, manually computing gradients and updating weights, then compare results to model.fit().
- Create an RNN/LSTM model to predict the next character in a text sequence (using a small text file), tracking loss over epochs and observing convergence.
- Build a data pipeline using tf.data.Dataset: load images, normalize, augment (rotation, flip, zoom), and batch them efficiently for training.
- Implement transfer learning: load a pretrained model (VGG16 or ResNet50), freeze early layers, add custom dense layers, and fine-tune on a small custom dataset (e.g., cats vs. dogs).
- Experiment with regularization: train the same CNN with and without dropout/L1/L2, plot training vs. validation loss, and identify the point of overfitting.
Next up: This stage equips you with the ability to execute deep learning architectures in production-ready code; the next stage will deepen your understanding of advanced architectures (Transformers, attention mechanisms, GANs) and specialized domains (NLP, computer vision at scale), building on the solid implementation foundation you've now established.
Written by the creator of Keras, this book is the gold standard for moving from equations to running experiments — covering CNNs, sequence models, and best practices for training real models.
Uniquely interleaves math derivations with runnable PyTorch/MXNet code for every concept — CNNs, RNNs, attention, and optimization — making it the perfect complement to Chollet's Keras-centric view.
Advanced Architectures — CNNs, Attention & Transformers
ExpertMaster the architectural innovations — convolutional networks, attention mechanisms, and the Transformer — that define modern deep learning, understanding both their design rationale and mathematical underpinnings.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (alternating between both books; approximately 4–5 weeks per book with overlap for integration)
- Convolutional Neural Networks (CNNs): filters, feature maps, pooling, and spatial hierarchy for vision tasks
- CNN architectures: ResNet, VGG, Inception, and their design innovations (skip connections, multi-scale processing)
- Attention mechanisms: query-key-value framework, scaled dot-product attention, and why attention solves sequence modeling limitations
- Multi-head attention: parallel attention subspaces and their role in capturing diverse relationships
- Transformer architecture: encoder-decoder structure, positional encoding, and self-attention as a replacement for recurrence
- Transfer learning and fine-tuning: leveraging pre-trained vision and language models for downstream tasks
- Practical implementation: building, training, and optimizing CNNs in TensorFlow/Keras and Transformers with Hugging Face
- Mathematical foundations: backpropagation through convolutional layers, attention weight computation, and gradient flow in deep architectures
- How do convolutional filters extract hierarchical features, and why is this superior to fully connected layers for vision tasks?
- What is the motivation behind skip connections in ResNet, and how do they address the vanishing gradient problem in very deep networks?
- Explain the scaled dot-product attention mechanism: what are the roles of query, key, and value, and why is scaling by √d_k important?
- How does multi-head attention allow a Transformer to attend to different types of relationships simultaneously, and why is this better than single-head attention?
- What is positional encoding in Transformers, why is it necessary (given that self-attention is permutation-invariant), and how does it differ from RNN position tracking?
- Compare CNNs and Transformers for vision: what are the trade-offs in terms of inductive bias, computational cost, and data efficiency?
- How do you fine-tune a pre-trained model (vision or language) for a new task, and what are the key hyperparameters and techniques to avoid overfitting?
- Implement a custom CNN from scratch in TensorFlow/Keras with multiple convolutional and pooling layers; train it on CIFAR-10 and visualize learned filters at different depths
- Build a ResNet-50 model using Keras applications, replace the final classification layer for a custom dataset, and compare accuracy with and without fine-tuning
- Implement scaled dot-product attention from scratch (matrix operations only, no frameworks); verify correctness by comparing outputs with tf.nn.softmax or PyTorch equivalents
- Code a multi-head attention layer manually; apply it to a toy sequence (e.g., 10 tokens, 8 heads) and visualize attention weights to understand what different heads learn
- Build a minimal Transformer encoder using Hugging Face Transformers; load a pre-trained model (e.g., BERT), inspect its attention patterns on a sample sentence, and explain what relationships it captures
- Fine-tune a pre-trained vision model (e.g., EfficientNet or Vision Transformer) on a custom image classification dataset; track validation metrics and experiment with learning rate schedules
- Fine-tune a pre-trained language model (e.g., DistilBERT) on a text classification task using the Hugging Face Transformers library; compare performance with a baseline CNN-based text classifier
Next up: This stage equips you with the architectural foundations and implementation skills for modern deep learning; the next stage will deepen your understanding of training dynamics, optimization strategies, and how to scale these architectures to production systems.
Focuses deeply on CNN architectures (ResNet, VGG, Inception) and their application to vision tasks, consolidating convolutional intuition before moving to attention-based models.
The most practical and thorough treatment of the Transformer architecture and its variants (BERT, GPT, T5) — connects the attention math to real fine-tuning workflows, completing the architectural journey.
Research Depth & Unified Perspective
ExpertSynthesize everything into a researcher's perspective — understanding generalization, probabilistic deep learning, and the open questions that connect today's models to tomorrow's breakthroughs.
▸ Study plan for this stage
Pace: 4–5 weeks, ~25–30 pages/day, with 2–3 days per week reserved for synthesis and reflection
- The historical arc of deep learning: from McCulloch-Pitts neurons to modern transformers, and how each era solved specific bottlenecks
- Why deep learning works: the role of hierarchical representations, distributed representations, and implicit regularization in achieving generalization
- Probabilistic perspectives on deep learning: Bayesian interpretations, uncertainty quantification, and connections to statistical physics
- The brain as inspiration and constraint: how neuroscience insights shaped architectures (CNNs, attention) and where biological plausibility breaks down
- Open problems in deep learning: adversarial robustness, interpretability, sample efficiency, and the path toward artificial general intelligence
- Scaling laws and emergent phenomena: how increasing model and data size reveals new capabilities and phase transitions
- The role of inductive biases: how architectural choices (convolution, attention, recurrence) encode assumptions about the world
- What were the key computational bottlenecks that prevented deep learning from succeeding before ~2012, and how were they overcome?
- How does Sejnowski explain why deep networks generalize well despite having far more parameters than training examples?
- What is the relationship between the brain's learning mechanisms and the algorithms we use to train artificial neural networks, and where do they diverge?
- How do probabilistic interpretations of deep learning (e.g., Bayesian neural networks, energy-based models) change our understanding of what networks learn?
- What are the major open questions in deep learning that Sejnowski identifies, and why are they difficult to solve?
- How do scaling laws and emergent capabilities challenge our theoretical understanding of deep learning, and what do they suggest about future directions?
- Create a timeline of deep learning milestones (perceptrons → backprop → CNNs → RNNs → transformers) and annotate each with the key problem it solved and the computational/theoretical insight that enabled it
- Write a 2–3 page synthesis essay: 'Why Deep Learning Generalizes' that integrates Sejnowski's explanations of implicit regularization, hierarchical learning, and inductive biases
- Implement or analyze a simple Bayesian neural network (e.g., using variational inference) and compare its uncertainty estimates to a standard network; document what you learn about probabilistic perspectives
- Select one open problem Sejnowski discusses (adversarial robustness, interpretability, or AGI alignment) and write a research proposal outline (1–2 pages) identifying the bottleneck and proposing a direction forward
- Create a concept map connecting: brain mechanisms → architectural inductive biases → empirical phenomena (e.g., how visual cortex structure → convolution → translation invariance)
- Critically read one recent deep learning paper (2023–2024) and annotate it against Sejnowski's framework: which historical insights does it build on, which open problems does it address, and what new questions does it raise?
Next up: This stage establishes the researcher's mental model—understanding *why* deep learning works, *what remains unknown*, and *how* historical insights constrain future directions—preparing you to either specialize deeply in a subfield or engage with cutting-edge research that pushes against current theoretical and practical limits.
A narrative history of deep learning by one of its pioneers — reading this last provides essential context for why architectures evolved as they did and where the field's open frontiers lie.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
