Computer vision reading path: from image basics to deep learning models
This curriculum takes an intermediate practitioner from the mathematical and algorithmic foundations of image processing through classical computer vision techniques, and into modern deep learning-based visual recognition systems. Each stage builds directly on the vocabulary and intuition of the last, culminating in the ability to design and train state-of-the-art convolutional and transformer-based vision models.
Image Processing Foundations
IntermediateUnderstand how digital images are represented, filtered, transformed, and analyzed at the pixel and frequency level — the bedrock vocabulary for everything that follows.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (Gonzalez: weeks 1–5, ~45 pages/day; Szeliski: weeks 6–10, ~35 pages/day)
- Digital image representation: pixels, color spaces (RGB, grayscale, HSV), bit depth, and image coordinate systems
- Spatial filtering and convolution: kernels, neighborhoods, and how filters modify pixel values based on local context
- Frequency domain analysis: Fourier transform, frequency components, and how images decompose into sinusoidal patterns
- Image enhancement and restoration: histogram equalization, noise reduction, and sharpening via spatial and frequency methods
- Geometric transformations: interpolation, rotation, scaling, and warping while preserving or modifying image structure
- Edge detection and feature extraction: gradients, Sobel/Laplacian operators, and identifying salient image structures
- Image pyramids and multi-scale analysis: coarse-to-fine representations for hierarchical image processing
- Sampling, aliasing, and the Nyquist theorem: why resolution matters and how to avoid artifacts in image manipulation
- How are digital images represented in memory, and what do bit depth and color spaces tell you about image quality and storage?
- Explain convolution: what is a kernel, how does it slide across an image, and why is this operation fundamental to filtering?
- What is the Fourier transform of an image, and how does it reveal frequency components that spatial filtering cannot easily show?
- How do you enhance a blurry or low-contrast image using histogram equalization and frequency-domain sharpening?
- Describe the difference between nearest-neighbor and bilinear interpolation when resizing an image, and when would you use each?
- How do edge detection operators (Sobel, Laplacian) work, and what do they reveal about image structure that raw pixel values do not?
- Load a grayscale image and manually compute convolution with a 3×3 Gaussian kernel on a small patch; verify against a library implementation (OpenCV, scikit-image)
- Apply multiple spatial filters (blur, sharpen, edge detection) to the same image and visualize the results side-by-side; explain what each filter reveals
- Compute the 2D Fourier transform of an image, visualize the magnitude spectrum, and reconstruct the image by zeroing out high or low frequencies; observe the visual effect
- Perform histogram equalization on a low-contrast image and plot the before/after histograms; explain why the contrast improves
- Resize an image using nearest-neighbor and bilinear interpolation; compare artifacts and discuss trade-offs between speed and quality
- Build an image pyramid (Gaussian or Laplacian) for a test image and explain how coarse levels capture global structure while fine levels preserve detail
Next up: This stage equips you with the low-level image analysis tools and vocabulary—filtering, frequency analysis, edge detection—that are essential building blocks for the next stage, which will apply these foundations to higher-level tasks like feature matching, segmentation, and object recognition.
The canonical reference for image processing fundamentals — spatial filtering, Fourier transforms, morphological operations, and segmentation. Reading this first ensures you speak the language every later CV text assumes.
Bridges low-level image processing to higher-level vision tasks (stereo, flow, recognition). Its breadth and freely available online edition make it the ideal transition into classical computer vision after Gonzalez.
Classical Computer Vision & Feature Detection
IntermediateMaster hand-crafted feature detectors (edges, corners, SIFT, HOG), geometric transformations, camera models, and multi-view geometry — the classical toolkit still used in production pipelines.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day from Hartley; 2–3 weeks, ~60 pages/day from Jan Erik Solem (practical implementation phase)
- Pinhole camera model and intrinsic/extrinsic parameters — the mathematical foundation for how cameras capture 3D scenes into 2D images
- Epipolar geometry and the fundamental matrix — the constraint that relates corresponding points across two views and enables 3D reconstruction
- Feature detection and matching (edges, corners, SIFT, HOG) — hand-crafted descriptors that identify and track salient points across images
- Geometric transformations (homography, affine, projective) — how 2D and 3D points transform under different camera poses and scene configurations
- Triangulation and multi-view reconstruction — recovering 3D structure from multiple 2D image correspondences
- Pose estimation and camera calibration — determining camera position/orientation and internal parameters from known 3D–2D correspondences
- Practical implementation in Python — translating mathematical theory into working code for real image processing pipelines
- What is the pinhole camera model and how do intrinsic and extrinsic parameters differ in describing the projection from 3D world to 2D image?
- Explain epipolar geometry: what is the fundamental matrix and why does it constrain corresponding points in two views?
- How do SIFT and HOG differ as feature descriptors, and when would you choose one over the other in a practical application?
- What is homography and how does it relate to planar scenes or camera rotation? How does it differ from the fundamental matrix?
- Given two calibrated cameras and matched feature points, how would you triangulate 3D points? What assumptions must hold?
- How does camera calibration work, and what is the relationship between the calibration matrix K and the fundamental matrix F?
- Implement the pinhole camera projection: write code to project 3D points onto a 2D image plane given K, R, and t matrices; verify with synthetic data
- Compute the fundamental matrix from 8+ matched point correspondences using the normalized 8-point algorithm; validate using the epipolar constraint
- Implement or use a library to detect corners (Harris corner detector) and edges (Canny) on real images; compare results and discuss trade-offs
- Extract and match SIFT features between two images using OpenCV or similar; visualize matches and identify outliers
- Compute a homography matrix between two images of a planar scene; warp one image to align with the other and assess accuracy
- Perform stereo triangulation: given two calibrated cameras with matched features, recover 3D point positions and visualize the reconstruction
- Implement camera pose estimation (PnP) using known 3D–2D correspondences; compare your results to ground truth or library implementations
- Build an end-to-end pipeline: detect features, match across two images, compute epipolar geometry, triangulate, and visualize the 3D point cloud
Next up: This stage equips you with the classical geometric and feature-based tools that form the backbone of production vision systems; the next stage will build on this foundation by introducing learned representations (deep learning for feature detection and matching) and modern end-to-end approaches that automate or improve upon these hand-crafted methods.
The definitive treatment of projective geometry, homographies, epipolar geometry, and structure-from-motion — essential for anyone building 3D vision or SLAM systems.
A concise, hands-on complement that reinforces feature matching, image stitching, and 3D reconstruction in Python, solidifying the geometric intuition from Hartley through direct implementation.
Deep Learning Essentials for Vision
IntermediateBuild a solid understanding of neural networks, backpropagation, regularization, and optimization — the theoretical engine powering all modern computer vision models.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (alternating between theory and hands-on chapters)
- Neural network architecture: neurons, layers, activation functions, and forward propagation
- Backpropagation algorithm: computing gradients through the chain rule and weight updates
- Loss functions and optimization: gradient descent variants (SGD, momentum, Adam) and learning rate scheduling
- Regularization techniques: L1/L2 penalties, dropout, batch normalization, and early stopping to prevent overfitting
- Convolutional neural networks (CNNs): filters, convolutions, pooling, and why they excel at image tasks
- Training dynamics: initialization strategies, vanishing/exploding gradients, and debugging neural networks
- Practical implementation: building, training, and evaluating models with Keras/TensorFlow on real datasets
- Explain how backpropagation uses the chain rule to compute gradients, and why this is more efficient than numerical differentiation.
- What is the difference between batch gradient descent, stochastic gradient descent, and mini-batch gradient descent, and when would you use each?
- How do dropout and batch normalization reduce overfitting, and what are their computational trade-offs?
- Describe the architecture of a convolutional neural network: what do filters, convolutions, and pooling layers do, and why are they suited for images?
- What causes vanishing and exploding gradients in deep networks, and what techniques (initialization, activation functions, normalization) help mitigate them?
- Given a trained neural network, how would you diagnose whether it is underfitting or overfitting, and what regularization or architectural changes would you try?
- Implement a simple feedforward neural network from scratch in NumPy (Goodfellow Ch. 6–7): compute forward pass, backpropagation, and weight updates on a toy dataset (e.g., XOR problem).
- Build a multi-layer perceptron with Keras/TensorFlow (Géron Ch. 10) on the MNIST dataset; experiment with different activation functions, layer sizes, and learning rates; plot training/validation loss curves.
- Implement and compare three optimizers (SGD, momentum, Adam) on the same dataset; visualize convergence speed and final loss (Goodfellow Ch. 8, Géron Ch. 11).
- Train a CNN on CIFAR-10 or Fashion-MNIST (Géron Ch. 14); visualize learned filters in the first layer and feature maps in intermediate layers to understand what the network learns.
- Apply L1/L2 regularization and dropout to an overfitting model; plot training vs. validation accuracy and measure the effect on generalization (Goodfellow Ch. 7, Géron Ch. 11).
- Implement batch normalization in a deep network and compare training speed and final accuracy with and without it (Goodfellow Ch. 8, Géron Ch. 11).
- Debug a failing neural network: identify vanishing gradients using gradient histograms, adjust initialization (He/Xavier), and verify the fix (Goodfellow Ch. 8, Géron Ch. 11).
Next up: Mastering these foundational neural network concepts—architecture, training dynamics, and regularization—equips you to understand and implement specialized vision architectures (ResNets, VGG, Inception) and tackle real-world vision tasks like object detection, segmentation, and image classification in the next stage.
The authoritative theoretical foundation for deep learning, covering the math of optimization, regularization, and network architectures. Reading this before CNN-specific texts prevents treating deep learning as a black box.
Translates deep learning theory into practical, runnable code and introduces CNNs in a project-driven way, making it the perfect applied companion to Goodfellow before diving into vision-specific architectures.
Convolutional Neural Networks & Modern Visual Recognition
ExpertUnderstand and implement modern CNN architectures (ResNet, Inception, EfficientNet), object detection (YOLO, Faster R-CNN), segmentation, and self-supervised vision models for real-world recognition systems.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of reading and implementation)
- Convolutional layer mechanics: filters, feature maps, receptive fields, and parameter sharing
- Modern CNN architectures: ResNet skip connections and residual blocks, Inception multi-scale feature extraction, EfficientNet scaling principles (depth, width, resolution)
- Object detection frameworks: YOLO real-time detection pipeline, Faster R-CNN region-based detection, anchor boxes, non-maximum suppression, and loss functions
- Semantic and instance segmentation: fully convolutional networks, U-Net encoder-decoder architecture, mask generation, and per-pixel classification
- Self-supervised learning for vision: contrastive learning (SimCLR, MoCo), pretext tasks, and transfer learning without labels
- Training optimization: batch normalization, learning rate scheduling, data augmentation strategies, and handling class imbalance
- Practical deployment: model quantization, pruning, inference optimization, and real-world performance trade-offs
- How do skip connections in ResNet address the vanishing gradient problem, and why is this critical for training very deep networks?
- Compare YOLO and Faster R-CNN: what are the speed/accuracy trade-offs, and when would you choose one over the other for a production system?
- Explain the Inception module's design philosophy: how does multi-scale parallel processing improve feature representation compared to sequential convolutions?
- What is the EfficientNet scaling strategy (compound scaling), and how does it achieve better accuracy and efficiency than simply increasing depth or width alone?
- How do self-supervised vision models (e.g., contrastive learning) learn useful representations without labeled data, and why is this valuable for real-world applications?
- Walk through the semantic segmentation pipeline: how does a U-Net encoder-decoder architecture preserve spatial information compared to standard classification CNNs?
- Implement a ResNet-50 from scratch (or modify a pre-trained version) and train it on CIFAR-10 or ImageNet subset; visualize residual blocks and analyze gradient flow through skip connections
- Build a YOLO v3 or v4 detector on a custom dataset (e.g., street scenes, products); tune anchor boxes, loss weights, and NMS thresholds; measure mAP and inference speed
- Implement Faster R-CNN using a framework (PyTorch/TensorFlow); train on COCO or Pascal VOC; analyze region proposal quality and compare with YOLO on the same data
- Create a semantic segmentation pipeline using U-Net on a medical imaging or scene understanding dataset; evaluate with IoU and Dice coefficient metrics
- Train an EfficientNet model with different scaling factors (B0–B7); compare accuracy, latency, and model size; profile inference on mobile/edge hardware
- Implement a self-supervised learning approach (SimCLR or momentum contrast) on an unlabeled image dataset; evaluate the learned representations via downstream classification task
- Deploy a trained CNN model for inference: quantize to INT8, apply pruning, measure latency/throughput on CPU/GPU/mobile; document real-world performance trade-offs
Next up: This stage equips you with production-ready CNN architectures and detection/segmentation systems; the next stage will likely deepen into transformer-based vision models (ViT, DETR), video understanding, or domain-specific applications (medical imaging, autonomous driving) that build on these foundational recognition capabilities.
Focuses exclusively on applying deep learning to vision problems — classification, detection, and segmentation — with architecture walkthroughs and training strategies that directly connect to the prior stages.
A rigorous, code-first textbook covering CNNs, attention mechanisms, and vision transformers (ViT) with runnable notebooks — the ideal capstone that connects classical CV intuition to the very latest architectures.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
