MLOps: a reading path for shipping machine learning models
This curriculum builds a rigorous, end-to-end understanding of MLOps starting from the engineering and ML foundations that underpin production systems, then progressing through deployment and pipelines, and finally tackling advanced operational concerns like monitoring, feature stores, and organizational maturity. Because the learner starts at an intermediate level, early-stage fundamentals are kept lean and practical, with the bulk of the path focused on the real-world engineering challenges of keeping ML systems alive and reliable in production.
Production ML Foundations
IntermediateEstablish a shared vocabulary for ML systems thinking — understanding how ML projects fail in production, what distinguishes software engineering from ML engineering, and the mental models needed to reason about the full lifecycle.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of reading and reflection). Allocate roughly 5–6 weeks for "Designing Machine Learning Systems" (~400 pages) and 2–3 weeks for "Machine Learning Engineering" (~300 pages), with 1 week for integration and review.
- ML systems as sociotechnical artifacts: understanding how business requirements, data, models, and infrastructure interact in production
- The distinction between ML engineering and traditional software engineering: why ML systems are harder to test, debug, and maintain
- Data flywheel and feedback loops: how production data shapes model performance over time and creates compounding risks
- Feature engineering and data quality as the bottleneck: why data problems dominate ML projects, not model complexity
- Model evaluation beyond accuracy: understanding loss functions, metrics, and the gap between offline and online performance
- Deployment and monitoring as first-class concerns: why a model in a notebook is not a system, and how to detect model degradation
- The full ML lifecycle: from problem formulation through labeling, training, evaluation, deployment, and monitoring
- Common failure modes in production ML: concept drift, data drift, feedback loops, and cascading failures in multi-stage systems
- Why is machine learning engineering fundamentally different from traditional software engineering? What are the key sources of non-determinism and technical debt?
- Describe the data flywheel and explain how feedback loops in production systems can degrade model performance over time.
- What is the relationship between feature engineering, data quality, and model performance? Why do data problems typically dominate ML projects?
- How would you design a monitoring and alerting system to detect model degradation in production? What signals would you track?
- Explain the difference between offline evaluation metrics and online performance. Why might a model perform well in validation but fail in production?
- Walk through the full ML lifecycle from problem formulation to monitoring. At which stages do most ML projects fail, and why?
- What are concept drift and data drift? How would you distinguish between them, and what interventions would you apply to each?
- Map a real ML project you know (or a case study from the books) to the full lifecycle stages. Identify where data, model, and infrastructure decisions interact, and where failures are most likely.
- Design a feature engineering and data quality pipeline for a specific problem (e.g., fraud detection, recommendation). Document assumptions, failure modes, and monitoring points.
- Build a simple ML system end-to-end (train, deploy, monitor) using a public dataset. Intentionally introduce data drift or concept drift, then design detection and response mechanisms.
- Write a monitoring specification for a production ML system: define SLOs, metrics, alert thresholds, and escalation procedures. Justify each choice.
- Conduct a post-mortem analysis of a failed ML project (real or hypothetical). Trace the root cause through the lifecycle and propose systemic changes.
- Compare two ML systems (e.g., from case studies in the books) on dimensions like data quality, feedback loops, and deployment strategy. Analyze trade-offs.
Next up: This stage equips you with the mental models and vocabulary to reason about ML systems as integrated wholes; the next stage will dive into the technical implementation details—how to actually build the infrastructure, pipelines, and tools that operationalize these principles.
The single best starting point for intermediate practitioners: it frames the entire ML lifecycle — data, training, deployment, monitoring — as a systems design problem, establishing the vocabulary and mental models used throughout the rest of this curriculum.
Complements Huyen by going deeper into the practical engineering discipline — data collection, feature engineering, training pipelines, and deployment hygiene — before the curriculum moves into dedicated MLOps tooling.
MLOps Core: Pipelines, CI/CD & Deployment
IntermediateUnderstand how to operationalize ML models with reproducible pipelines, version control for data and models, and CI/CD practices that treat ML artifacts as first-class software deliverables.
▸ Study plan for this stage
Pace: 6–8 weeks, ~40–50 pages/day with hands-on labs
- MLOps lifecycle: from experimentation to production monitoring, treating ML as a software engineering discipline
- Reproducible ML pipelines: orchestration tools, dependency management, and containerization (Docker) for consistent model training and deployment
- Data and model versioning: tracking data lineage, model artifacts, and experiment metadata to enable reproducibility and rollback
- CI/CD for ML: automated testing of data quality, model performance, and code; continuous integration and deployment of models
- Infrastructure as Code (IaC): provisioning and managing ML infrastructure reproducibly using tools like Terraform or CloudFormation
- Model serving and deployment patterns: batch prediction, real-time APIs, and monitoring model performance in production
- Collaboration and governance: cross-functional workflows, experiment tracking, and establishing ML governance practices
- What are the key stages of the MLOps lifecycle, and how does each stage differ from traditional software development?
- How do you design a reproducible ML pipeline, and what tools (e.g., Airflow, Kubeflow, DVC) address different orchestration needs?
- What is data versioning, and why is it as critical as code versioning in ML projects?
- How do you implement CI/CD for ML models, including data validation, model testing, and automated deployment?
- What are the trade-offs between batch prediction, real-time APIs, and streaming inference for model serving?
- How do you monitor model performance in production, detect drift, and trigger retraining pipelines?
- Build a reproducible ML pipeline using Airflow or Kubeflow: ingest data, train a model, and log artifacts with versioning
- Set up DVC (Data Version Control) for a dataset and model; create branches to simulate experiment tracking and model rollback
- Containerize an ML training script and inference service using Docker; push to a registry and document the build process
- Write unit tests and integration tests for data preprocessing, feature engineering, and model prediction; integrate into a CI/CD workflow (GitHub Actions, GitLab CI, or Jenkins)
- Implement a simple model serving endpoint (Flask/FastAPI) with input validation and performance logging; deploy to a cloud platform (AWS, GCP, or Azure)
- Create an Infrastructure as Code template (Terraform or CloudFormation) to provision a training environment and model registry
- Set up experiment tracking with MLflow or Weights & Biases; log hyperparameters, metrics, and artifacts; compare runs and promote a model to production
Next up: This stage establishes the operational foundation for ML systems; the next stage will deepen your expertise in advanced deployment patterns, scaling to production workloads, and building resilient, observable systems that handle real-world complexity.
A concise, practitioner-oriented overview of the MLOps landscape — governance, model risk, and the organizational structures around deployment — providing the strategic framing before diving into technical tooling.
Moves from strategy to hands-on implementation: covers CI/CD for ML, cloud deployment patterns, containerization, and automation workflows, making it the core technical reference for this stage.
Data & Feature Infrastructure
IntermediateMaster the data layer of MLOps — building reliable data pipelines, understanding feature stores, and ensuring the data contracts that ML models depend on remain consistent between training and serving.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (mix of theory and code examples)
- Data pipeline architecture: sources, ingestion, transformation, and storage patterns that support both training and serving
- Feature engineering and feature stores: designing reusable features, managing feature lineage, and keeping training/serving features in sync
- Data quality and validation: schema enforcement, data contracts, and testing strategies to catch issues before they reach models
- Scalable data processing: batch vs. streaming trade-offs, orchestration tools, and handling late-arriving or out-of-order data
- Data governance and metadata: tracking data lineage, versioning datasets, and documenting dependencies between pipelines and models
- Practical pipeline implementation: using Apache Beam/Dataflow and TensorFlow Data Validation (TFDV) to build production-grade systems
- What are the key differences between batch and streaming data pipelines, and when should you choose each for an ML system?
- How do feature stores solve the training-serving skew problem, and what are the trade-offs of different feature store architectures?
- What is a data contract, and how does it help prevent silent failures in ML pipelines?
- How would you design a data pipeline to handle late-arriving data, schema changes, and data quality issues without breaking downstream models?
- What role does metadata and data lineage play in debugging a failed ML model in production?
- How do you validate data quality at different stages of a pipeline, and what tools does TensorFlow Data Validation provide?
- Build an end-to-end batch data pipeline using Apache Beam (or Dataflow) that reads from a public dataset, applies transformations, and writes to a data warehouse; document the schema and data contracts
- Design and implement a feature store schema for a realistic use case (e.g., e-commerce or fraud detection); define online and offline feature retrieval paths and test training-serving consistency
- Create a data validation suite using TensorFlow Data Validation (TFDV) that detects schema drift, missing values, and anomalies; run it on synthetic data with intentional quality issues
- Implement a simple streaming pipeline (e.g., using Kafka + Beam or Spark Streaming) that ingests events, aggregates features, and serves them to a model; test behavior under out-of-order data
- Write data lineage documentation for a 3–4 stage pipeline: map inputs, transformations, outputs, and dependencies; identify potential failure points and mitigation strategies
- Refactor a poorly designed pipeline (provided or self-created) to enforce data contracts, add validation checkpoints, and separate concerns (ingestion, transformation, feature engineering)
Next up: This stage equips you with the infrastructure and discipline to reliably produce clean, consistent data for models; the next stage will focus on how to train, validate, and deploy those models with confidence, knowing the data layer is solid.
Provides a rigorous grounding in the data engineering lifecycle — ingestion, storage, transformation, serving — which is the foundation every feature store and training pipeline is built on.
Focuses specifically on TFX-based ML pipelines end-to-end, covering data validation, feature engineering, model analysis, and serving — directly addressing the training-serving skew problem central to MLOps.
Monitoring, Reliability & Production Health
ExpertLearn to detect and respond to model degradation, data drift, and system failures in production — building the observability and reliability practices that keep ML systems trustworthy over time.
▸ Study plan for this stage
Pace: 4–5 weeks, ~40–50 pages/day (mix of dense technical content and case studies)
- Model monitoring and observability: tracking prediction quality, latency, and system health in real time
- Data drift detection: identifying when input distributions shift and impact model performance
- Model degradation and retraining strategies: recognizing performance decay and implementing automated or manual retraining pipelines
- Reliability engineering for ML: designing systems that gracefully handle failures, maintain SLAs, and recover from incidents
- Alerting, logging, and incident response: setting up thresholds, dashboards, and runbooks for production ML systems
- A/B testing and canary deployments: validating model changes safely before full rollout
- System design for scale: architecting monitoring and reliability infrastructure that handles high-volume production traffic
- What are the key differences between monitoring ML systems versus traditional software systems, and why do standard metrics often miss model degradation?
- How do you detect and quantify data drift, and what are the trade-offs between statistical tests and heuristic-based approaches?
- Describe a complete incident response workflow for a model that suddenly drops in accuracy: detection, diagnosis, and remediation steps.
- What SLOs and SLIs are appropriate for an ML system, and how do you balance model performance with infrastructure reliability?
- How would you design a monitoring and alerting system for a high-traffic recommendation model, including what metrics to track and alert thresholds?
- What are the pros and cons of different retraining strategies (scheduled, drift-triggered, performance-triggered), and when would you choose each?
- Set up a monitoring dashboard for a deployed model using Prometheus/Grafana or similar tools; track prediction latency, error rates, and a custom model performance metric.
- Implement a data drift detector using statistical tests (e.g., Kolmogorov-Smirnov, chi-squared) on a real or synthetic dataset; log drift signals and visualize them over time.
- Design and document a complete incident runbook for your team: what to check first, how to diagnose root cause (data vs. code vs. infrastructure), and rollback/mitigation steps.
- Build an A/B test harness that safely routes a percentage of traffic to a new model version and logs performance metrics for both variants; analyze results.
- Create a retraining pipeline that automatically detects performance degradation and triggers a model retrain; include validation gates before deployment.
- Conduct a failure mode analysis (FMEA) on a production ML system: identify 5–8 failure scenarios, their likelihood, impact, and mitigation strategies.
Next up: This stage equips you with the operational and architectural mindset needed to keep ML systems healthy at scale; the next stage will deepen your ability to design entire end-to-end ML systems from scratch, integrating monitoring and reliability as first-class concerns from the start.
Applies SRE principles directly to ML systems — SLOs for models, on-call practices, incident response, and reliability engineering — filling the critical gap between deployment and long-term operational health.
Synthesizes the full production ML stack through the lens of system design, reinforcing monitoring, scalability, and trade-off reasoning — an excellent capstone that stress-tests everything learned in prior stages.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
