Build AI apps with large language models
This curriculum takes an intermediate developer from LLM fundamentals through production-grade AI application engineering. Each stage builds on the last: you first internalize how LLMs work and are prompted, then master the core patterns (RAG, embeddings, agents), then tackle the hardest production concerns — evaluation, reliability, and system design at scale.
Foundations: How LLMs Think and How to Talk to Them
Some backgroundUnderstand transformer-based LLMs well enough to reason about their capabilities and failure modes, and write prompts that reliably shape model behavior.
▸ Study plan for this stage
Pace: 6–8 weeks total. Week 1–4: "Hands-On Large Language Models" by Jay Alammar (~25–35 pages/day, including time to run the book's code notebooks). Week 5–8: "Prompt Engineering for LLMs" by John Berryman (~20–25 pages/day, with dedicated sessions to replicate every prompt example in a live playground o
- Transformer architecture fundamentals: attention mechanisms, tokens, embeddings, and how meaning is encoded in high-dimensional vector space (Alammar, Part I)
- The autoregressive generation loop: how LLMs predict the next token, and why temperature, top-p, and top-k sampling parameters shape output diversity and determinism (Alammar, Ch. 3–4)
- The role of pretraining vs. instruction-tuning vs. RLHF in shaping a model's default behavior and its responsiveness to prompts (Alammar, Ch. 5–6)
- Tokenization edge cases and their practical consequences: why LLMs struggle with character counting, arithmetic, and rare words (Alammar, Ch. 2)
- Prompt anatomy: system messages, user turns, assistant turns, and how chat-formatted models interpret the conversation structure (Berryman, Ch. 1–2)
- Core prompting strategies: zero-shot, few-shot, chain-of-thought, and role prompting — when each works and why (Berryman, Ch. 3–5)
- Failure modes and mitigation: hallucination, sycophancy, prompt injection, and instruction-following brittleness (Berryman, Ch. 6–7)
- Prompt evaluation and iteration: how to treat prompts as code — versioning, regression testing, and systematic comparison across model outputs (Berryman, Ch. 8)
- After reading Alammar: Given a user input, can you trace the full pipeline from raw text → tokenization → embedding → attention layers → logit output → sampled token, and explain what could go wrong at each step?
- Why does raising the temperature parameter increase output creativity but also increase the risk of factual errors, and what does this reveal about how LLMs 'know' things?
- What is the practical difference between a base (pretrained) model and an instruction-tuned model, and why does this matter when choosing which model to call in your app?
- After reading Berryman: Given a task that a model is failing at, how would you systematically diagnose whether the problem is prompt structure, few-shot example quality, missing context, or a fundamental model capability gap?
- How does chain-of-thought prompting exploit the autoregressive nature of LLMs to improve reasoning, and in what scenarios does it fail to help?
- What is prompt injection, how can it undermine an LLM-powered application, and what prompt-level defenses can you put in place?
- Tokenizer exploration (Alammar, Ch. 2): Use the Hugging Face tokenizer for GPT-2 or a similar model to tokenize 20 varied inputs — including emojis, code, non-English text, and misspellings. Record surprising token boundaries and write a one-paragraph explanation of how each edge case could affect a downstream app.
- Sampling parameter lab (Alammar, Ch. 3–4): Using the OpenAI or Hugging Face Inference API, call the same prompt 10 times each at temperature 0, 0.7, and 1.5. Build a small table comparing output variance, coherence, and factual accuracy. Formulate a personal rule of thumb for when to use each setting.
- Attention visualization (Alammar, Part I notebooks): Run the book's BertViz or equivalent attention-head visualization notebook on a sentence of your choice. Identify at least two attention heads that appear to track syntactic or semantic relationships and annotate what you think they are capturing.
- Prompt format stress-test (Berryman, Ch. 1–2): Take one task (e.g., 'extract action items from a meeting transcript') and write it in five different prompt formats — no system message, detailed system message, few-shot with 2 examples, chain-of-thought instruction, and role-assigned persona. Evaluate outputs on 3 test inputs and rank the formats.
- Chain-of-thought vs. direct answer comparison (Berryman, Ch. 4–5): Select 10 multi-step reasoning problems (math word problems, logic puzzles, or policy questions). Prompt the same model with and without 'Let's think step by step.' Record accuracy and identify the category of problems where CoT helps most and least.
- Build a prompt regression suite (Berryman, Ch. 8): For a mini-project of your choice (e.g., a customer-email classifier), write 5 prompt variants and 10 labeled test cases. Script a loop that runs all variants against all test cases, scores outputs, and prints a leaderboard — treating prompt engineering as a software engineering discipline.
Next up: Mastering how LLMs generate text and how prompts shape that generation gives you the mental model needed to understand why retrieval-augmented generation, tool use, and agent architectures are designed the way they are — the next stage builds directly on these foundations to wire LLMs into real application pipelines.
A highly visual, code-first tour of how LLMs work internally — tokenization, attention, embeddings — giving you the mental model every later pattern depends on. Read this first to build vocabulary.
Systematically covers prompt design patterns (zero-shot, few-shot, chain-of-thought) with practical examples; bridges raw model understanding to reliable application behavior.
Core Patterns: Embeddings, RAG, and Chains
Some backgroundDesign and implement retrieval-augmented generation pipelines, work fluently with vector stores and embeddings, and compose multi-step LLM chains.
▸ Study plan for this stage
Pace: 3–4 weeks, ~20–30 pages/day; read Developing Apps with GPT-4 and ChatGPT cover-to-cover, spending extra time on the embeddings and advanced-patterns chapters (typically chapters 4–6). Aim to finish a chapter every 2 days, pausing after each to complete the associated hands-on exercise before moving
- Token limits, prompt structure, and how the ChatGPT/GPT-4 API request-response cycle works at the code level
- Text embeddings: what they are, how OpenAI's embedding models produce vector representations, and why semantic similarity enables retrieval
- Vector stores and similarity search: indexing document chunks, choosing distance metrics (cosine, dot-product), and querying for nearest neighbors
- Retrieval-Augmented Generation (RAG) pipeline: chunking source documents → embedding → storing → retrieving relevant chunks → injecting into a prompt → generating a grounded answer
- Prompt engineering patterns used inside chains: system messages, few-shot examples, and dynamic context injection
- LLM chains and sequential composition: passing the output of one LLM call as the input to the next to accomplish multi-step reasoning tasks
- Memory and conversation history management: summarization buffers, sliding-window context, and when to use each
- Evaluation and cost control: measuring retrieval quality, tracking token usage, and iterating on chunk size and overlap
- Given a corpus of PDF documents, how would you design a complete RAG pipeline using the OpenAI API as demonstrated in the book — from raw text to a grounded answer?
- What trade-offs exist between chunk size and retrieval quality, and how does the book's code illustrate the impact of these choices?
- How do embeddings produced by OpenAI's embedding endpoint differ from the token representations inside GPT-4, and why does that distinction matter for retrieval?
- How does the book implement multi-step LLM chains, and what problem does chaining solve that a single prompt cannot?
- What strategies does the book recommend for managing conversation memory when context windows are limited?
- How can you measure whether your RAG system is retrieving the right chunks, and what levers can you adjust when retrieval quality is poor?
- Build a minimal RAG pipeline from scratch using only the OpenAI Python SDK (no orchestration framework): chunk a 20-page PDF, embed the chunks with text-embedding-ada-002, store vectors in a simple in-memory structure (e.g., NumPy array), retrieve the top-3 chunks by cosine similarity, and feed them into a GPT-4 chat completion.
- Experiment with chunk sizes (128, 256, 512 tokens) and overlaps (0, 50, 100 tokens) on the same document set; log retrieval precision for 10 fixed test questions and plot the results to internalize the trade-off.
- Implement a two-step chain modeled after the book's examples: Step 1 — use GPT-4 to extract structured key facts from a retrieved passage; Step 2 — use those facts as context for a second GPT-4 call that writes a concise user-facing answer. Compare output quality against a single-step prompt.
- Add a conversation memory layer to your RAG chatbot: implement both a sliding-window approach (keep last N messages) and a summarization approach (ask GPT-4 to summarize older turns), then compare coherence over a 15-turn dialogue.
- Swap the in-memory vector store for a lightweight persistent store (e.g., ChromaDB or FAISS), re-run your pipeline, and document the integration points that changed — reinforcing the abstraction boundaries the book describes.
- Conduct a cost and latency audit: instrument your pipeline to log token counts and API latency for every call, then optimize prompt templates and retrieval depth to cut total tokens by at least 20% without degrading answer quality.
Next up: Mastering RAG pipelines and manual chaining with the raw OpenAI API creates the perfect foundation for the next stage, where orchestration frameworks (such as LangChain or LlamaIndex) are introduced as higher-level abstractions that automate and extend exactly the patterns you have just built by hand.
A concise, practical guide to building real applications with the OpenAI API — covers embeddings, fine-tuning hooks, and chaining calls; ideal first project-oriented book.
Agents and Advanced Orchestration
Going deepBuild autonomous, tool-using agents; understand planning loops, memory architectures, and multi-agent coordination patterns.
▸ Study plan for this stage
Pace: 10–13 weeks total. Weeks 1–6: "AI Engineering" by Chip Huyen (~25–35 pages/day, focusing on chapters covering LLM APIs, tool use, agents, and orchestration pipelines). Weeks 7–13: "Multiagent Systems" by Yoav Shoham (~20–25 pages/day, a denser academic text — slow down for game-theoretic and coordin
- Agent loops and the Reason-Act (ReAct) pattern: how LLMs alternate between reasoning traces and tool calls to complete multi-step tasks, as detailed in Huyen's agent architecture chapters
- Tool use and function calling: defining, registering, and safely invoking external tools (APIs, code interpreters, retrieval systems) from within an LLM agent, per Huyen's coverage of LLM APIs and augmentation
- Memory architectures: distinguishing in-context (short-term) memory, external vector-store memory, and episodic/procedural memory, and knowing when to use each — a theme Huyen addresses in the context of stateful agents
- Planning and task decomposition: how agents break a high-level goal into sub-tasks, select tools, and recover from failures, including chain-of-thought and tree-of-thought planning strategies covered in AI Engineering
- Multi-agent coordination patterns: Shoham's framework for how autonomous agents communicate, negotiate, and allocate tasks — including contract-net, auction-based, and blackboard architectures
- Game-theoretic foundations of agent interaction: Nash equilibria, dominant strategies, and mechanism design as covered in Multiagent Systems, and why they matter for designing reliable agent collectives
- Communication languages and protocols: FIPA-ACL style speech acts, ontologies, and message-passing semantics that Shoham uses to formalize inter-agent dialogue
- Emergent behavior and safety in multi-agent systems: how local agent rules produce global outcomes, alignment challenges, and failure modes that arise specifically in orchestrated agent networks
- After reading Huyen, can you trace the full execution loop of a tool-using agent — from user prompt through reasoning, tool selection, tool execution, observation, and final response — and identify where failures most commonly occur?
- How does Huyen distinguish between single-agent orchestration (one LLM driving a pipeline) and multi-agent orchestration (multiple specialized LLMs collaborating), and what are the engineering trade-offs of each?
- Drawing on Shoham, what is the difference between a cooperative multi-agent system and a competitive one, and how does mechanism design help align self-interested agents toward a socially desirable outcome?
- How do the memory architectures described in AI Engineering (in-context, retrieval-augmented, external storage) map onto the knowledge-representation concepts Shoham discusses for individual agents?
- What communication primitives does Shoham define for agent interaction, and how would you implement an analogous message-passing protocol between two LLM-based agents in a system described by Huyen?
- What are the key failure modes — hallucinated tool calls, infinite planning loops, coordination deadlocks — that emerge from combining the engineering patterns in Huyen with the multi-agent dynamics in Shoham, and how would you mitigate them?
- Build a ReAct agent from scratch using only an LLM API (no agent framework): implement the thought→action→observation loop manually, wire up at least three tools (web search, calculator, file reader), and log every step to make the reasoning trace inspectable — directly applying Huyen's agent architecture guidance.
- Implement a two-tier multi-agent system: an 'orchestrator' LLM that decomposes a complex research task and a pool of 'worker' LLMs each with a specialized tool. Measure task completion rate and latency vs. a single-agent baseline, reflecting the coordination patterns from both Huyen and Shoham.
- Design and run a simulated auction among three LLM agents for resource allocation (e.g., which agent handles which sub-task). Implement a simple Vickrey (second-price) auction mechanism from Shoham's mechanism-design chapter and verify that truthful bidding is the dominant strategy.
- Build a persistent memory layer for an agent: store episodic summaries in a vector database, retrieve relevant past episodes at each planning step, and demonstrate that the agent solves a multi-session task it could not solve with in-context memory alone — grounding Huyen's memory architecture discussion.
- Conduct a failure-mode audit: deliberately introduce a broken tool, a misleading tool response, and a circular sub-task dependency into your agent system, document how the agent behaves in each case, and implement a guard (timeout, retry limit, or verifier LLM) to handle each failure.
- Write a two-page design document for a production multi-agent system of your choice (e.g., a coding assistant team, a research pipeline). Specify agent roles, communication protocol, memory strategy, and safety constraints, citing specific patterns from Huyen's engineering chapters and Shoham's coordination frameworks.
Next up: Mastering autonomous agent loops, tool use, and multi-agent coordination here provides the architectural intuition needed to tackle the next stage's focus on evaluation, safety, and deployment — because you can only rigorously evaluate and harden systems whose internal orchestration mechanics you fully understand.
Covers the full stack of production AI systems including agentic architectures, tool use, and orchestration trade-offs; written by a practitioner for practitioners and is the field's emerging canonical reference.
Provides the theoretical grounding for agent reasoning, coordination, and game-theoretic interaction that underpins modern multi-agent LLM frameworks — read after hands-on agent work to deepen intuition.
Evaluation, Reliability, and Production
Going deepSystematically evaluate LLM applications, detect and mitigate hallucinations, instrument pipelines for observability, and operate LLM apps reliably in production.
▸ Study plan for this stage
Pace: 5–6 weeks, ~25–35 pages/day; read chapters sequentially but budget extra time for Chapters 6 (model evaluation), 8 (data distribution shifts), and 9 (continual learning), which are the densest and most directly applicable to LLM production systems.
- Evaluation metrics beyond accuracy: calibration, slice-based evaluation, behavioral testing, and why aggregate metrics hide failure modes in LLM apps
- Data distribution shift (covariate, label, and concept drift) and how it manifests in LLM pipelines when user intent or world knowledge evolves
- Hallucination as a reliability failure: framing confabulation as a model confidence/calibration problem and designing detection layers around it
- Observability and monitoring: logging predictions, tracking input/output distributions, setting up alerting thresholds, and distinguishing software bugs from model degradation
- Human-in-the-loop feedback loops: collecting production labels, active learning, and using real user signals to drive continual improvement
- Continual learning and model refresh strategies: when to retrain, shadow deployments, canary releases, and rollback policies for LLM-backed services
- Infrastructure for reliable ML systems: feature/prompt stores, pipeline orchestration, versioning of prompts and models, and reproducibility requirements
- Business and stakeholder alignment: translating model quality metrics into product KPIs and building internal trust through explainability and auditability
- How would you design a slice-based evaluation suite for an LLM application to surface failure modes that aggregate benchmark scores would miss?
- What are the three types of data distribution shift described in Designing Machine Learning Systems, and how would each manifest specifically in an LLM-powered product (e.g., a customer-support chatbot)?
- What monitoring signals would you instrument in an LLM pipeline to distinguish a model quality regression from an upstream data pipeline failure?
- How does Chip Huyen's framework for continual learning apply to prompt versioning and model fine-tuning cycles — what triggers a refresh, and how do you validate it before full rollout?
- What strategies does the book recommend for handling natural labels and delayed feedback, and how would you adapt them to collect implicit quality signals from LLM app users?
- How would you construct a canary or shadow deployment for an LLM service, and what metrics would you monitor during the rollout window before promoting the new version?
- Audit an existing LLM app (or build a minimal one) and define at least five evaluation slices (e.g., by query length, topic domain, user cohort); run your current model against each slice and document where quality drops.
- Implement a lightweight hallucination-detection layer: log model outputs, attach a confidence proxy (e.g., self-consistency sampling across 5 runs), and flag responses where answer variance exceeds a threshold.
- Set up an observability dashboard (using any open-source tool such as Prometheus + Grafana, or LangSmith) that tracks input token distribution, output length, latency p50/p95, and an LLM-as-judge quality score over time.
- Simulate a concept-drift scenario: take a fixed prompt template, inject 'drifted' user queries (changed terminology or new topics), measure performance degradation, and write a drift-detection rule that would have caught it.
- Design a human feedback collection pipeline: define what a 'label' means for your app (thumbs up/down, correction, rating), instrument the UI to capture it, store it in a structured log, and write a script that converts it into a fine-tuning or few-shot example dataset.
- Write a production readiness checklist for an LLM service, drawing directly from Designing Machine Learning Systems chapters on deployment and monitoring; include sections on rollback criteria, alerting thresholds, prompt/model versioning, and incident runbooks.
Next up: Mastering evaluation rigor, drift detection, and production observability from Designing Machine Learning Systems gives the reader the operational mindset needed to tackle advanced topics such as LLM security, alignment, and responsible scaling — where the same systematic thinking must be applied to safety, fairness, and adversarial robustness at production scale.
The gold-standard reference for ML system design: data pipelines, monitoring, drift detection, and deployment patterns that apply directly to LLM apps in production. Read last to tie everything together.