Natural language processing reading path: from text basics to transformers
This curriculum takes an intermediate developer from classical NLP foundations through modern deep learning-based language systems in four tightly sequenced stages. Each stage builds the vocabulary, intuition, and mathematical scaffolding needed for the next, culminating in a deep understanding of transformer architectures and large language models.
Classical NLP Foundations
IntermediateUnderstand core NLP concepts — tokenization, text normalization, n-gram language models, POS tagging, parsing, and information extraction — using statistical and rule-based approaches before any deep learning.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (alternating between Speech and Language Processing chapters and NLTK hands-on labs)
- Tokenization and sentence segmentation: splitting raw text into meaningful units and handling edge cases (contractions, punctuation, abbreviations)
- Text normalization: lowercasing, stemming, lemmatization, and stop-word removal to prepare text for downstream tasks
- N-gram language models: understanding unigrams, bigrams, and higher-order n-grams for probability estimation and text generation
- Part-of-speech (POS) tagging: recognizing word categories using rule-based taggers and statistical models (HMM, Viterbi algorithm)
- Syntactic parsing: constituency and dependency parsing to extract hierarchical sentence structure using context-free grammars and shift-reduce parsers
- Information extraction: named entity recognition (NER), relation extraction, and semantic role labeling to identify and link key entities and events
- Evaluation metrics: precision, recall, F1-score, and perplexity for assessing NLP system performance
- Statistical foundations: probability, Bayes' theorem, and maximum likelihood estimation as the backbone of classical NLP models
- What are the main challenges in tokenization, and how do rule-based and statistical approaches differ in handling them?
- Explain the difference between stemming and lemmatization, and when you would use each approach in a text preprocessing pipeline.
- How do n-gram language models estimate word probabilities, and what is the role of smoothing techniques in handling unseen n-grams?
- Describe how a Hidden Markov Model (HMM) is applied to POS tagging, including the role of the Viterbi algorithm.
- What are the key differences between constituency parsing and dependency parsing, and what linguistic phenomena does each capture?
- How would you design a named entity recognition system using rule-based patterns and statistical models, and what are the trade-offs?
- Implement a tokenizer from scratch using regular expressions in NLTK (Chapter 3 of NLTK book); test it on text with contractions, hyphens, and abbreviations.
- Build a text normalization pipeline: apply lowercasing, stemming (Porter Stemmer), lemmatization (WordNet), and stop-word removal; compare outputs on a sample corpus.
- Train an n-gram language model on a corpus using NLTK; compute probabilities for unseen bigrams and trigrams; experiment with add-one smoothing and Laplace smoothing.
- Implement a simple POS tagger using NLTK's pre-trained models; manually tag a sentence and compare with the model; analyze error cases.
- Parse 5–10 sentences using both constituency and dependency parsers (NLTK and spaCy); draw parse trees and identify differences in structure.
- Extract named entities from a news article using NLTK's NER module and rule-based patterns; evaluate precision and recall against a gold-standard annotation.
Next up: This stage establishes the statistical and algorithmic foundations of NLP, preparing you to understand why deep learning models (RNNs, Transformers) were developed as solutions to the limitations of classical approaches—such as sparsity, manual feature engineering, and inability to capture long-range dependencies.
The definitive NLP textbook, covering tokenization, n-gram models, tagging, parsing, and semantics with rigorous clarity. Start here to build the shared vocabulary and mental models every subsequent book assumes.
Hands-on companion to classical NLP using the NLTK library; reinforces Jurafsky's theory with practical Python code for tokenization, corpora, and text classification — essential for developers who learn by doing.
Machine Learning for Text
IntermediateApply machine learning to NLP tasks — feature engineering, text classification, sentiment analysis, and sequence labeling — bridging the gap between classical statistics and neural approaches.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day with 2–3 days per week dedicated to hands-on coding exercises
- Text preprocessing and normalization (tokenization, stemming, lemmatization, stop word removal) as the foundation for machine learning pipelines
- Feature engineering for text: bag-of-words, TF-IDF, n-grams, and word embeddings as numerical representations of text
- Text classification workflows: training, validation, and evaluation using classical ML algorithms (Naive Bayes, SVM, logistic regression)
- Sentiment analysis as a practical application of text classification with domain-specific challenges and evaluation metrics
- Sequence labeling and named entity recognition (NER) for extracting structured information from unstructured text
- Model evaluation, hyperparameter tuning, and handling class imbalance in text classification tasks
- Real-world considerations: data quality, scalability, and moving from prototype to production NLP systems
- Bridging classical ML approaches with neural network foundations for NLP
- What are the key preprocessing steps required before applying machine learning to text, and why is each step necessary?
- How do TF-IDF and bag-of-words differ as feature representations, and when would you choose one over the other?
- Walk through the complete pipeline for building a text classifier: from raw text to model evaluation. What are the critical decision points?
- How do you evaluate a sentiment analysis model, and what metrics matter most when classes are imbalanced?
- What is named entity recognition, and how does sequence labeling differ from document-level text classification?
- How would you approach feature engineering and model selection for a new text classification task you've never seen before?
- Build a complete text preprocessing pipeline (tokenization, lowercasing, stop word removal, stemming/lemmatization) on a raw text dataset and compare the impact of each step on downstream model performance
- Implement bag-of-words and TF-IDF vectorization on the same dataset, train classifiers with both, and document the performance differences and computational trade-offs
- Train multiple text classifiers (Naive Bayes, SVM, logistic regression) on a binary classification task, compare their performance using precision, recall, F1-score, and ROC-AUC, and explain why one outperforms others
- Build a sentiment analysis classifier on a movie review or product review dataset, handle class imbalance using techniques like SMOTE or class weights, and evaluate using a confusion matrix and classification report
- Implement a named entity recognition system using sequence labeling (e.g., with conditional random fields or similar approaches covered in the book) and evaluate on a standard NER dataset
- Conduct a hyperparameter tuning experiment using grid search or random search on a text classification model, document the results, and explain how you selected the final hyperparameters
Next up: This stage grounds you in classical machine learning approaches to NLP, giving you the statistical intuition and practical skills needed to understand why neural networks (covered next) are more powerful for capturing complex patterns in text and sequential dependencies.
Deepens feature engineering and ML-based NLP with real-world case studies; reading it second in this stage consolidates sklearn-era techniques before neural methods replace them.
Word Embeddings & Neural NLP
IntermediateMaster distributed word representations (Word2Vec, GloVe, FastText), recurrent architectures (RNNs, LSTMs), and sequence-to-sequence models — the conceptual backbone of modern NLP.
▸ Study plan for this stage
Pace: 8–10 weeks, ~40–50 pages/day (Goldberg first 4–5 weeks, Rao next 4–5 weeks)
- Distributed word representations: why dense vectors outperform sparse one-hot encodings and how they capture semantic relationships
- Word2Vec (Skip-gram and CBOW): training objectives, negative sampling, and the intuition behind predicting context from words or vice versa
- GloVe and FastText: how global co-occurrence statistics and subword information improve upon Word2Vec
- Recurrent Neural Networks (RNNs) and LSTMs: the vanishing gradient problem, gating mechanisms, and why LSTMs enable learning long-range dependencies
- Sequence-to-sequence (seq2seq) models: encoder-decoder architecture, attention mechanisms, and their application to machine translation and other tasks
- Practical implementation in PyTorch: building embeddings, training RNNs/LSTMs from scratch, and composing seq2seq systems
- Evaluation and debugging: measuring embedding quality, interpreting learned representations, and diagnosing training failures in neural models
- Explain the conceptual difference between Word2Vec's Skip-gram and CBOW objectives, and why negative sampling makes training tractable
- What is the vanishing gradient problem in RNNs, and how do LSTM gating mechanisms (input, forget, output gates) solve it?
- Describe the encoder-decoder architecture in seq2seq models and explain why attention is necessary for handling long sequences
- How do GloVe and FastText improve upon Word2Vec, and in what scenarios would you choose one over the others?
- Walk through the forward and backward pass of an LSTM cell, identifying where gradients flow and how gates control information
- Given a PyTorch embedding layer and a sequence of token IDs, how would you feed data through an LSTM and extract meaningful outputs for downstream tasks?
- Implement Word2Vec (Skip-gram with negative sampling) from scratch in NumPy or PyTorch, training on a small corpus and visualizing learned embeddings with t-SNE
- Load pre-trained GloVe or FastText embeddings and perform analogy tasks (e.g., 'king' - 'man' + 'woman' ≈ 'queen') to verify semantic structure
- Build a character-level LSTM language model in PyTorch that generates text, experimenting with hidden size and number of layers to observe quality differences
- Implement a bidirectional LSTM for sequence tagging (e.g., POS tagging or NER) on a small dataset, comparing unidirectional vs. bidirectional performance
- Code a seq2seq model with attention for a toy task (e.g., reversing sequences or simple machine translation), visualizing attention weights to understand what the model learns
- Train an LSTM on a sentiment classification task, then ablate components (remove LSTM gates, use vanilla RNN) and measure performance degradation to understand each mechanism's role
Next up: This stage equips you with the core neural architectures and learned representations that power modern NLP, positioning you to tackle advanced topics like Transformers, BERT, and large language models in the next stage.
The most rigorous treatment of neural NLP before transformers: covers embeddings, feed-forward networks, RNNs, and LSTMs with mathematical depth, making it the essential bridge from ML to deep NLP.
Implements the concepts from Goldberg's book in PyTorch — embeddings, CNNs for text, RNNs, and attention — giving developers hands-on fluency with the tools used in production NLP systems.
Transformers & Modern Language Models
ExpertDeeply understand the transformer architecture, self-attention, BERT, GPT, and the full landscape of large language models, and be able to fine-tune and deploy them for real NLP tasks.
▸ Study plan for this stage
Pace: 12–14 weeks, ~40–50 pages/day with 2–3 days/week for hands-on coding and experimentation
- Transformer architecture fundamentals: multi-head self-attention, positional encoding, feed-forward networks, and layer normalization
- Self-attention mechanism: how queries, keys, and values enable the model to weigh relationships between tokens
- BERT (Bidirectional Encoder Representations from Transformers): pretraining objectives (MLM, NSP), fine-tuning for classification, NER, and token-level tasks
- GPT (Generative Pre-trained Transformer): autoregressive decoding, causal masking, and prompt-based few-shot learning
- Transfer learning and fine-tuning workflows: adapting pretrained models to downstream tasks with limited data
- Large language model scaling: training efficiency, distributed training, quantization, and deployment considerations
- Tokenization strategies and vocabulary design: BPE, WordPiece, and SentencePiece for handling diverse languages
- Practical implementation: building transformers from scratch, using the Hugging Face Transformers library, and evaluating model performance
- Explain the self-attention mechanism: how do queries, keys, and values interact, and why is scaling by √d_k important?
- What are the key differences between BERT and GPT in terms of architecture, pretraining objectives, and use cases?
- How do you fine-tune a pretrained transformer model for a specific downstream task, and what hyperparameters matter most?
- What is positional encoding, why is it necessary in transformers, and how do absolute vs. relative positional encodings differ?
- Describe the full pipeline for training a large language model from scratch, including tokenization, data preparation, and distributed training.
- How do techniques like quantization, knowledge distillation, and prompt engineering help deploy and optimize large language models in production?
- Implement a single-head attention mechanism from scratch in PyTorch, then extend it to multi-head attention; verify outputs match reference implementations
- Build a minimal transformer encoder from scratch (positional encoding, multi-head attention, feed-forward layers) and train it on a toy sequence-to-sequence task
- Fine-tune a pretrained BERT model (using Hugging Face) on a text classification task (e.g., sentiment analysis or topic classification) and evaluate with precision, recall, and F1
- Fine-tune a pretrained GPT-2 or GPT-3 model for text generation on a domain-specific dataset (e.g., product reviews, technical documentation) and generate coherent completions
- Implement tokenization from scratch (BPE or WordPiece) and compare vocabulary coverage and compression ratios against Hugging Face tokenizers
- Train a small transformer language model from scratch on a subset of a real corpus (e.g., WikiText-2 or a domain corpus) using distributed training techniques, then evaluate perplexity
- Deploy a fine-tuned transformer model as a REST API using FastAPI or Flask, and benchmark inference latency and throughput under load
- Experiment with prompt engineering and few-shot learning on a pretrained LLM; document how prompt design affects output quality and consistency
Next up: This stage equips you with deep architectural understanding and hands-on mastery of transformers and LLMs, preparing you to explore specialized applications (domain adaptation, multimodal models, retrieval-augmented generation) and advanced optimization techniques (efficient attention, parameter-efficient fine-tuning, reinforcement learning from human feedback) in the next stage.
The most practical and comprehensive guide to the Hugging Face ecosystem; covers BERT, GPT-2, T5, and beyond with fine-tuning recipes — the ideal first transformer book after mastering neural NLP fundamentals.
Goes broader across transformer variants (RoBERTa, XLNet, GPT-3, vision transformers) and deployment patterns, reinforcing architectural intuition built in the previous book with diverse real-world applications.
Caps the curriculum by walking through implementing a GPT-style LLM from the ground up in PyTorch, cementing deep understanding of every architectural decision — tokenization, attention, pretraining, and fine-tuning.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
