Best Books on Design of Experiments, in Order
Design of experiments is the part of statistics that comes before any analysis: how to arrange an experiment so that the data it produces can actually answer the question, and so that the effects you care about are not confounded with the ones you do not. The literature splits sharply between practitioner books, which assume arithmetic and a laboratory, and graduate texts, which assume linear models in matrix form and some probability theory. This path runs the practitioner track first, then the origin texts, then the graduate material on factorial and optimisation designs, and ends with the two settings — online experimentation and field trials in the social sciences — where the classical theory meets very different constraints.
The Ideas, Before the Algebra
BeginnerUnderstand randomisation, replication, blocking and confounding well enough to spot a badly designed experiment, without needing matrix notation.
▸ Study plan for this stage
Pace: 8 to 10 weeks. Statistics for Experimenters is long — around 650 pages in the second edition — but it is a book you work rather than read; six weeks at a chapter a week, doing the arithmetic, is the right pace. Cox's Planning of Experiments is about 300 pages of near-prose from 1958 with very little
- Randomisation, replication, blocking and confounding as four separate devices doing four separate jobs — Box, Hunter and Hunter introduce them operationally, Cox is the one who is precise about what each does and does not buy you.
- The iterative cycle — conjecture, design, run, analyse, redesign — which BHH treat as the actual practice of experimentation and most textbooks quietly drop in favour of one-shot designs.
- Why changing one factor at a time is inefficient and, when factors interact, actively misleading: the factorial argument, made in BHH with worked yields rather than with algebra.
- Main effects and interactions as arithmetic on a table of runs, before any model is written down. A 2^3 factorial's seven effects are seven sums and differences and nothing more.
- The randomisation reference distribution: BHH derive a significance test by physically enumerating the rerandomisations, which shows where the t-distribution comes from and what it is standing in for.
- Blocking as the deliberate confounding of a nuisance factor with something you do not care about, and paired comparison as the smallest case of it.
- Cox's distinction between the units of an experiment and the treatments applied to them, which is the idea Bailey later formalises and which most misdesigned experiments get wrong.
- Cobb's organising move: naming designs by their structure rather than by tradition, so that a Latin square and a randomised block are seen as the same object with different constraints instead of two entries in a catalogue.
- What exactly does randomisation protect against, and what does it not protect against? Give Cox's answer, not the folk one.
- Under what conditions does one-factor-at-a-time experimentation give the wrong answer rather than merely a slow one?
- How would you compute the three main effects and four interactions of a 2^3 factorial from eight yields, using arithmetic alone?
- What is being held constant in a randomised block design, and what is the price paid for holding it?
- In a paired comparison, what is the unit of analysis, and why is the pairing worth the lost degrees of freedom?
- Work the boys' shoes paired-comparison example in Statistics for Experimenters all the way through with the book's own numbers: compute the paired differences, the t statistic, and then construct the randomisation reference distribution by enumerating sign flips. Compare the two p-values yourself.
- Take BHH's pilot-plant 2^3 factorial and compute all seven effects by hand from the eight yields before reading their table. Then check your signs against theirs — sign errors are the single commonest mistake in this material and doing it once by hand fixes it.
- Using Cox's principles chapters, write a one-page critique of a real experiment from your own field or from a recent paper: name its experimental unit, its treatment structure, what it blocked on, and what is confounded with what.
- Take one design from Cobb's structural notation and redraw it as a table of runs. Being able to move between the structural description and the actual run sheet is the skill this stage is for.
Next up: You can now recognise a design and compute its effects by hand; the next stage is the taught curriculum that puts models, ANOVA tables and software behind the same objects.
The readable industrial classic, written by Box, Hunter and Hunter around real process-improvement problems rather than around theorems. Start here: it assumes only basic algebra and teaches the iterative habit — design, run, learn, redesign — that the rest of the field takes for granted.
A short, almost prose-only book from 1958 on the principles of design, with very little computation. Read it second as the conceptual spine — Cox is unusually clear about what randomisation actually buys you and what it does not.
A gentle undergraduate treatment that builds the standard designs from a small number of structural ideas rather than presenting them as a catalogue. Useful bridge between Box and the course texts in the next stage.
The Standard Course
IntermediateWork through the canonical curriculum — completely randomised and blocked designs, Latin squares, factorials, ANOVA — at the level a taught course assumes, with software.
▸ Study plan for this stage
Pace: 12 to 16 weeks, semester pace. Montgomery's Design and Analysis of Experiments runs about 750 pages and is written to be taught over a full course — a chapter a week with the problem sets is the honest rate, and skipping the problems wastes the book. Oehlert's A First Course in Design and Analysis o
- The ANOVA table as a decomposition of sums of squares, and the F test as a ratio of mean squares under a null of no treatment effect — this is the machinery the whole taught course is built on.
- The completely randomised design, the randomised complete block, the Latin and Graeco-Latin square, and the balanced incomplete block, each as a different answer to the question of what you are willing to confound.
- Fixed versus random effects, and the fact that the choice changes which mean square goes in the denominator of the F ratio — the commonest source of wrong answers in applied work.
- Multiple comparisons: why testing all pairs inflates error rate, and what Tukey, Dunnett, Bonferroni and Scheffé each control. Oehlert is markedly more careful about this than Montgomery.
- Model checking as part of the design: residual plots, normal probability plots of effects, and transformation of the response when the variance depends on the mean.
- Split-plot designs and the reason they arise by accident — when one factor is hard to change, you have a split plot whether or not you intended one, and analysing it as a factorial gives the wrong standard errors.
- Nested versus crossed factors, and how to read which you have from the way the experiment was physically run rather than from the data file.
- Lawson's contribution is executable: every design in the syllabus expressed as R code, which turns notation into something you can run and perturb.
- Given an experiment description, can you write down its ANOVA table — sources, degrees of freedom and expected mean squares — before seeing any data?
- When does the denominator of the F statistic stop being the residual mean square, and how do you tell from the design?
- What is the difference between a randomised complete block and a Latin square in terms of what each removes?
- Which multiple-comparison procedure would you use for all pairwise comparisons, and which for comparisons against a single control, and why are they different?
- You ran an experiment where temperature could only be changed once a day. What design do you actually have, and what happens if you analyse it as a full factorial?
- Reproduce Montgomery's single-factor tensile-strength example end to end: build the ANOVA table by hand from the raw observations, then repeat it in R using Lawson's code for the same design and check that every number matches.
- Take Montgomery's 2^4 filtration-rate experiment, make a normal probability plot of the sixteen effects yourself, decide which are real, and refit the reduced model. Then compare your judgement with his.
- Work one full split-plot example from Oehlert twice — once analysed correctly and once deliberately as if it were a completely randomised factorial — and record how much the standard errors and conclusions differ. This is the most expensive mistake in the field and doing it once on purpose immunises you.
- For three designs in Lawson, change one structural feature in the R code — add a block, remove a replicate, unbalance a cell — and record what breaks in the output. Learning what balance is worth is easier by removing it than by reading about it.
- Write out the expected mean squares for a two-factor experiment under all four fixed/random combinations. It is a derivation, it takes an afternoon, and it settles the F-denominator question permanently.
Next up: You have the standard curriculum; the next stage goes back to where randomisation and the factorial came from, and forward to the cleanest modern statement of the same theory.
The most widely assigned textbook in the field and the default reference for engineers. Assumes a prior course in statistics through regression. Note when buying: a solutions manual is published under a nearly identical title, so check that you are getting the textbook.
Covers similar ground to Montgomery with more attention to the underlying model and to multiple-comparison problems. A good second pass, and the one to read if Montgomery's engineering framing does not match your application.
The same syllabus worked entirely in R, design by design. Read it alongside either of the two above — this is where the material stops being notation and becomes something you can run.
The Origin Texts
IntermediateRead where randomisation, blocking and the factorial experiment came from, and see the modern theory stated in its cleanest algebraic form.
▸ Study plan for this stage
Pace: 6 weeks. Fisher's The Design of Experiments is around 250 pages and is short but not easy — the notation is dated, the argument is compressed, and it is a book to read in small daily portions over three weeks. Bailey's Design of Comparative Experiments is about 330 pages and is a rigorous modern tex
- Fisher's argument for randomisation is that it is what makes the significance test valid — the physical act of randomising supplies the probability model, rather than an assumption about the population.
- The lady tasting tea is not an anecdote but a worked example of an exact test: eight cups, four of each, and the null distribution obtained by counting the ways of choosing four from eight.
- Fisher's introduction of the factorial experiment and of confounding as a deliberate design tool, both stated here for the first time in book form.
- The null hypothesis in Fisher's sense — a hypothesis that can be disproved and never proved — which is not the Neyman–Pearson framework most courses actually teach.
- Bailey's reorientation: a design is a pair of structures on the same set of experimental units, the block structure and the treatment structure, and the analysis follows from how the two sit relative to each other.
- Orthogonality restated geometrically as the decomposition of the observation space into mutually orthogonal subspaces, which makes the ANOVA table a projection rather than a bookkeeping convention.
- Hasse diagrams as a way of reading degrees of freedom and expected mean squares off the structure of a design, which is faster and less error-prone than the rules memorised in stage two.
- Read chronologically these two are ninety years apart, but Bailey is doing exactly what Fisher started: deriving the designs that exist from the structure of the problem rather than cataloguing them.
- What, precisely, does Fisher claim randomisation licenses? Why does he think a significance test is invalid without it?
- Work out why the lady tasting tea gives a one-in-seventy probability, and what would change if she were told only that some cups were of each kind.
- What is a block structure in Bailey's sense, and how does it differ from a list of blocking factors?
- How do you read the expected mean squares off a Hasse diagram, and why does that method not depend on remembering the fixed/random rules?
- Where does Fisher's account of the factorial experiment differ from Montgomery's presentation of the same thing?
- Enumerate all seventy ways of selecting four cups from eight and confirm Fisher's exact probability yourself. Then work out the power of the experiment against a lady who is right three times out of four, and see why he chose eight cups.
- Take a randomised block design from stage two, draw its Hasse diagram following Bailey, and derive the degrees of freedom and expected mean squares from the diagram alone. Check them against the ANOVA table you built earlier.
- Work Bailey's treatment of orthogonal projection for a two-factor design on paper: write the design matrix, project onto each subspace, and confirm the sums of squares add. Doing this once turns the ANOVA table from a recipe into a consequence.
- Find a passage where Fisher's notation is opaque, rewrite it in the notation of stage two, and note what he means that the modern version does not say.
Next up: You now know where the theory came from and how it is cleanly stated; the next stage is the graduate material for the case that actually dominates industrial practice — many factors and very few runs.
The 1935 book that invented the subject, including the lady-tasting-tea argument for randomisation. Read it after a modern course, not before — Fisher's notation is dated and his argument is much easier to follow once you know where it ended up.
A rigorous modern account built on the structure of the design — blocks, plots, treatment factors — rather than on a catalogue of named designs. Assumes comfort with linear algebra; the most conceptually satisfying book here for a reader who wants to know why the standard designs are the ones that exist.
Factorials, Response Surfaces and Optimisation
IntermediateHandle the designs used when there are many factors and few runs — fractional factorials, screening, response surface methods and robust design.
▸ Study plan for this stage
Pace: 5 to 6 months, and this is a genuinely graduate stage. Dean and Voss's Design and Analysis of Experiments is around 750 pages and requires the linear model in matrix form; six to eight weeks on the confounding, fractional factorial and incomplete-block chapters. Wu and Hamada's Experiments is compar
- The defining relation and the alias structure: a fractional factorial buys runs by making effects indistinguishable, and the design is chosen by choosing which effects you are willing to confuse.
- Resolution and minimum aberration as the formal criteria for that choice, and why two designs of the same resolution are not equally good.
- Screening designs — Plackett–Burman and the wider class of orthogonal arrays — for the case where there are more factors than runs and only a few are expected to matter.
- Effect sparsity, hierarchy and heredity: the three empirical assumptions the whole fractional-factorial enterprise rests on, which Wu and Hamada state explicitly where most books leave them implicit.
- Response surface methodology as a two-phase procedure: a first-order design and steepest ascent to get near the optimum, then a second-order design — central composite or Box–Behnken — to characterise it.
- Canonical analysis of the fitted second-order model, which turns the estimated surface into a statement about whether you have found a maximum, a minimum or a saddle, and in which directions the response is flat.
- Robust parameter design: the distinction between control factors and noise factors, and the goal of finding settings where the response is insensitive to the noise rather than merely optimal on average.
- The Taguchi dispute in specific terms — his goals (loss functions, designing for variability, quality at the design stage) are broadly accepted; his signal-to-noise ratios and crossed inner/outer arrays are widely held to be statistically inefficient, and Wu and Hamada present the response-model alt
- Given a 2^(7-3) design and its generators, can you write the full defining relation and the alias set of every main effect?
- What does minimum aberration minimise, and why is it preferred to resolution alone as a criterion?
- What do effect sparsity, hierarchy and heredity assert, and what happens to a fractional factorial analysis when hierarchy fails?
- Why does response surface methodology use a first-order design first, and how do you know when to stop moving along the path of steepest ascent?
- What is the statisticians' objection to Taguchi's signal-to-noise ratios, and what is the response-model approach offered in its place?
- In a robust parameter design, why is a control-by-noise interaction the thing you are actually hunting for?
- Take a 2^(7-3) design, write its generators, derive the complete defining relation and the alias structure of all seven main effects by hand, then confirm it in software. Doing this once on paper is what makes aliasing stop being an abstraction.
- Reproduce a full response-surface case study from Response Surface Methodology with the book's own data: fit the first-order model, compute the path of steepest ascent, then fit the second-order model and perform the canonical analysis to classify the stationary point.
- Work a Wu and Hamada robust-design example both ways — once with Taguchi's signal-to-noise ratio and once with a response model for mean and dispersion separately — and compare the recommended factor settings. The disagreement between the two methods is the entire argument, reduced to two answers.
- Take a full 2^5 factorial with real or textbook data, then analyse the half fraction obtained by dropping sixteen runs and see which conclusions survive. That comparison is the price of fractionation, measured on a case you can check.
- Compute Taguchi's quadratic loss for a process with a given tolerance and compare it with a step-function loss at the specification limit. His argument for designing to target rather than to tolerance is entirely contained in that comparison.
Next up: Everything so far assumes a laboratory or a plant where you control the assignment; the final stage is what happens when randomisation is either continuous and nearly free, or expensive and only partly enforceable.
Dean and Voss's graduate text, notable for its thorough treatment of confounding, fractional factorials and incomplete block designs. Requires linear models in matrix form; this is the step up from Montgomery, not a substitute for it.
Wu and Hamada's Experiments: Planning, Analysis, and Optimization, catalogued under the bare title. The standard graduate reference for fractional factorial and robust parameter design, and the most complete modern treatment of screening. Demanding, and the right destination for this stage.
The book-length treatment of fitting and optimising a response surface once screening has identified the factors that matter. Read after Wu, since it assumes the factorial machinery.
Taguchi's own statement of the loss-function and robust-design approach that dominated industrial practice in the 1980s. Read it in this position and read the statisticians' criticism alongside it — Wu and Box both treat Taguchi's goals as sound and his specific designs as inefficient.
Where the Classical Theory Meets Different Constraints
IntermediateSee how experimental design changes when randomisation is cheap and continuous, as online, or expensive and politically constrained, as in the field.
▸ Study plan for this stage
Pace: 6 to 8 weeks. Trustworthy Online Controlled Experiments is about 275 pages, written for practitioners, and reads quickly — two weeks, and the case studies repay slow reading more than the framework chapters do. Gerber and Green's Field Experiments is around 480 pages, is a graduate methods text, and
- Sample ratio mismatch: when the observed split departs from the intended one, the randomisation itself is suspect and no analysis of that experiment is trustworthy. This is the single most useful diagnostic in the online setting and has no analogue in the classical texts.
- Peeking and continuous monitoring inflate the false-positive rate because the decision rule is not the one the fixed-sample test assumes; sequential methods and pre-committed horizons are the fixes.
- Overall Evaluation Criterion — the problem of choosing one metric that is both sensitive enough to move and aligned with the long-term outcome you actually want, which is a design decision rather than an analysis one.
- Network effects and interference break the stable-unit-treatment-value assumption: in a social product, one user's treatment leaks into another user's outcome, which classical randomisation does not address and cluster designs only partly repair.
- Gerber and Green's core distinction between the intention-to-treat effect and the effect on compliers, and the instrumental-variables argument that recovers the latter from the former.
- Noncompliance, attrition and spillover as the three ways a field experiment stops delivering the estimate it was designed for, and the identification assumptions needed to salvage each.
- Blocking and covariate adjustment as precision tools in a setting where you cannot increase the sample: the field literature is far more careful about this than the online literature, because runs are expensive.
- The classical design principles are the same in both settings — randomise, block, replicate — but the failure modes are entirely different, and neither book's failure modes appear in Montgomery.
- How do you test for sample ratio mismatch, and what should happen to an experiment that fails the test?
- Why does stopping an A/B test as soon as it reaches significance inflate the false-positive rate, and by roughly how much?
- What is the difference between the intention-to-treat effect and the complier average causal effect, and what assumptions license estimating the second?
- What does attrition do to a randomised comparison, and under what condition can it be ignored?
- In what specific way does interference between units break the analysis, and what does a cluster-randomised design fix and fail to fix?
- Both books use randomisation for the same reason Fisher did. Where do their designs diverge from anything in Montgomery, and why?
- Take a two-arm split with a stated intended ratio and run the sample-ratio-mismatch chi-square test on real or simulated counts. Then simulate a logging bug that drops one per cent of one arm and confirm the test catches it.
- Simulate a null A/B test and evaluate it repeatedly as data accumulates. Record how often it crosses significance at least once. That number, against the nominal five per cent, is the peeking problem measured on your own machine.
- Work a Gerber and Green compliance example by hand: build the compliance table, compute the intention-to-treat effect, the compliance rate and the complier average causal effect, and state which assumption each step used.
- Take one experiment from Trustworthy Online Controlled Experiments and write out what its design would look like as a classical blocked factorial in Montgomery's terms. Where the translation fails is precisely the content of this stage.
- Design a field experiment for a question in your own domain and write, before any data, the pre-analysis plan Gerber and Green would require: hypotheses, primary outcome, blocking variables, and the rule for handling noncompliance and attrition.
Next up: You have the classical theory, its origins, the graduate machinery and two settings that stretch it; from here the field splits into specialisms — optimal design theory, sequential and adaptive trials, clinical trial methodology and causal inference from observational data — each with its own literature and none of them a prerequisite for the others.
A/B testing at industrial scale, by people who ran it at Microsoft, Google and Airbnb. The design principles are the classical ones, but the failure modes — sample ratio mismatch, peeking, network effects, metric choice — are specific to the setting and barely appear in the textbooks above.
Gerber and Green on randomised trials in the social sciences: noncompliance, attrition, spillover, and what you can identify when the treatment cannot be perfectly delivered. The right book to end on because it is about designing under constraints the industrial literature never faces.
Discussion
Keep reading
Paths that share books, cover the same subject, or open a related topic.
