Numerical methods is where continuous mathematics meets finite computers, and the subject punishes anyone who ignores either side. Learn only the formulas and you will write code that diverges; learn only the coding and you will not know why. The best reading order keeps both concerns in view from the start.
The arc runs from concrete engineering recipes, through the linear algebra that underlies almost everything, into the differential equations that model the physical world, and finally to the sober study of accuracy itself. Each stage assumes the tools of the last.
Fundamentals and analysis
Start with Numerical methods for engineers, whose worked applications make root-finding, integration, and interpolation tangible for a first pass. Numerical analysis by Burden is the standard rigorous companion, adding the error bounds and convergence proofs that turn recipes into understanding. Together they give you the vocabulary the rest of the field assumes.
The linear algebra core
Almost every large computation reduces to solving linear systems, so this is the heart of the subject. Numerical linear algebra by Trefethen is beloved for its clarity, teaching conditioning, stability, and iterative methods through short, insightful lectures. Matrix computations by Golub is the encyclopedic reference you graduate to, dense with the algorithms that production libraries implement. For the related problem of representing functions well, Approximation Theory and Approximation Practice connects theory to real computation with Chebyshev methods.
Differential equations and finite elements
Modeling the physical world means solving differential equations numerically. Numerical Solution of Ordinary Differential Equations lays out the time-stepping schemes and their stability, while Numerical solution of partial differential equations extends the ideas to space, covering finite-difference approaches. The finite element method by Hughes is the definitive treatment of the technique that dominates engineering simulation. Numerical mathematics by Quarteroni ties these threads into one graduate-level synthesis.
Take errors seriously
The final lesson is humility about precision. Numerical methods for scientists and engineers by Hamming is a wise, opinionated classic on how to think about computation, and Accuracy and stability of numerical algorithms is the deep, careful study of floating-point error that separates practitioners from tourists.
Read these in order and you will not just run the algorithms — you will trust, and distrust, them correctly. Follow the full path to move from first recipes to reliable scientific computing.