High-performance computing is two subjects wearing one name. One is numerical: how you turn a continuous problem into arithmetic a machine can do, and how much of the answer survives. The other is architectural: how you keep thousands of cores fed. People who learn them in the wrong order get stuck in a predictable way — they can distribute a computation across a cluster and cannot tell whether the result means anything.
The order that works starts with numerics you can run on a laptop, adds parallelism only once you have something worth parallelising, and finishes with the two hard topics almost everyone skips: memory-hierarchy behaviour and floating-point error analysis.
Start with computation you can see
Computational Physics by Mark Newman is the friendliest possible entry, and it is useful even if you are not a physicist. It teaches numerical integration, root finding, ODE solvers and Monte Carlo methods in Python, with problems whose answers you can plot and sanity-check. The point of starting here is calibration: you learn what a discretisation error looks like before anyone asks you to hide it behind a scheduler.
Introduction to high performance computing for scientists and engineers by Georg Hager and Gerhard Wellein is the single best second book in the field, and it is the one that reorders most people's mental model. Its argument is that almost all real scientific codes are memory-bandwidth bound rather than compute bound, and it teaches you to reason about that with simple performance models before writing any parallel code. Read it slowly.
Learn parallelism, then pick your model
Introduction to Parallel Programming by Peter Pacheco is the standard teaching text and covers MPI, OpenMP and Pthreads in one volume with small, complete examples. For most readers it is enough.
Using MPI and Using OpenMP are the reference treatments of the two dominant models, written largely by the people who standardised them. These two overlap heavily with Pacheco and with each other, and you almost certainly do not need all three. Read Pacheco first; then read Using MPI only if you are actually writing distributed-memory code across nodes, and Using OpenMP only if you are working within a single shared-memory machine. Buying both before you know which world you live in is the most common wasted purchase in this subject.
The numerical core
Scientific Computing by Michael T. Heath is the survey that ties the numerics together — linear systems, eigenvalues, interpolation, quadrature, optimisation — at a level a working scientist can absorb. NUMERICAL RECIPES IN C (the catalogue stores that all-capitals form) is the famous practical companion, and it deserves an honest note: numerical analysts have criticised several of its algorithm choices and its error handling for decades, and its licence is unusually restrictive about reusing the printed code. It remains a superb explanatory book. Treat it as a place to understand methods, not as a library to copy from.
Numerical linear algebra by Trefethen and Bau is short, elegant and the best introduction to the subject anyone has written; Matrix computations by Golub and Van Loan is the encyclopaedic reference behind it. These substantially overlap. Read Trefethen and Bau cover to cover, and keep Golub and Van Loan on the shelf to consult rather than to read.
Hardware, and then the truth about your answers
Structured Parallel Programming by McCool, Reinders and Robison teaches patterns — map, reduce, scan, stencil — which is how experienced people actually think about parallel decomposition. Programming Massively Parallel Processors by Kirk and Hwu is the standard GPU text; note that a field this close to hardware dates quickly, so check which edition you have against the hardware you are using.
Finish with Accuracy and stability of numerical algorithms by Nicholas Higham. It is a demanding book and the right last one, because it answers the question every earlier book postponed: given finite precision, how wrong is this, and why. Work through the full reading path stage by stage rather than jumping to the parallel books first.
Follow the full ordered path here: How to Learn High-Performance and Scientific Computing from Books, in Order.
