Operating systems are where every hard idea in systems programming meets at once: virtual memory, scheduling, concurrency, file systems, and the hardware underneath. Open a kernel source tree too early and it reads like noise, because you are trying to learn the concepts and their real-world implementation in the same breath.
The reliable order separates those two jobs. Learn the concepts from a text built for teaching, drill concurrency until it is second nature, and only then open a production kernel where those concepts appear tangled together with decades of practical detail. Each step makes the next one legible.
Start with the concepts
Begin with Operating Systems: Three Easy Pieces, the free and widely loved text organized around three pillars — virtualization, concurrency, and persistence — that make the whole subject click. Modern Operating Systems is the natural companion and reference, broader in scope and strong on the design trade-offs behind real systems. Between them you get both an approachable narrative and an authoritative survey, which is enough grounding to reason about any OS you later encounter.
Master concurrency
Concurrency is the topic that trips up the most people, and it deserves dedicated reading. The art of multiprocessor programming is the definitive treatment of synchronization, locks, and lock-free structures, explaining not just how to write concurrent code but why it is correct. Programming with POSIX threads is the practical counterpart, a careful guide to threads, mutexes, and condition variables at the level you actually program against on Unix systems. Working through both turns race conditions from mysteries into things you can reason about.
Read a real kernel
The payoff is understanding a kernel that ships. Linux Kernel Development is the friendliest entry point, explaining the design and subsystems of Linux without drowning you in source. Understanding The Linux Kernel goes deeper into how memory management, scheduling, and I/O actually work internally. For a contrasting and famously well-documented design, The design and implementation of the FreeBSD operating system shows how a different mature Unix makes its own choices, which sharpens your sense of what is essential versus incidental.
Read in this order and an operating system stops being an intimidating monolith and becomes a set of comprehensible layers. Follow the full path to go from the core concepts to reading and reasoning about a production kernel.