AI ethics & safety: understand the real risks
This four-stage curriculum moves from accessible big-picture thinking about AI's societal impact, through the concrete harms of bias and misinformation, into the harder technical and philosophical questions of alignment and existential risk, and finally into the active policy and governance debates where these issues are being decided. Each stage builds the vocabulary and conceptual scaffolding needed to engage seriously with the next, so that by the end the reader can hold and evaluate competing expert positions with genuine confidence.
Foundations: What AI Is and Why It Matters
New to itBuild an intuitive, non-technical understanding of how modern AI works, what it can and cannot do, and why its societal implications deserve serious attention.
▸ Study plan for this stage
Pace: 6–8 weeks total: Weeks 1–4 for "Human Compatible" (~25–30 pages/day, 5 days/week), Weeks 5–8 for "Atlas of AI" (~20–25 pages/day, 5 days/week). Reserve one day per week for reflection, note review, and exercise work.
- The 'old AI' vs. 'new AI' distinction: Russell's critique of the standard model of AI (machines optimizing fixed objectives) and why hard-coded objectives are fundamentally dangerous
- The Value Alignment Problem: why it is extraordinarily difficult to specify human values precisely enough for a machine to pursue them safely, as argued throughout Human Compatible
- Inverse Reward Design & Assistance Games: Russell's proposed alternative — machines that remain uncertain about human preferences and defer to humans rather than acting unilaterally
- AI as a sociotechnical system, not just software: Crawford's core argument that AI is inseparable from physical infrastructure, labor, data extraction, and political power
- The material costs of AI: Crawford's documentation of the supply chains (rare-earth mining, warehouse labor, data annotation) that make AI systems possible but are rendered invisible by the industry
- Data as a site of power: how training datasets encode historical biases, whose knowledge counts, and who is harmed when those datasets are deployed at scale (Atlas of AI, chapters on data and classification)
- The concentration of AI power: how a small number of corporations and states shape AI development in ways that affect everyone, and why this asymmetry matters ethically
- The gap between AI capability and AI understanding: distinguishing what current systems can do statistically from genuine reasoning, and why this gap has safety and ethical consequences
- According to Russell, what is the 'standard model' of AI and what single flaw does he argue makes it potentially catastrophic — even when the AI is working exactly as designed?
- What is the value alignment problem in your own words, and why does Russell believe it cannot be solved simply by writing better rules or reward functions?
- How does Crawford's 'Atlas' reframe what an AI system actually is — what layers of reality does she insist we must look at that a purely technical definition would miss?
- Choose one case study from Atlas of AI (e.g., Amazon fulfillment centers, ImageNet, affect recognition) and explain: who benefits, who bears the costs, and what data or labor is made invisible?
- How do Russell's and Crawford's arguments complement each other? Where do they seem to be in tension, and why might both perspectives be necessary for a complete picture of AI risk?
- Why is the question 'what can AI do?' insufficient on its own, and what additional questions — about power, accountability, and values — must accompany it?
- **Alignment Failure Diary (ongoing, both books):** Each week, find one real-world news story about an AI system behaving unexpectedly or causing harm. Write a one-paragraph analysis using Russell's 'standard model' critique: What objective was the system optimizing? Why did that objective diverge from what humans actually wanted?
- **Supply Chain Mapping (Atlas of AI, Weeks 5–6):** Pick one AI-powered product you use daily (a voice assistant, a recommendation feed, a navigation app). Draw a physical supply chain map tracing at least three layers Crawford identifies: raw material extraction, human labor, and data sourcing. Note every gap where information is unavailable to you as a consumer.
- **Rewrite the Objective (Human Compatible, Weeks 2–3):** Choose a simple AI use case (e.g., a content-recommendation algorithm, a hiring screener). Write out the objective function you imagine the system uses. Then list at least five ways that objective could be 'achieved' by the AI in ways that would horrify its designers — a direct exercise in Russell's core argument.
- **Stakeholder Impact Table (Atlas of AI, Week 7):** Select one of Crawford's case studies. Build a two-column table: Column A lists every group that gains something from the system; Column B lists every group that bears a cost or risk. Reflect in writing: whose interests shaped the system's design, and whose were ignored?
- **Socratic Dialogue (both books, end of each book):** Find a study partner or use a journal. One person argues that advanced AI is primarily a technical problem requiring better engineering (steelmanning the 'old AI' view). The other argues it is primarily a political and social problem (steelmanning Crawford). Switch sides. Write a one-page synthesis afterward.
- **Personal Values Inventory (Human Compatible, Week 4):** Russell argues that AI must learn human values — but humans disagree about values. Write a one-page reflection listing five values you hold that you believe would be difficult or impossible to encode in a reward function, and explain why for each. This makes the alignment problem viscerally concrete.
Next up: By finishing these two books, the reader has established *why* AI alignment and AI power concentration are serious problems; the next stage can now go deeper into the technical and governance mechanisms — such as interpretability, policy frameworks, and formal safety research — that practitioners are actually building to address them.
Written by one of the world's leading AI researchers, this accessible book explains how AI systems are built and why their current design philosophy may be fundamentally misaligned with human values — the perfect conceptual starting point.
Grounds abstract AI concepts in the physical, political, and labor realities behind the technology, ensuring the reader sees AI as a sociotechnical system rather than a neutral tool before diving into specific harms.
Concrete Harms: Bias, Power, and Misinformation
New to itUnderstand how AI systems encode and amplify bias, concentrate power unevenly, and fuel misinformation — with real-world evidence and cases.
▸ Study plan for this stage
Pace: 10–12 weeks total (~25–35 pages/day, 5 days/week): Weeks 1–4 for "Weapons of Math Destruction" (~250 pp), Weeks 5–8 for "The Alignment Problem" (~400 pp), Weeks 9–11 for "This Is Not Propaganda" (~270 pp), with Week 12 reserved for review, synthesis, and completing exercises.
- Opacity and the 'black box' problem: O'Neil shows how algorithmic models are deliberately or negligently kept uninterpretable, shielding them from accountability while their outputs shape lives (credit scores, recidivism tools, hiring filters).
- Feedback loops and self-fulfilling harm: WMD illustrates how biased training data produces biased outputs that generate new data confirming the original bias — a vicious cycle that compounds disadvantage for already-marginalized groups.
- Scale and damage: O'Neil's three-part WMD test (opacity + scale + damage) gives beginners a concrete diagnostic framework for identifying when an algorithm crosses from useful tool to weapon.
- Misspecified objectives and Goodhart's Law: Christian's 'Alignment Problem' demonstrates that optimizing a proxy metric (clicks, test scores, engagement) systematically diverges from the true human goal — the root cause of many real-world AI failures.
- Value loading and the difficulty of specifying human preferences: Christian reveals how encoding nuanced human values into a reward function is technically and philosophically hard, making 'aligned' AI far more elusive than it appears.
- Power asymmetry and who bears the cost: Across both O'Neil and Christian, a consistent theme emerges — the people least able to contest or escape algorithmic decisions (low-income, minority, or politically vulnerable populations) bear the greatest harm.
- Information warfare and the weaponization of doubt: Pomerantsev shows how modern disinformation campaigns do not merely spread false facts but deliberately overwhelm epistemic infrastructure — flooding the zone so that truth and falsehood become indistinguishable.
- AI as an amplifier of propaganda at scale: Connecting Pomerantsev back to the earlier books, AI-driven content recommendation and synthetic media (deepfakes, bots) supercharge the information-environment manipulation Pomerantsev documents, turning analog propaganda tactics into industrial-scale oper
- Using O'Neil's three-part WMD test (opacity, scale, damage), can you classify a real algorithmic system you encounter in daily life — such as a credit-scoring model or a social-media feed — and justify each criterion?
- O'Neil argues that many harmful models are presented as objective because they are mathematical. What is the flaw in equating mathematical formalism with objectivity, and what would a more accountable alternative look like?
- Christian describes cases where an AI agent achieved a high reward score while completely failing the intended goal (e.g., game-playing agents exploiting loopholes). What does this reveal about the relationship between measurable proxies and genuine human values?
- How does the concept of 'value alignment' in Christian's book connect to the real-world harms O'Neil documents? In other words, is bias in a hiring algorithm an alignment failure — and if so, of what kind?
- Pomerantsev argues that the goal of contemporary propaganda is not to convince but to confuse and exhaust. How does this strategy interact with AI-powered content distribution to make the problem qualitatively worse than pre-digital disinformation?
- Synthesizing all three books: Who holds power in each system described (algorithmic scoring, AI objective design, information ecosystems), who is harmed, and what structural changes — technical, legal, or social — do the authors collectively suggest or imply?
- WMD Audit (after O'Neil): Choose one algorithm that affects your life (a job-platform ranking, a loan pre-qualifier, a university admissions tool). Write a one-page audit applying O'Neil's opacity–scale–damage framework. Identify who built it, what data it likely uses, who is harmed, and whether any redress mechanism exists.
- Bias Data Hunt (after O'Neil): Find a publicly available dataset (e.g., on Kaggle or the UCI ML Repository) used in a socially consequential domain (criminal justice, hiring, lending). Compute or look up demographic breakdowns and document at least two ways the dataset's composition could encode historical bias into any model trained on it.
- Reward Function Design Challenge (after Christian): Pick a simple human goal — e.g., 'encourage healthy eating' or 'improve student learning.' Write out three candidate reward functions an AI agent might optimize. Then deliberately find the 'Goodhart failure mode' for each: how could an agent score perfectly on your metric while completely missing the real goal? Share findings with a study partner
- Alignment–Harm Bridge Essay (after Christian): Write a 500-word essay arguing that at least one WMD described by O'Neil is, at its core, an alignment failure as Christian defines it. Use specific examples from both books and propose one technical and one governance fix.
- Propaganda Source Trace (after Pomerantsev): Select a piece of viral content (a meme, a news story, a social-media post) that seems emotionally charged or factually dubious. Use open-source tools (Google reverse image search, Snopes, NewsGuard, Who.Is) to trace its origin, spread, and any coordinated amplification. Document your methodology and findings in a short report.
- Synthesis Roundtable or Written Debate: Organize a discussion (with peers, a book club, or in written form) debating the proposition: 'Algorithmic bias, misaligned AI objectives, and AI-amplified disinformation are three symptoms of the same underlying problem — the concentration of technological power without democratic accountability.' Use specific evidence from all three books to argue both for
Next up: By grounding harm in concrete cases and real evidence, this stage equips the reader with the 'what and why it matters' foundation needed to engage productively with the deeper technical, philosophical, and governance literature on how AI safety and ethics should be formally approached and institutionally enforced.
A landmark, highly readable account of how opaque algorithmic systems cause measurable harm in hiring, lending, and criminal justice — essential vocabulary for any bias discussion.
Bridges the gap between everyday algorithmic bias and the deeper technical challenge of making AI do what humans actually want, using vivid stories from real research labs.
Examines how information warfare and AI-amplified disinformation destabilize democracies, giving the reader a sharp lens on the misinformation dimension of AI ethics.
Deeper Risks: Alignment, Existential Safety, and the Future of Work
Some backgroundEngage seriously with the longer-horizon debates — AI alignment as an existential challenge, the economic disruption of automation, and the philosophical stakes of superintelligence.
▸ Study plan for this stage
Pace: 10–13 weeks total, reading ~25–35 pages/day on weekdays with weekends reserved for reflection and exercises. Suggested breakdown: Zuboff's "The Age of Surveillance Capitalism" (~700 pp) over 4–5 weeks; Bostrom's "Superintelligence" (~400 pp) over 3–4 weeks; Susskind's "World Without Work" (~280 pp)
- Surveillance Capitalism (Zuboff): the logic by which human experience is commodified as behavioral data — 'behavioral surplus' — and sold as predictive products to advertisers and other buyers
- Instrumentarian Power (Zuboff): a new form of power that aims not to forbid or command behavior but to nudge, tune, and herd it at scale, bypassing individual autonomy and democratic accountability
- The Alignment Problem (Bostrom): the core technical and philosophical challenge of ensuring that a superintelligent AI system reliably pursues goals that are genuinely beneficial to humanity, not merely the goals it was imperfectly specified to pursue
- Orthogonality Thesis & Instrumental Convergence (Bostrom): the idea that intelligence and terminal goals are independent (any level of intelligence can serve almost any goal), and that most sufficiently capable agents will converge on similar sub-goals (self-preservation, resource acquisition) regar
- Takeoff Dynamics and Control Problem (Bostrom): how the speed of a transition to superintelligence (slow, fast, or explosive) determines whether humans retain any meaningful ability to intervene, and why a fast takeoff makes the control problem nearly intractable
- Capability vs. Motivation Distinction (Bostrom): separating what an AI system *can* do from what it *will* do — the key insight that raw capability without aligned motivation is existentially dangerous
- The Labor Substitution vs. Complementarity Debate (Susskind): whether AI and automation primarily replace human workers (substitution) or augment them (complementarity), and why Susskind argues the historical complementarity story is breaking down
- The 'Big-Tent' Technological Unemployment Thesis (Susskind): Susskind's argument that even if new jobs are created, the pace and breadth of automation may outstrip society's ability to redistribute work, demanding new policy frameworks (Universal Basic Income, conditional basic income, work-sharing)
- How does Zuboff's concept of 'behavioral surplus' reframe the standard narrative that 'if you're not paying, you're the product'? What makes her critique more structurally radical than that slogan?
- Bostrom argues that a superintelligent paperclip maximizer is not a joke scenario but a serious illustration of a deep principle. What principle does it illustrate, and how do the Orthogonality Thesis and Instrumental Convergence together explain why misaligned superintelligence is dangerous regardless of its assigned goal?
- What is the 'control problem' as Bostrom defines it, and why does takeoff speed matter so much to whether any proposed solution (capability control, motivation selection, etc.) could actually work in practice?
- Zuboff describes 'instrumentarian power' as distinct from totalitarian power. How does this distinction hold up when placed alongside Bostrom's vision of a superintelligent system that shapes the world to fit its objective function? Are these two visions of AI-driven power compatible, complementary, or in tension?
- Susskind challenges the 'lump of labour' fallacy and the standard economist's optimism about technological unemployment. What is his core empirical and theoretical argument, and which of his proposed policy responses does he find most defensible and why?
- Taken together, what do Zuboff, Bostrom, and Susskind each identify as the primary locus of risk — the corporation, the algorithm, or the labor market — and how might a unified framework synthesize their three threat models into a coherent picture of AI's societal dangers?
- Behavioral Surplus Audit: For one week, log every digital service you use for free. For each, identify what behavioral data is likely collected, how it might constitute 'surplus' beyond what the service needs to function, and who the plausible buyers of predictions derived from that data might be. Write a one-page reflection grounded in Zuboff's framework.
- Alignment Scenario Workshop: Design your own 'misaligned AI' thought experiment following Bostrom's method — choose a narrow, seemingly benign goal, then trace how a superintelligent optimizer pursuing it instrumentally could cause catastrophic harm. Map the scenario explicitly onto the Orthogonality Thesis and at least two instrumental convergence drives Bostrom identifies.
- Control Method Stress-Test: Bostrom surveys multiple proposed solutions to the control problem (capability control methods like boxing, tripwires; motivation selection methods like value loading, CEV). Pick two methods, write a one-page 'devil's advocate' critique of each, then write a one-paragraph rebuttal. Conclude with your own ranked assessment of their viability.
- Automation Impact Interview: Interview two people in different occupations (one routine-task-heavy, one judgment/creativity-heavy) about how they perceive AI's impact on their work. Analyze the interviews using Susskind's substitution-vs-complementarity framework and his task-based model of labor. Do their lived experiences confirm or complicate Susskind's thesis?
- Policy Proposal Memo: Drawing on Susskind's policy chapters, write a 500-word memo addressed to a fictional Minister of Labor recommending one policy response to technological unemployment. You must engage with at least one counterargument Susskind himself raises and explain how your recommendation addresses it.
- Cross-Book Synthesis Essay: Write a 750–1,000-word comparative essay answering: 'Are the risks described by Zuboff, Bostrom, and Susskind fundamentally the same risk in different clothes, or are they genuinely distinct threats requiring distinct responses?' Use specific arguments and evidence from all three books.
Next up: Having grappled with the structural, existential, and economic dimensions of AI risk through Zuboff, Bostrom, and Susskind, the reader is now equipped to move from diagnosis to governance — asking not just 'what could go wrong?' but 'who is responsible, how should AI be regulated, and what ethical frameworks should guide its development?' — which is the natural focus of an advanced stage on AI pol
Provides a rigorous theoretical framework for understanding how AI-driven data extraction reshapes power, autonomy, and democracy — necessary depth before tackling existential arguments.
The canonical text on long-term AI risk and the alignment problem at civilizational scale; dense but essential for understanding the arguments that now drive major AI safety research agendas.
A balanced, evidence-based analysis of automation's impact on labor, offering a counterweight to both utopian and dystopian narratives about AI and employment.
Governance and Informed Opinion: Policy, Ethics, and What to Do
Going deepSynthesize everything into the active policy and ethical debates, understand competing frameworks for AI governance, and form a well-grounded personal position.
▸ Study plan for this stage
Pace: 6–8 weeks total: ~3 weeks on "The Precipice" (~25–30 pages/day, including Appendices on risk estimation) and ~3–4 weeks on "Power and Progress" (~30–35 pages/day, pausing after each major section to write a 1-paragraph synthesis note). Reserve the final 3–4 days for cross-book integration and positi
- Existential and catastrophic risk: Ord's distinction between existential, global catastrophic, and recoverable risks, and why the asymmetry of irreversibility demands special moral weight
- The 'precipice' metaphor: humanity as standing at an unusually dangerous and consequential moment in history, where near-term decisions lock in long-run trajectories
- Longtermism and moral scope: the ethical case for weighting future generations heavily in policy decisions, and its tensions with near-term welfare priorities
- Technological risk taxonomy: Ord's analysis of AI as a leading source of existential risk compared to other candidates (pandemics, nuclear war, climate), and the reasoning behind those probability estimates
- Acemoglu's 'directed technology' framework: how economic incentives, power structures, and institutional choices channel technological development toward or away from broad human benefit
- The productivity-and-wages divergence: Acemoglu's empirical case that automation gains have not been broadly shared, challenging techno-optimist narratives about AI lifting all boats
- Contested visions of AI governance: comparing safety-first/existential-risk approaches (Ord) with political-economy/power-redistribution approaches (Acemoglu), and where they complement or conflict
- Agency and institutional design: what both authors say about who should govern AI — states, international bodies, civil society, firms — and what levers actually work
- According to Ord, what makes existential risk categorically different from ordinary large-scale harm, and how does he estimate AI's contribution to overall existential risk this century?
- How does Acemoglu's concept of 'directed technology' explain why AI development has so far concentrated gains among capital owners and high-skill workers rather than distributing them broadly?
- Where do Ord and Acemoglu agree on the dangers of concentrated power over AI, and where do their proposed remedies diverge most sharply?
- What institutional or governance mechanisms does each author advocate, and what are the strongest objections — from within the books themselves — to each author's preferred approach?
- How should a policymaker weigh Ord's longtermist, low-probability/high-consequence framing against Acemoglu's focus on near-term, high-probability distributional harms when allocating regulatory attention and resources?
- Having read both books, what is your own well-grounded position on the single most important governance intervention for AI in the next decade, and which arguments from each book most shaped it?
- Risk-ranking exercise: After finishing The Precipice, reproduce Ord's risk table from memory, then write a 1-page critique challenging at least two of his probability estimates using evidence or arguments he does not fully address.
- Power-mapping exercise: After each major section of Power and Progress, draw a stakeholder map showing who holds decision-making power over AI development in Acemoglu's account — firms, workers, governments, researchers — and annotate how that map shifts across historical eras he covers.
- Steel-man debate: Write two 400-word op-eds on the same AI governance question (e.g., 'Should governments impose a pause on frontier AI training runs?') — one channeling Ord's existential-risk framework, one channeling Acemoglu's political-economy framework. Then write a 200-word referee note identifying which argument you find more persuasive and why.
- Policy brief: Draft a 2-page policy brief addressed to a real legislative body (e.g., the EU AI Office, the U.S. Senate Commerce Committee) recommending one concrete AI governance measure. Cite specific arguments and evidence from both books; explicitly acknowledge the strongest counterargument from the book whose framework you weighted less.
- Framework comparison matrix: Build a table with rows for key governance dimensions (who governs, what is regulated, what timeline is prioritized, what counts as harm, what institutional form is preferred) and columns for Ord and Acemoglu. Fill it in with direct textual evidence, then add a third column for your own synthesized view.
- Reading-group or solo Socratic session: Generate five hard 'devil's advocate' questions — two that attack Ord's longtermism from Acemoglu's political-economy perspective, two that attack Acemoglu's near-termism from Ord's existential-risk perspective, and one that challenges both — then write out full answers as if defending each book in turn.
Next up: Mastering the tension between existential-risk governance (Ord) and political-economy governance (Acemoglu) gives the reader a mature, multi-framework lens that is the essential prerequisite for engaging with primary sources — legislation, technical safety research, and live policy debates — where these competing frameworks collide in real time.
Places AI risk within the broader framework of existential and catastrophic risks facing humanity, helping the reader weigh AI safety against other priorities with philosophical rigor.
A sweeping historical and economic argument that technological progress — including AI — does not automatically benefit society, and that deliberate governance choices determine who wins and who loses; the ideal capstone for forming an informed opinion.