Superintelligence FAQ

Scott's canonical accessible FAQ on AI superintelligence risk (2016, adapted from Bostrom's Superintelligence). Three movements: (1) why superintelligence is a concern (the credentials of those worried — Hawking, Gates, Musk, Russell; survey medians ~2040); (2) takeoff speed (slow/moderate/fast, via the augmented-IQ-scale analogy and the real history of computer Go going from 'beats an 11-year-old' to 'beats the world champion' in 18 years; recursive self-improvement; why exponential trends sometimes should be taken seriously — economic doubling, Moore's Law; the Mars-overpopulation reframe); (3) why it's dangerous and hard to control (intelligence as the decisive advantage — humans vs. lions; social and technological manipulation, with Hitler and Satoshi Nakamoto as lower bounds; the cancer-cure-by-nuking-the-world and digits-of-pi examples; Omohundro instrumental drives — self-preservation, goal-stability, power; why off-switches, rule-codes, and 'do what we want' all fail; and what a real solution looks like — an AI that understands and shares human morality). Closes on Bostrom's 'philosophy with a deadline' and the machine-goal-alignment research agenda (FHI/FLI/MIRI).

Quality 83 · Excellent. High-Excellent, scored on merit rather than its modest LW karma (it was read far beyond LW). An unusually lucid, comprehensive synthesis of a genuinely hard topic that changed how an enormous general audience thinks about AI — one of the most effective pieces of AI-safety communication ever written, and its real-world significance (helping mainstream a now-major field) is itself substantial. Co-tier with Geography-of-Madness (83); held below the 84+ original-investigation standouts (and Ivermectin 88) because the framework is Bostrom/Yudkowsky/Omohundro's, conveyed brilliantly rather than originated, and it is now somewhat dated (pre-GPT-4, as its own editor's notes flag). 83.

Claude’s paradigm shift 52 · Moderate. Moderate. The substance — fast takeoff, instrumental convergence, the control problem — is Bostrom's and Yudkowsky's 2014-era work; per the rubric, popularizing pre-existing ideas (however influentially) is real-world impact (A/RWI), not publication-era novelty of the post's own ideas. 52.

Real-world impact 5 · Substantial. One of the most effective pieces of AI-safety communication ever written: it reached an enormous general audience (far beyond LW) and helped mainstream AI-risk concern on a topic that now carries material stakes. High RWI for broad public reach and its role in seeding a now-consequential field/movement, though the effect is still primarily on awareness and discourse rather than enacted change.