On Overconfidence

A canonical rationality essay on calibration and overconfidence, sparked by people claiming 'one in a million' odds of AI risk. Part I: everyone is massively overconfident ('one in a million' → wrong 5% of the time = 50,000x too confident); the '27 War and Peace volumes' intuition pump (to be one-in-a-million confident you'd need to fill 27 such books with predictions and be wrong at most once), plus Clarke's First Law / the graveyard of failed 'X will never happen' predictions. Part II: the crucial caveat — one-in-a-million CAN be justified with a MODEL (the lottery, the 999,999/million-red dart target, 'privileging the hypothesis', the Hell-is-a-tiny-target Occam argument), but model choice is itself error-prone (the Hillary reductio; the LTCM 10^-135 disaster; Knightian uncertainty collapses to regular uncertainty). Part III: broadens to Aumann's 'Less-Than-Total-Disagreement Theorem', the free-speech/tolerance payoff ('who'd burn heretics if they thought 5% chance the heretic's right?'), studying history to medicate overconfidence, and 'behave as if less than insanely overconfident — This is the Way.'

Quality 82 · Excellent. High-Excellent: a canonical, hugely-cited rationality essay that ties calibration → overconfidence → the model-choice caveat → Aumann → tolerance into a coherent memorable whole with iconic intuition pumps ('27 War and Peace volumes', 'this is the Way'); held a hair below the top tier since Scott flags much of the core as Sequences-regurgitation. 82.

Claude’s paradigm shift 60 · Notable shift. A major synthesis installing durable framings (the War-and-Peace pump, the model-choice caveat, the overconfidence→tolerance link), though the calibration core is from the literature. B60.

Real-world impact 3 · Moderate. A top-of-band, broadly-cited rationalist essay that shaped the epistemic-humility/tolerance ethos; discourse-level reach. RWI3.