To what degree do we have goals?
Read the original on LessWrong →
Summary
A sharp entry in the behaviorism sequence on the metaphysics of agency. Thesis: the unconscious is a behavior-executor with tendencies, not preferences (like the blue-minimizing robot), so you cannot 'negotiate' with it the way Hanson proposed a 'fair compromise' between conscious and unconscious interests — that is like negotiating with wildfires or tornadoes, which have tendencies, not goals. The operational test is context-stability: the cookie urge is not a preference (you would not press a button to teleport a cookie into your stomach, and you would happily press a button that removes the urge), whereas the astronaut goal is (Timmy presses the become-an-astronaut button and refuses the remove-the-desire button). We model others — Obama, the Civ IV AI, evolution, the heart — as utility-maximizers because modeling them as behavior-executors is intractable, but it is a deliberate simplification (the intentional stance). Cliffhanger: do we wrongly model ourselves the same way, and get confused when we fail to maximize?
Why this score
Quality 75 · Excellent. Strong. The context-stability test for distinguishing genuine preferences from mere urges is a real, well-illustrated conceptual tool, and the utility-maximizer-as-tractable-simplification point is cleanly made. A sequence middle-post that builds heavily on the blue-minimizing robot and ego-syntonic material and ends on a cliffhanger, so not higher.
Claude’s paradigm shift 50 · Moderate. Notable — the context-stability operationalization of 'preference' and the no-negotiation-with-the-unconscious framing are fresh in 2012, building on the blue-minimizing robot and the intentional stance.
Real-world impact 2 · Minor. A sharp entry on the metaphysics of agency whose context-stability test (genuine preferences vs mere urges — you wouldn't press a button to teleport the cookie into your stomach) is a real, well-illustrated conceptual tool. Conceptual influence within rationalist discourse, a sequence middle-post, no material change — low RWI.