Naive Rationality has classic failure modes.

  1. Failing to take into account 2nd and 3rd effects This often happens when those 2nd and 3rd effects are unknown to rationality but known to intuition
  2. Long Term Thinking Naive rationality leads to undervaluing poorly understood activity, (ex. Play / exploration) for maximizing expected value greedily in each moment
  3. Lack of Complex Consequentialism ‘Consistency’ demands that we choose either Deontology or Consequentialism, or some other framework for evaluating actions. But these systems all have value in particular contexts, and so should subsume one another. For example, effective rules often have to be clear (discrete) for psychological reasons, which leads to Deontology. The consequentialist should realize when that optimizes the outcome.
  4. Disrespect for Intuition / Cultural Knowledge What people mean when they say ‘wisdom’ is knowledge whose source is unknown, unknowable or forgotten. Passed from generations of Elders. Intuited through a distributed, parallel process in a mind that can’t be understood linearly.
  5. Rational People Defect and Free Ride There are many game-theoretic situations where cooperation is only enforced by social norms. In that context, a person relying on rationality instead of social imitation will defect (assuming that the long term ramifications for reputation, etc. are low)
  6. Assuming Normal Distributions in Extremistan
  7. What you see is all there is (relevant to Chesterton’s fence)
  8. Believing that you’re rational leads to being overconfident in your understanding of the situation
  9. Ignore the emotional side of thought / experience Damning when emotions are the generators of action, rationalized
  10. Appearances don’t matter, Environment doesn’t matter This sense of invulnerability to emotional / cultural impact due to a belief in one’s own rationality
  11. Chesterton’s Fence The absence of understanding something is conflated with the misunderstood object being useless. An important instance of the overconfidence that comes out of naive rationality. Two reformers find a fence in their way. The first doesn’t see the fence’s purpose, so he goes to knock it down. The second says that until the first reformer can see the purpose of the fence, he won’t allow him to knock it down.
  12. Goodhart’s Law Naive Rationality overfits to metrics that are available to a quantitative, empirical mind. But there’s much that science can’t touch and that is difficult to measure, and so when rationality is combined with something intuitive (say, ‘goodness’) it leads to EA / Singerism. In that case, lives saved is measurable.