The Cognitive Advantages of Being of Two Minds, or Twelve, or Twelve Hundred

I was of three minds, Like a tree In which there are three blackbirds. [from Wallace Stevens, Thirteen Ways of Looking at a Blackbird] The literature on superintelligence and AI alignment tends to assume that an intelligent AI will have one set of values, one reward function, and consistent, determinate preferences. But speaking for myself, I'm not as consistent as that. You aren't either, I suspect. Now, maybe this is a defect in us that would be absent from a well-engineered intelligent system -- but I don't think so. There are advantages to having, so to speak, a splintered mind. Two Minds for Two Theories Suppose you're 85% confident Theory A is true and 15% confident Theory B is true. And suppose it's not obvious what follows from the two theories; discovering the implications will require substantial cognitive work. With that work, you could learn that Theory A implies C, D, and E, while Theory B implies F, G, and H. Here are three cognitive strategies: All-in on the most…