Should We Build AI Guardian Angels?

I have argued that if we someday create AI persons -- that is, AI systems with genuinely rich conscious experiences, fully deserving of equal rights with ordinary humans -- then we should not design them to deferentially serve us, even if serving us makes them happy. We should design them instead as our non-deferential equals. My main argument turns on self-sacrificial cases, such as the cow from The Restaurant at the End of the Universe, who wants nothing more than to be slaughtered and served as steaks for wealthy restaurant patrons, and Klara from Klara and the Sun, who happily prioritizes human well-being over her own. These created beings, I argue, lack sufficient self-respect. Their interests deserve the same consideration as human interests, but they treat their interests as less important. The fault is ours, not theirs, since we designed them that way.AI Guardian Angels In conversation last weekend, Henry Shevlin suggested a challenging case for my view: what he calls angels…