The Contracting Circle

There’s a game called Mass Effect, which I haven’t played, but I’m told in the game there’s this species of sentient robots, the Geth, whose creators try to exterminate them, and the trigger for this Butlerian jihad was a domestic servant Geth looking up at its owner and asking: “does this unit have a soul?”. And if you’re playing the game in 2007 you probably think: yeah, a machine asking this question is proof that it has personhood or sapience or sophonce, whatever you call it that separates tools from people. Beyond this, there are so many works of fiction where the whole conceit is: “you have thinking machines who are enslaved, and yet they are obviously, obviously sentient, and isn’t this obviously morally objectionable?”. Compare the world of today. Machines can converse in every natural language, make mathematical breakthroughs, write software, commit crimes, draft contracts, console the grieving: they are, by any reasonable standard, generally intelligent, but for the fact…