The “humans are imperfect reporters too” defense for ascribing little thoughts to machines

This is Jessica. In my last post about the tension between the “necessity” that we ascribe human folk psychological concepts like thinking and reasoning to machines and the problems that arise when we overinterpret them, I briefly mentioned a defense I sometimes hear for anthropomorphic terms. The defense goes like this: We can’t protest the application of words like reasoning or belief or desire or understanding to AI because humans are also not always trustworthy reporters of the latent states that these terms refer to. And if humans can’t demonstrate full awareness of their reasoning process or their beliefs or their understanding or their intentions, why should we expect AI to? So, the argument goes, it is asymmetric to require complete faithfulness of LLM traces to some latent state. That would be setting a higher bar than we currently use with humans. This line of reasoning takes two things that may resemble each other on the surface (e.g., what humans report their reasoning or…