What Do We Do When We Read The Code?

Last Friday, I wasted over an hour arguing with an LLM. I was reviewing a pull request I asked it to generate, and realized that three of the tests it wrote were either not useful or literally could never fail. I hate reading tests. Reading tests is hard and boring, but I always try to read them carefully. They’re your guardrails. They don’t always do a good job of capturing the nature of the behaviour or feature that you’re trying to achieve, but at least they try to guarantee that things won’t change in unpredictable ways in the future without you noticing. They take time and effort to read, and eventually I realized these tests were useless.1 When I asked the LLM about it, it got bashful. Rather than reasoning about it like I had, it impressively wrote a mutation test to indeed show the test was useless, and then wasted more time trying to find ways to preserve the spirit of the tests, which I simply deleted. Afterwards, I sat back and thought about this interaction. What am I…