Are LLMs still surprisingly bad at some simple tasks?

Last year I ran an experiment to test the ability of modern LLMs to correctly answer a relatively straightforward question. Every single one of them got it wrong. Some missed information, some made up false statements, none were right. Of course the fanbois variously claimed that I was holding it wrong, my prompts were shit, I should have chosen better defaults, and - my favourite - that it would be better next year. Well, next year is now. 365 days after the original experiment, let's see if these self-reinforcing-learning machines have achieved anything close to intern-levels of competence. The Question That Started It All I asked: Which TLDs have the same name as valid HTML5 elements? Why It Matters This is the sort of question that I would expect a moderately bright teenager to answer. There exists a list which comprehensively includes all TLDs. There is a separate list which contains every HTML element. One could either manually go through the TLD list comparing it to the HTML…