Suppose your manager asks you next week to demonstrate that the AI coding tools your company signed up for are worth the subscription cost. Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way; the sections below explain why. Note: this post is about how people are assessing AI, not at LLM-assisted coding itself; with a little rewording, these criticisms could be applied to a lot of the claims that have been made about agile development, test-driven development, and other practices. If I’ve learned anything in the last twenty years, it’s that software engineering would be a lot further ahead today if we had been willing to let our peers in the human sciences teach us how to study these kinds of things properly. Counting Lines of Code Generated Proxy metrics stand in for concepts that are hard to measure directly, and lines of code is one of the…
No comments yet. Log in to reply on the Fediverse. Comments will appear here.