Non-Determinism Doesn't Matter

One of my fundamental objections to LLM-driven software development was that model output is non-deterministic. Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions. I used to think that this was a severe limitation of AI tools. How could we possibly use this unpredictable building block to craft systems that require precise and exact calculations? But as I started to get more comfortable running AI as my primary means of "writing" code, I had a revelation: While the result needs to be exact, there is not one correct way to get there. The means by which you obtain a correct result do not need to be deterministic! Previously, I was used to specifying behavior in terms of code: precise sets of instructions that compile and execute in a predictable way. And mentally, I was treating LLM output like code. But over time, I realized that LLMs play a fundamentally different role than code does in software…