Why output caching doesn't work 0 ▲ seangoedecke.com 35 minutes ago · Tech · hide LLM inference economics depends on caching1 model inputs, to prevent the entire conversation history being recomputed on each message2. Should we cache model outputs in the same way? As it turns out, no. Why cache outputs? First, here’s why we might think caching outputs is a good idea. We cache inputs to prevent repeated work, but LLMs do a lot of repeated work in their output as well: When I get agents to write PRs I often ask them to repeat and explain relevant code snippets in the conversation Long-running agent investigations produce lots of evidence docs, but also need to quote from those docs when they respond to the user Every time I’ve played around with automated AI research I’ve asked the agent to repeat some sample model outputs so I can make my own qualitative check In all of these cases3 I could go and read the files myself, but I don’t. Agents produce such a volume of output, and reading agent output is tiresome, so it’s better to simply ask for the key points. It’s an… Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town