Making agents fast - what I've learnt over the last 12 months

I've spent a lot of my career optimising websites and apps to run faster, and it's been interesting to apply some of those principles to agentic workflows. I'll run over some of my key learnings - there's many more I want to get written down at some point, but for brevity I think this is my rough mental model of tackling this. Inference speed The obvious thing to start with is inference speed. This has two key components, which will be familiar if you have done any backend API optimisation. The first is TTFT (time to first token). This is analogous to TTFB (time to first byte) in web servers, namely how quickly does the first token come after you've done a request. This includes prefill (input token processing), so this scales with how many tokens the agent is sending to the inference provider too - so you should expect to see (potentially significant) variance on this number, much like you'd expect TTFB to be higher on an endpoint that e.g. has to do more database calls to prepare…