A Straight Line Predicts How Fast My Laptop Generates LLM Tokens

My previous post ended on a cliffhanger. For the model Qwen2.5-0.5B, I found that the time per output token (TPOT) was roughly 9 ms for different prompt sizes. But I didn’t have an explanation for why. I’ve since learned that in the decode phase for large language model (LLM) inference the primary bottleneck is memory bandwidth, the rate at which the GPU reads LLM model weights from RAM. To explore this, I measured the TPOT of six models of varying sizes on my laptop. Contents# The Memory Bandwidth Bottleneck TPOT vs Model Size How I Measured This Caveats of This Measurement What I Took Away From This The Memory Bandwidth Bottleneck# In LLM inference, the decode phase produces one token at a time. Each time, the weights have to be streamed from RAM to the GPU chip, where the next token is computed. In other words, to generate NNN tokens, you’d have to re-read the model weights NNN times. This is because the GPU chip has a limited amount of memory, on the order of kilobytes or…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town