Building intuition about LLM parameter counts

When I was building my GPT-2 implementation in JAX, I started with just token embeddings for the input, and a separate output head (as I was not using weight tying). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train -- specifically, an embedding dimension of 768. However, I realised I shouldn't be -- with a vocab size of 50,257, each of those components is essentially a 768×50,257 matrix, and that is indeed over 38 million numbers. But the finished LLM at the end of the project was only 163 million parameters -- that meant that the input and output components alone were almost half of it. That felt like a surprisingly large percentage. I had a similar shock when I was first looking into the feed-forward network, and realised that it had roughly twice as many parameters as the attention layers. When we learn…