Use the built-in GELU, don't roll your own!

Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training the same model on the same data, ran at about: 21,000 tokens per second using the hand-rolled GELU from Sebastian Raschka's book "Build a Large Language Model (from Scratch)". 25,000 tokens per second using PyTorch's built-in GELU with no arguments. 25,000 tokens per second using the built-in GELU with approximate="tanh", which uses the same maths as Raschka's version under the hood. That's a 20% increase in throughput for both of the built-in versions -- definitely nothing to be sneezed at. And what is particularly interesting is that there aren't that many GELUs going on -- it's a GPT-2…