91 days ago · 6 min read1265 words · Tech · hide · Fediverse post

Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training the same model on the same data, ran at about: 21,000 tokens per second using the hand-rolled GELU from Sebastian Raschka's book "Build a Large Language Model (from Scratch)". 25,000 tokens per second using PyTorch's built-in GELU with no arguments. 25,000 tokens per second using the built-in GELU with approximate="tanh", which uses the same maths as Raschka's version under the hood. That's a 20% increase in throughput for both of the built-in versions -- definitely nothing to be sneezed at. And what is particularly interesting is that there aren't that many GELUs going on -- it's a GPT-2…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town