Exploring 1-Bit LLMs 0 ▲ Thejesh GN 85 days ago · Tech · hide · Fediverse post My interest in LLMs has increasingly shifted towards smaller models, small in terms of size and their ability to run on modest local machines. For some time now, I have been using models with fewer than 40 B parameters on my machines in some quantized way. Then I came across BitNet, a project that introduced me to the concept of using just 1 bit per weight. Quantization Weights are represented by numbers in a model. A normal model might store each weight using 16 bits, i.e FP16 (half-precision floating-point). Quantization reduces that dramatically, where weights are represented using INT8 (8-bit integer) or INT4 (4-bit integer). They reduce precision but increase inference speed and reduce hardware requirements. Most of the time, quantization is performed post-training, which reduces model accuracy. But there is INT4 QAT (4-bit Integer Quantization-Aware Training), which does this during training, thereby reducing the extreme accuracy drops common with standard 4-bit quantization.… Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town