Exploring 1-Bit LLMs

My interest in LLMs has increasingly shifted towards smaller models, small in terms of size and their ability to run on modest local machines. For some time now, I have been using models with fewer than 40 B parameters on my machines in some quantized way. Then I came across BitNet, a project that introduced me to the concept of using just 1 bit per weight. Quantization Weights are represented by numbers in a model. A normal model might store each weight using 16 bits, i.e FP16 (half-precision floating-point). Quantization reduces that dramatically, where weights are represented using INT8 (8-bit integer) or INT4 (4-bit integer). They reduce precision but increase inference speed and reduce hardware requirements. Most of the time, quantization is performed post-training, which reduces model accuracy. But there is INT4 QAT (4-bit Integer Quantization-Aware Training), which does this during training, thereby reducing the extreme accuracy drops common with standard 4-bit quantization.…