94 days ago · Tech · hide · Fediverse post

There is certainly quite a bit more to optimise in local inference–Simon got around 72% more throughput from MTP speculative decoding–even if most readily available (and not hugely overpriced) consumer hardware still can’t quite get to the point where memory bandwidth makes dense models usable interactively. I can’t wait for an A3B or adaptive quantisation version to come out to see how it fares on even lower-end hardware.

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town