How I run LLMs locally on Mac 0 ▲ Avil Page 82 days ago · Tech · hide · Fediverse post Harness OpenCode & oh-my-pi come with batteries included and works out of the box. Pi is barebones & lightweight. There are quality plugins available to extend the functionality & customise the behaviour. It is extremely extensible. Inference Server Ollama MLX doesn't generate tokens efficiently. Rapix-MLX, omlx are excellent at utilizing resources and generating tokens efficiently. With M5 Pro 48GB RAM, they are able to generate ~50 tokens/second with Qwen3.6-35B-A3B-4bit model. Llama.cpp is cross-platform and it is extremely fast as well. Local LLMs Qwen3.6-35B-A3B model made locall LLMs usable on decent hardware. Qwen3.6-35B-A3B - has 35B parameters & only 3B are active at a time. I use 4bit quantized model as it provides a better tradeoff between speed and accuracy. It fits in 48GB RAM with some headroom for other applications. Bonsai has highest intelligent densisty and it can run on ~5 GB RAM. It is surprising how fast & powerful the local LLMs have become. Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town