Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090

I mentioned I'd got a second RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool. He's right that it's not really been my thing -- I've been focusing on my own LLMs recently. I decided to dig in a little, and in particular to play with Llama.cpp, which I haven't used for a while. And then things got a tad out of control, and I wound up doing some relatively detailed benchmarking. The headline results: I downloaded Unsloth's UD-IQ4_NL_XL quantisation of the model from Hugging Face. With that, using the default Arch build of Llama.cpp, which uses Vulkan under the hood: Using the GPU only, I was able to get the model to generate at just over 120 tokens per second, and it was able to process the prompt at just less than 2,800 tok/s. However, having the whole…