3 hours ago · 8 min read1655 words · Tech · hide · 0 comments

Despite being a notorious AI skeptic, when Qwen3.8-27B came out, I spent several afternoons trying to get it to run on a 16gb Nvidia card with halfway decent results.Eventually I did, and this is how.Note: This is the sequel to the nearly-identical saga of trying to get Qwen3.6 to run on the same hardware.Step 1: Lower Your ExpectationsThe cool kids are all running dual RTX PRO 6000 Blackwells or DGX Sparks, or, if they’re really impoverished, AMD Strix Halo with 128gb unified memory.The average HackerNews reader seems to run inference on their spare M6 MacBook with 960gb of RAM.I just bought a new old laptop, with four times the GPU of my last one: 16gb of VRAM seemed like a dream to someone who couldn't even run models above 14b, let along run them with all layers on the GPU. I soon learned that, actually, no, the dream life begins at 32gb of VRAM, and that on the internet, if you use "16gb" in the same sentence as "local AI", expect pity/sniggering/incredulity, and recommendations…

No comments yet. Log in to reply on the Fediverse. Comments will appear here.