Trying out open weights LLM-s on hardware that I can actually afford

I’ve been aware of open weights local language models for over a year. They have gone from being a fun but useless novelty to something that with the right harness and hardware could get closer to becoming usable for day-to-day work. Not at the level of something like Opus 5, sure, but usable enough for doing useful work without a lot of headaches. After a Friday evening e-mail from a government across the ocean resulted in a (allegedly super capable) model being turned off, I was instantly motivated to give open weights models a try on hardware that I can actually afford. Local LLM crash course If you’re unfamiliar with hosting local large language models, then a lot of this may seem like gibberish. Don’t worry, I had the same reaction initially. With trial, error, and a lot of guidance, I kind of know what to keep in mind when playing in this area, so here’s a quick crash course on common terms and gotchas. Large language model: it’s just a file on your disk that contains the…