poppy the training box, part 1: the beginnings

For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, perry. I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX), but there are a couple of problems. perry is my daily driver. If he's doing a training run, then everything is just a little bit sluggish as CPU and GPU alike are busy. Although I don't play games often, it's annoying to have the option ruled out for days at a time. While the GPU is busy with a training run, I can't do other experiments in parallel -- for example, to scope out what the next step might be. And relatedly to all of those: the two-day limit to the training runs I've been doing is something I set because that's the maximum amount of time I'm willing to have perry tied up. It would be really interesting to try longer training runs! I also have longer-term plans; a multi-GPU box would be interesting to put…