A Local LLM Infrastructure

I built an LLM API gateway. Between that and all the infrastructure modernisation I’ve been doing at work, the months have blurred. I’ve been really busy, and there was never a good moment to write. But this is really exciting. So here goes. The project is called SoCLaaS. Nevermind the name — it means what it means. It’s our LLM infrastructure so that we can make LLM API service broadly available free-of-charge to our NUS School of Computing community, scalable for volume consumption without easily falling apart from overload or contenton. SoCLaaS begun as an experiment with Codex. I was exploring some ideas and thought to give Codex some work. That experiment grew into what it is today. It’s not at all a small project as it stands now. If not for AI, I estimate it would have taken 3 experienced developers 3 months to complete. Codex itself estimates 18 man-months. It’s amazing what we can do with AI. In a nutshell, SoCLaaS gives developers one stable OpenAI-compatible API while…