Ginjinha: Pretraining LLMs on European Portuguese

For the past year, I've been very curious about the intersection of Large Language Models (LLMs) and the European Portuguese language. I've built evaluations, classifiers, and eventually the largest pretraining dataset in European Portuguese. Now, that's all fine and dandy, but it raises the obvious question: Can we pretrain an LLM fully in European Portuguese? And is it any good? Welcome to project Ginjinha. But before we start training models, we need a harness. A pretraining harness: NanoChat In 2025, Andrej Karpathy released NanoChat: a small, hackable codebase for training GPT-2-style language models. It's easy to understand and modify. Exactly what I needed. The first step was to adapt it to Ginjinha (code). NanoChat includes the three key stages of training LLMs: Pretraining: We train a text-completion model to predict the next token. Supervised fine-tuning (SFT): We further train the model to follow instructions (i.e., chat). Reinforcement learning (RL): We "align" the model…