Ginjinha: Pretraining LLMs on European Portuguese 0 ▲ Duarte O.Carmo 107 days ago · 9 min read1812 words · Tech · hide · Fediverse post For the past year, I've been very curious about the intersection of Large Language Models (LLMs) and the European Portuguese language. I've built evaluations, classifiers, and eventually the largest pretraining dataset in European Portuguese. Now, that's all fine and dandy, but it raises the obvious question: Can we pretrain an LLM fully in European Portuguese? And is it any good? Welcome to project Ginjinha. But before we start training models, we need a harness. A pretraining harness: NanoChat In 2025, Andrej Karpathy released NanoChat: a small, hackable codebase for training GPT-2-style language models. It's easy to understand and modify. Exactly what I needed. The first step was to adapt it to Ginjinha (code). NanoChat includes the three key stages of training LLMs: Pretraining: We train a text-completion model to predict the next token. Supervised fine-tuning (SFT): We further train the model to follow instructions (i.e., chat). Reinforcement learning (RL): We "align" the model… Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town