AMALIA vs. smaller reasoning models

A few weeks ago there was a bit of a scandal back home. Everyone was mad that the government had taken so long to grade this year's high school exams. At the time, I saw a weird post on LinkedIn saying we should have AMALIA grade all those exams because that would be faster. Let's be serious. Using an LLM to grade high school exams is a horrible idea. But it made me think for a bit. How good is AMALIA at answering those questions? What the author might not know is that the AMALIA team also thought of this. PT Exams is one of the benchmarks the team created to test exactly that. It contains 1,819 questions taken from high school exams, formatted as multiple-choice questions. The question stands. How good is AMALIA at doing this? I was not particularly interested in comparing it with much bigger/closed models. I was more interested in understanding how smaller models perform vs. AMALIA. The results are interesting. In the following interactive chart, you can see how different models…