65 days ago · 25 min read5055 words · Tech · hide · Fediverse post

Intro2 months back, I started the Prime Residency with Florian Brand as my peer or shall I say my verifier. My first task was to port the Agents’ Last Exam Linux CLI subset to verifiers. Then one day Florian asked me to look into the data and this quest slowly evolved into finding issues in the benchmark and the making of what we call ALE-Gold. It was more challenging than I expected. I was new to this kind of work, so there was some skill issue on my part. Along the way, I also hit benchmark defects, inference provider failures, and gaps in the freshly released Verifiers v1 (that have been fixed now so you don't need to worry). This post is a work log of my porting work, what the full runs revealed about the benchmark, how that led to ALE-Gold, followed by an analysis of how the models performed. A small odyssey from porting ALE to ALE-Gold, with a few unexpected detours. Table of contents Porting ALE Linux CLI subset to verifiers What is a Rollout? What porting ALE involved and some…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town