@adlrocha - Decision Models-as-a-judge

This is post 2 of a 3-part series on decision models. Part 1 introduced what they are, and part 3 will cover how they are built and trained.One of the things that I hated the most about evaluating agents with LLM-as-a-judge is that it can be expensive and slow, forcing me in many cases to run those evals out-of-band instead of baking them into CI. So you can imagine that when I first read about decision models and all this idea of “Jev-as-a-judge”, I thought that this could be the perfect opportunity to finally introduce some evals in the critical path of my SDLC, as they may finally have become fast and cheap enough to be able to run them in CI. To test this, I built an eval pipeline to compare two code-review agents. Originally, it used an LLM-as-a-judge as the scorer, but I decided to replace it with a decision model as the scorer to compare their results and performance.What I wanted to test with this experiment is the hypothesis that LLM judges are too slow, too expensive and too…