Artisanal Language Models: Define a task and write evals

A defining feature of an ALM is that it is purpose built for a well-defined task. So far the task has only been described in general terms: proofreading English prose of a novel draft. It’s time to get more specific about what this ALM will do, what its inputs and outputs will be, so that I can actually start building it. It’s also time to define some tests and performance measures. I’ll need them when I have something running and I make a change; I’ll need some way to measure whether it...