The Holdout Is a One-Way Door · Evidence cannot independently confirm the claim it helped shape.

The first cross-validation scheme in the audio project looked clean at the row level and failed inside each row. Each row contained a pair of performances, and the pair rows did not overlap. But an individual performance could appear in more than one pair. Performance A could be part of training pair (A, B) and then return inside held-out pair (A, C). This is the useful way to think about a holdout: it is not a folder, dataframe column, or call to train_test_split. It is a promise about unavailable information. The promise says that the evidence used for final evaluation did not help choose the model, features, thresholds, exclusions, aliases, or story being evaluated. If the evidence shaped those decisions, moving its files to a new directory does not make it independent again. The unit that has to stay apart Live audio makes the obvious version of this problem unusually easy to create. One Goose show may exist as a Bandcamp FLAC, an Archive.org recording, a stream capture, an MP3…