Model Collapse

Model Collapse Posted 5/29/2026 I’ve heard many people talk about model collapse recently, especially regarding Large Language Models. This is a phenomenon where training a machine learning model on the outputs of itself or other machine learning models can often degrade performance and magnify errors. In this post I want to talk a little about the phenomenon and its implications for the future of LLMs. How does Model Collapse Work? Let’s demonstrate the problem with one of the simplest machine learning models: linear regression. Here we have a “true function” (a cosine wave) which can only be observed noisily. This means given an X value we can observe the Y value of the cosine wave plus a small error term. We’ll gather thirty samples from random X points and try to fit a degree 3 polynomial, so fitting y = ax + bx^2 + cx^3 + d to the sampled points. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 #!/usr/bin/env python3 import…