The Generalizability Problem, the Natural Field Experiments Answer

There’s a standard problem in the social sciences in moving from the results of a smaller-scale study to larger-scale applicability. Consider a program in a certain school district that seems to increase academic performance and high school graduation rates. Would the program work in another school district? Will it work statewide? Would it work nationwide? Would it work in other countries? How “generalizable” is the result? The question doesn’t have a simple answer, but John A. List offers his own angle in “Make science more reliable: study people as they go about their lives” (Nature, June 25, 2026, 654, pp. 863-866). List offers various memorable examples of the generalizability problem. Take Scared Straight, a programme run by more than 30 US states between 1978 and 2015 that aimed to dissuade at-risk teenagers from becoming hardened criminals by bringing them face-to-face with people incarcerated in maximum-security prisons. The programme was extended after a pilot project, the…