Trust in data: reproducibility packages do not guarantee validity

Last year the World Bank made news when it announced that all its outputs would include a reproducibility package, which contains all the data and code needed to reproduce the figures and tables in the report. In this regard it is following the lead of major journals in economics. Compared with some other fields, such as clinical trials (Avenell & Bishop, 2026), the widespread availability of data and code in economics publications is impressive. Nevertheless, the confidence given by reproducibility packages needs to be tempered by one important consideration: a reproducibility package is only as good as the data that it is based on and the method of analysis that is selected. If the underlying dataset is flawed or the analysis isn't suited to the research question, then the conclusions may be unjustified, even if the whole analysis reproduces perfectly. An example: the "edge science" study by Packalen and Bhattacharya(2020)Before he became Director of the US National Institutes of…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town