When statistical models fail

An investigative article just appeared in BMJ that has important statistical content (link).The problem concerns turning raw data on “adverse events” (side effects) related to the Covid-19 vaccines into statistical “red flags”. How do we know if the number of reported events is large enough to sound the alarm?This problem exists even before the development of Covid-19 vaccines. Those vaccines just made the scale of everything much larger, since so many people got vaccinated.Two statistical methods were mentioned in the article. The first method is called “proportional reporting ratios” (PRR). The second method is dubbed “empirical Bayes”.BMJ reports that the CDC leadership only relied on the empirical Bayes method during the pandemic. This is controversial because the investigation discovered that the two methods yielded diametrically opposite results. Each method produced extreme results but in opposite directions!The empirical Bayes method produced zero signals while the PRR method…