Survey Statistics: should MRP workflow include LOCO-CV ?

Due tomorrow (June 10): Enter a contest for Alexandre Andorra’s interview of Aki, Richard, and Andrew about their new book Bayesian Workflow. I hope folks ask about evaluating MRP models. We’ve seen: Individual-level Loss(y_i, yhat_i) may not be great for choosing models for MRP. (“individualism doesn’t work”) Weighted-to-the-population individual-level loss also isn’t great. (“individualism doesn’t work (even when weighted)”) Cross-validation noise can swamp important model differences. (“Individualism and the CV Noise Problem”) In probability samples, splitting a cluster between training and test fits models with more information than we should, but not splitting a stratum between training and test fits models with less information than we should. How does this apply to MRP ? (“dCV for MRP ?”) At Andrew Gelman’s 60-ish Birthday workshop Aki gave a great talk about loo’s 10ish birthday. The loo R package computes approximate leave-one-out (loo) cross-validation. Aki covered a huge…