Survey Statistics: dCV for MRP ?

Three weeks ago we learned about design-based cross validation (dCV), shown in Figure 1(d) of Iparragirre et al. (2023): Each dot is a PSU (primary sampling unit), which can be an individual but is often a group/cluster of individuals. Each color is a stratum. dCV is the usual K-fold CV but: keep PSUs together within a fold reject a split if a whole stratum falls into one fold modify the weights so that each subsample replicates the original sample Let’s return to the problem of using CV to assess Multilevel Regression and Poststratification (MRP) models. We saw that individual-level Loss(y_i, yhat_i) might not be great for assessing MRP models, even when weighted, and that CV noise can swamp model differences. The dCV method from Iparragirre et al. (2023) is for a probability sample. MRP is usually used for a nonprobability sample (e.g. an online survey). But maybe there’s still something to learn here. Bayesian Data Analysis Chapter 7 about evaluating predictive accuracy p.169 says…