The Kappa Zoo: David Eubanks’s online monograph on rating models

David Eubanks writes: My site is kappazoo.com, and it’s still a work in progress. I would rather have emailed after I had the new goodness-of-fit code done, but I saw that you’re doing a summer workshop (on Andrew’s blog) [editor: Modern Modeling Methods (M3)], so thought I’d mention it now. It may be billed as a work in progress, but it’s a complete draft with no missing sections that provides a really nice overview of rating/crowdsourcing models. These are the models that dragged me into statistics, namely Bayesian rating models formulated as noisy measurement models. The first model of this kind that I or Eubanks could find was Phil Dawid and Allan Skene’s (1979) paper on rating. Eubanks works through a great deal of workflow without calling it that. There are multiple model evaluation and comparison measures used and explained with connections to information-theoretic notions like entropy. There’s a long discussion of Cohen’s kappa statistic, which is a commonly reported statistic…