Skip to main content
1 of 3
Henrik
  • 14.4k
  • 11
  • 70
  • 131

Besides the various measures of Minimum Description Length (e.g., normalized maximum likelihood, Fisher Information approximation), there are two other methods worth to mention:

  1. Parametric Bootstrap. It's a lot easier to implement than the demanding MDL measures. A nice paper is by Wagenmaker and colleagues:
    Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28-50.
    The abstract:

We present a general sampling procedure to quantify model mimicry, defined as the ability of a model to account for data generated by a competing model. This sampling procedure, called the parametric bootstrap cross-fitting method (PBCM; cf. Williams (J. R. Statist. Soc. B 32 (1970) 350; Biometrics 26 (1970) 23)), generates distributions of differences in goodness-of-fit expected under each of the competing models. In the data informed version of the PBCM, the generating models have specific parameter values obtained by fitting the experimental data under consideration. The data informed difference distributions can be compared to the observed difference in goodness-of-fit to allow a quantification of model adequacy. In the data uninformed version of the PBCM, the generating models have a relatively broad range of parameter values based on prior knowledge. Application of both the data informed and the data uninformed PBCM is illustrated with several examples.

  1. Cross-Validation: It is also quite easy to implement. See the answers to this question. However, note that the issue with it is that the choice among the sample-cutting rule (leave-one-out, K-fold, etc) is an unprincipled one.
Henrik
  • 14.4k
  • 11
  • 70
  • 131