Say I have a set of three models $M = \{M_1, M_2, M_3\}$ that are nested, i.e. $M_1$ is a constrained version of $M_2$ which again is a constrained version of $M_3$. I have a set of data $X$ with some empricial (univariate) distribution. Each of my models generates data with some univariate distribution like this and I would like to know which model fits the empirically observed distribution best. At the same time I would like to penalize model complexity, i.e. the number of free parameters. Unfortunately I can not calculate the likelihood of the data under each of my models and use a BIC or AIC for model selection. Is there any way I can assess goodness of fit of nested models (while penalizing complexity) that is not likelihood-based? Ideally, the criterion for goodness of fit would be some distance measure betweeen the distributions.

  • $\begingroup$ Have you considered Approximate Bayesian Computing (ABC)? $\endgroup$ – Xi'an Mar 4 '19 at 9:11
  • $\begingroup$ Thanks @Xi'an. I didn't now about ABC but it seems to be what I am looking for. One question though: Based on this example my understanding is that I use the ABC rejection algorithm to get an approximation of the posterior distribution of the model parameters. The quality of this approximation will depend on the number of parameter samples and the tolerance (among other factors). Can I then simply use the posterior instead of the unknown likelihood to calculate the BIC for my three candidate models? $\endgroup$ – nhoeft Mar 4 '19 at 11:39
  • $\begingroup$ The BIC is a poor approximation to the (log-) Bayes factor so indeed there is no need to revert to BIC once ABC has produced a posterior sample. $\endgroup$ – Xi'an Mar 4 '19 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.