5
$\begingroup$

I have a simple question about model comparison:

Let's say you fit two models using MCMC: Model A and model B, where model B is model A minus one parameter. You want to assess whether dropping the parameter improves the model fit.

You calculate the likelihood and AIC for each MCMC sample. So you end up with a distribution of AIC values for each model (e.g., if you take 10,000 MCMC samples for each model, then you have 10,000 AIC values from the posterior for each model).

How do I compare the AIC distributions? I know that I could compare means, medians, or do other simple tests, but I don't want to guess wrong. I've seen libraries such as AICcmodavg that automate model comparison, but my model is complicated enough that I'll probably have to write this by hand, and I'm embarrassed to say that I don't know what goes on behind the curtain.

$\endgroup$
3
  • 1
    $\begingroup$ Model selection in a Bayesian context is usually done via model evidence aka marginal likelihood, not AIC. $\endgroup$
    – A. Donda
    Jul 23, 2018 at 1:33
  • 1
    $\begingroup$ Thank you. That led me to Bayes factors--hopefully that is the right way to go. $\endgroup$
    – John
    Jul 23, 2018 at 6:03
  • $\begingroup$ This is not correct, if only because this uses the data twice. Check Murray Aitkin's book for a similar (and flawed) approach. $\endgroup$
    – Xi'an
    Jul 23, 2018 at 8:08

1 Answer 1

1
$\begingroup$

Firstly, you do not necessarily need to decide on a single model. Bayesian model averaging or a single model with some kind of sparsity-inducing prior on the extra parameter are often very good options for prediction modeling.

Secondly, tying a decision about a model choice to any particular threshold for Bayes factors, DIC or AIC can be quite problematic - except in extreme cases, when it is completely clear that the simple model is too simple. Thus, approaches that allow for uncertainty in this respect (see "Firstly, ..."). Just because one does this in a Bayesian context does not make the problems of model selection with hypothesis tests/AIC/whatever go away.

Finally, there are further options to look at how well a model is performing. These include posterior predictive checks, evaluation on a hold-out test set, leave-one-out cross-validation (which can be very easy to implement in MCMC samplers with a lot of useful discussion available on Andrew Gelman's blog).

$\endgroup$
1
  • $\begingroup$ Thanks. I found Andrew Gelman's blog enormously useful in providing a broad overview of how all of these confusingly similar techniques are related, and when each might--or might not--be appropriate. $\endgroup$
    – John
    Jul 24, 2018 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.