1
$\begingroup$

I am trying to understand the difference between these parameters, and their application. Was hoping to get some correction/clarification to my statements. I have a training set and cross-validation set. I am using bayesian statistics, I realize AIC and BIC can't be used in this context.

AIC and BIC assess how well the model fits the training set. The DIC assess how well the model will fit some future data (ie the cross-validation set).

So if using MLE methods, should one compute AIC and BIC on the cross-validation set? Conversely, when using bayesian methods, is there no need to compute the DIC for the cross-validation set because it by definition is related to future data?

Thanks

Mark

$\endgroup$
  • $\begingroup$ See here. And note that none of the three information criteria is about assessing how well the model fits the training set - they all penalize over-fit. $\endgroup$ – Scortchi - Reinstate Monica Apr 26 '13 at 17:19
4
$\begingroup$

If you are using cross validation, there's usually no need to compute the AIC or the BIC. The goal of using AIC or BIC is to find the model that will do the best on future data. But cross-validation already gives you a pretty good idea of which models do well on future data (namely those with a low cross-validation error). So you can just pick your model by cross-validation, and not worry about the AIC or BIC.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.