My understanding is that if you (1) have a sufficiently large test dataset, and (2) your models have the same likelihood (noise assumption), then you should compare/select the model likelihood (or log-likelihood) on the test data, unadjusted by AIC/BIC/DIC/etc.

Is there any justification for adjusting your likelihood for the test (or cross-validation) dataset for model selection? What is the best practice when such test dataset is available?

  • $\begingroup$ What do you mean by "adjusting model likelihood by AIC"? AIC is a measure (derived from likelihood) that has a different purpose and different domain than hypothesis testing. $\endgroup$ – January Sep 24 '13 at 13:18
  • $\begingroup$ @January please enlighten me. Isn't AIC used for selecting the number of parameters, which is a form of nested model selection? $\endgroup$ – Memming Sep 24 '13 at 13:37
  • $\begingroup$ Yes, AIC is used for model comparison (nested, not nested, overlapping...). It is an alternative to model selection with hypothesis testing. However, I still do not understand how do you adjust model likelihood (or test data) by AIC (this is not a criticism, I think it is either confusion or my lack of knowledge). $\endgroup$ – January Sep 24 '13 at 14:25
  • $\begingroup$ @January Perhaps I'm not using the right language here. I see AIC as penalizing the log-likelihood by the number of parameters. That's why I said "adjust". Feel free to edit the question for clarification. $\endgroup$ – Memming Sep 24 '13 at 14:45
  • $\begingroup$ Hmmm, maybe, though I'm still quite unsure on what you are asking. Maybe you could come up with an example? $\endgroup$ – January Sep 24 '13 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.