2
$\begingroup$

If a set of models is being compared using BIC and AIC, given the fact that the true model (the one which generated the data) is in this set (and given the other assumptions that guarantee BIC consistency in selecting the true model);

As N goes to infinity, if I understand things correctly, BIC will always pick the true model, AIC will either pick the true model or a slightly more general one with more parameters. This would mean that BIC will always pick a model that predics as well or better that the one picked by AIC.

Q1) So, can we say that as N goes to infinity, BIC will perfom better that AIC not only form an "explanation" point of view (selecting the correct model) but also from a "prediction" point of view (selecting the model that minimizes the prediction error)?

Q2) I know there are some situations where the true model is not the best for prediction, but when this happens the best should have less parameters and not more. Anyway, will this situasions happen only if N is small or are there examples of this happening as N goes to infinity?

Improve this question
$\endgroup$
7
  • $\begingroup$ Under some assumptions, AIC is asymptotically efficient regardless of whether the true model is in the set of candidate models or not. Under these assumptions, BIC cannot be asymptotically efficient because it differs from AIC. Now if we take the assumptions that yield consistency of BIC, we do not have that AIC is asymptotically efficient (the two sets of assumptions do not coincide). Thus it is possible that BIC is asymptotically efficient under these assumptions, though I would it as I have not encountered such a result before. I am not skilled enough to prove or disprove that, however. $\endgroup$
    – Richard Hardy
    Commented Jun 9, 2021 at 16:27
  • $\begingroup$ Regarding Q2, this is where the "Paradox in model selection (AIC, BIC, to explain or to predict?)" comes in. Also, I do not think this can happen when $N\rightarrow\infty$ because the omission of a relevant variable (model bias) becomes relatively ever more costly while the other source of prediction error (model variance) shrinks due to the sample size exploding. $\endgroup$
    – Richard Hardy
    Commented Jun 9, 2021 at 16:31
  • $\begingroup$ Thank you for your comments; wouldn't the fact that BIC is consistent in identifying the true model, and given that the true model minimises the prediction error (at least for infinite N), imply that BIC is efficient? $\endgroup$
    – Marco Rudelli
    Commented Jun 9, 2021 at 20:48
  • $\begingroup$ @RichardHardy In this paragraph of a paper by Vrieze (2012), he seems to imply so: "If the number of parameters in the true model is infinite, or increases with increasing N, or if the true model is not in the candidate model set, then the AIC is asymptotically efficient in mean squared error of estimation/ prediction and in K-L divergence, whereas the BIC is not. The BIC is efficient when the true model is among the candidates. In this context, efficiency just means that the loss function of interest is asymptotically minimized—that it is as small as possible given the candidate models." $\endgroup$
    – Marco Rudelli
    Commented Jun 9, 2021 at 21:16
  • $\begingroup$ wouldn't the fact <...> imply that BIC is efficient? It may look like that on the surface, but I am not entirely persuaded it is correct. The precise assumptions under which BIC may be efficient would need to be worked out carefully. You have not given a full reference to the paper, so I cannot check, but keep in mind that papers sometimes contain mistakes. Not saying this one does, just that it is a possibility. $\endgroup$
    – Richard Hardy
    Commented Jun 10, 2021 at 5:41

1 Answer 1

Reset to default
2
$\begingroup$

Not a definite answer but some thoughts.

Under some assumptions, AIC is asymptotically efficient regardless of whether the true model is in the set of candidate models or not. Under these assumptions, BIC cannot be asymptotically efficient because it differs from AIC to a nontrivial extent. Now if we take the assumptions that yield consistency of BIC, we do not have that AIC is asymptotically efficient (the two sets of assumptions do not coincide). Thus it seems possible that BIC is asymptotically efficient under these assumptions. However, a consistent criterion can never be efficient (Yang, 2005 as cited in Chapter 4 of Claeskens & Hjort, 2008). This suggests BIC is likely not efficient under the assumptions that make it consistent.

A theoretically solid source comparing AIC and BIC and discussing consistency and efficiency is Chapter 4 of Claeskens & Hjort "Model selection and model averaging" (2008) (a book).

References

  • Claeskens, G. and Hjort, N.L. (2008). Model Selection and Model Averaging. Cambridge University Press.
  • Yang, Y. (2005). Can the strengths of AIC and BIC be shared? Biometrika, 92:937–950
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, I will surely check out the book and the paper $\endgroup$
    – Marco Rudelli
    Commented Jun 10, 2021 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.