This question is a follow-up or attempt to clear up possible confusion regarding a topic I and many others find a bit difficult, regarding the difference between AIC and BIC. In a very nice answer by @Dave Kellen on this topic (https://stats.stackexchange.com/a/767/30589) we read:
Your question implies that AIC and BIC try to answer the same question, which is not true. AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered. On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.
In a comment below, by @gui11aume , we read:
(-1) Great explanation, but I would like to challenge an assertion. @Dave Kellen Could you please give a reference to where the idea that the TRUE model has to be in the set for BIC? I would like to investigate on this, since in this book the authors give a convincing proof that this is not the case. – gui11aume May 27 '12 at 21:47
It seems that this assertion comes from Schwarz himself (1978), although the assertion was not necessary: By the same authors (as @gui11aume links to), we read from their article "Multimodel inference: Understanding AIC and BIC in Model selection" (Burnham and Anderson, 2004):
Does the derivation of BIC assume the existence of a true model, or, more narrowly, is the true model assumed to be in the model set when using BIC? (Schwarz's derivation specified these conditions.) ... The answer ... no. That is, BIC (as the basis for an approximation to a certain Bayesian integral) can be derived without assuming that the model underlying the derivation is true (see, e.g. Cavanaugh and Neath 1999; Burnham and Anderson 2002:293-5). Certainly, in applying BIC, the model set need not contain the (noexistent) true model representing full reality. Moreover, the convergence in probability of the BIC-selected model to a targbet model (under the idealization of an iid sample) does not logically mean that that target model must be the true data-generating distribution).
So, I think it is worth a discussion or some clarification (if more is needed) on this subject. Right now, all we have is a comment from @gui11aume (thank you!) under a very highly voted answer regarding the difference between AIC and BIC.