2
$\begingroup$

I'm wondering what parameter BIC estimates. It seems that AIC is estimating the cross entropy of the estimated model and the true model, and asymptotically is estimating the out of sample entropy loss. If you divide AIC by 2n it seems that this scaled AIC ($\hat{L}/n - p/n$, where $\hat{L}$ is the maximum likelihood of the observed data, p is the number of parameters, and n is the sample size) is estimating the following:

$\int g(x) log\left( f(x|\theta)\right)$

Akaike mentions this fact in his 1974 paper.

But BIC has a different penalty term ($\log(n)p$ instead of $2p$) so I am wondering what parameter (if any) BIC is estimating.

Additionally, in the limit, it appears that the penalty terms for AIC and BIC are both approaching zero, so in the limit I would expect them to both estimate the same parameter. But AIC and BIC have different asymptotic properties (consistency of BIC and efficiency of AIC). Any insight on this would be helpful.

References:

Akaike, Hirotugu. "A new look at the statistical model identification." IEEE transactions on automatic control 19.6 (1974): 716-723.

$\endgroup$
  • $\begingroup$ Regarding the latter question, the asymptotic setup is different and does not allow for a direct comparison between AIC and BIC. I do not remember or fully understand the technical details, however. $\endgroup$ – Richard Hardy Jun 25 at 18:17
1
$\begingroup$

BIC is estimating the asymptotic marginal likelihood under a Bayesian model. Since its an asymptotic argument, the choice of prior doesnt matter which is why the BIC formula is prior-independent. See the original Schwartz paper: https://projecteuclid.org/euclid.aos/1176344136

"But AIC and BIC have different asymptotic properties (consistency of BIC and efficiency of AIC). Any insight on this would be helpful."

AIC is inconsistent when comparing nested models, when the simpler model is the true one. This is because the out-of-sample prediction error of both models is asymptotically equal, since the more complex model can reduce to the simpler one. Suppose we are doing variable selection and have:

Model 1: Y = B0 + B1 X + error

Model 2: Y = B0 + B1 X + B2 X2 + error

If model 2 is correct then the AIC will correctly identify it. But if Model 1 is correct then Mddel 2 is reducible to Model 1 by taking B2=0 (which will be its asymptotic estimate), so the AIC may not be consistent.

BIC will be consistent in this scenario. However the advantage of AIC comes when all of your models are false (i.e. when none correctly capture the data). In this case, AIC will asymptotically choose the model which is most similar to the true model (in terms of KL divergence) whereas the BIC generally doesnt have such well-defined nice asymptotic behavior.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.