AIC/BIC parameter

Question

I'm wondering what parameter BIC estimates. It seems that AIC is estimating the cross entropy of the estimated model and the true model, and asymptotically is estimating the out of sample entropy loss. If you divide AIC by 2n it seems that this scaled AIC ($\hat{L}/n - p/n$, where $\hat{L}$ is the maximum likelihood of the observed data, p is the number of parameters, and n is the sample size) is estimating the following:

$\int g(x) log\left( f(x|\theta)\right)$

Akaike mentions this fact in his 1974 paper.

But BIC has a different penalty term ($\log(n)p$ instead of $2p$) so I am wondering what parameter (if any) BIC is estimating.

Additionally, in the limit, it appears that the penalty terms for AIC and BIC are both approaching zero, so in the limit I would expect them to both estimate the same parameter. But AIC and BIC have different asymptotic properties (consistency of BIC and efficiency of AIC). Any insight on this would be helpful.

References:

Akaike, Hirotugu. "A new look at the statistical model identification." IEEE transactions on automatic control 19.6 (1974): 716-723.

Answer 1

BIC is estimating the asymptotic marginal likelihood under a Bayesian model. Since its an asymptotic argument, the choice of prior doesnt matter which is why the BIC formula is prior-independent. See the original Schwartz paper: https://projecteuclid.org/euclid.aos/1176344136

"But AIC and BIC have different asymptotic properties (consistency of BIC and efficiency of AIC). Any insight on this would be helpful."

AIC is inconsistent when comparing nested models, when the simpler model is the true one. This is because the out-of-sample prediction error of both models is asymptotically equal, since the more complex model can reduce to the simpler one. Suppose we are doing variable selection and have:

Model 1: Y = B0 + B1 X + error

Model 2: Y = B0 + B1 X + B2 X2 + error

If model 2 is correct then the AIC will correctly identify it. But if Model 1 is correct then Mddel 2 is reducible to Model 1 by taking B2=0 (which will be its asymptotic estimate), so the AIC may not be consistent.

BIC will be consistent in this scenario. However the advantage of AIC comes when all of your models are false (i.e. when none correctly capture the data). In this case, AIC will asymptotically choose the model which is most similar to the true model (in terms of KL divergence) whereas the BIC generally doesnt have such well-defined nice asymptotic behavior.