15
$\begingroup$

In the Wikipedia entry for Akaike information criterion, we read under Comparison with BIC (Bayesian information criterion) that

...AIC/AICc has theoretical advantages over BIC...AIC/AICc is derived from principles of information; BIC is not...BIC has a prior of 1/R (where R is the number of candidate models), which is "not sensible"...AICc tends to have practical/performance advantages over BIC...AIC is asymptotically optimal...BIC is not asymptotically optimal...the rate at which AIC converges to the optimum is...the best possible.

In the AIC talk section, there are numerous comments about the biased presentation of the comparison with BIC section. One frustrated contributor protested that the whole article "reads like a commercial for cigarettes."

In other sources, for example in this thesis appendix, the tenor of the claims for AIC seem more realistic. Thus, as a service to the community, we ask:

Q: Are there circumstances in which BIC is useful and AIC is not?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
8
$\begingroup$

According to Wikipedia, the AIC can be written as follows: $$ 2k - 2 \ln(\mathcal L) $$ As the BIC allows a large penalization for complex models there are situations in which the AIC will hint that you should select a model that is too complex, while the BIC is still useful. The BIC can be written as follows: $$ -2 \ln(\mathcal L) + k \ln(n) $$ So the difference is that the BIC penalizes for the size of the sample. If you do not want to penalize for the sample there

A quick explanation by Rob Hyndman can be found here: Is there any reason to prefer the AIC or BIC over the other? He writes:

  • AIC is best for prediction as it is asymptotically equivalent to cross-validation.
  • BIC is best for explanation as it allows consistent estimation of the underlying data generating process.**

Edit: One example can be found in Time Series analysis. In VAR models the AIC (as well as its corrected version the AICc) often take to many lags. Therefore one should primarily look at the BIC when choosing the number of lags of a VAR Modell. For further information you can read chapter 9.2 from Forecasting- Principles and Practice by Rob J. Hyndman and George Athana­sopou­los.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Can you add more, please? In particular, cannot BIC be used to converge on an appropriate prior given its post-hoc? I appreciate the answer, thanks. BTW, "too" complex not "2". Strangely limited not temporal concept of "prediction" seems limited to predicting only in the sense of interpolation of values from a nearly identical range of withheld values. Usually the word prediction would apply to extrapolation beyond the range of an observed time series, which is not what either cross-validation or AIC are especially good at. Maybe the term "predicted interpolation" should be used. $\endgroup$
    – Carl
    Sep 30, 2016 at 15:00
  • $\begingroup$ The bold text is a one to one citation from Rob Hyndman, who is a famous statistics professor from Australia. I think by "prediction" he means "inference". So the AIC would be more useful for inferential statistics while the BIC would be more useful for descriptive statistics. $\endgroup$
    – Ferdi
    Oct 1, 2016 at 12:39
  • $\begingroup$ Yes, prolific as well. Still, what I am asking for is one good example of what AIC cannot do that BIC can. $\endgroup$
    – Carl
    Oct 1, 2016 at 15:40
  • 2
    $\begingroup$ @Ferdi, no, definitely "prediction" does not mean "inference" in that blog post. "Prediction" is "prediction", or "forecasting" where you don't care whether your model is "correct" (in some sense) as long as it forecasts well. Following that post, it seems that BIC is the preferred one for inference. $\endgroup$
    – Richard Hardy
    Nov 28, 2016 at 9:50
  • $\begingroup$ Thank you for your reply. Prediction or Forecast is "inferring" from observed data on "non-observed data". $\endgroup$
    – Ferdi
    Dec 1, 2016 at 9:40
4
$\begingroup$

Q: Are there circumstances in which BIC is useful and AIC is not?

A: Yes. BIC and AIC have fundamentally different goals. BIC estimates the probability that a model minimizes the loss function (specifically, the Kullback-Leibler divergence); a BIC difference of .1 between A and B implies that model A is roughly 10% more likely to be the best model -- assuming you start with close to no information and have a large sample size. AIC, by contrast, measures how good a model is at making predictions -- a difference of .1 means (very roughly) that model A will be about 10% better at making new predictions than model B.

This means that BIC can be better if you want to know the probability that a model is true. AIC can't give you that; if you try using AIC in this way, you get inconsistent estimates (i.e. AIC will not always select the true model).

On the other hand, AIC will be better at minimizing the expected loss.

AIC and BIC have two fundamentally different goals (BIC tries to maximize the chances of picking the best model, while AIC tries to maximize the expected quality of the model you select).

Improve this answer
$\endgroup$
33
  • 1
    $\begingroup$ @Carl I don't know about the physical analogy (I am not a physicist), but I think what you're saying about AIC/BIC having problems is correct. AIC and BIC are approximations that work in the limit with infinite data and few parameters (the number of data points per parameter has to be large for the approximation to hold). AICc partially corrects for this by adding another term, so it works better in smaller samples, but a perfect calculation would require an infinitely long series. $\endgroup$
    – Closed Limelike Curves
    Oct 23, 2021 at 19:24
  • 1
    $\begingroup$ @Carl You're right that adjusted R^2 doesn't mean anything when R^2 equals 1 -- because adjusted R^2 is undefined when you get a perfect fit from k >= n-1 parameters. If you get a perfect fit with less than n-1 parameters, that's not just interpolation anymore, it's a valid result (assuming it's not a post-hoc hypothesis). With regards to KL loss function, it depends on what estimator you're using for KL loss. If you use AICc, a perfect fit with k=n-1 parameters is undefined. $\endgroup$
    – Closed Limelike Curves
    Oct 24, 2021 at 2:18
  • 1
    $\begingroup$ @Carl yes, you're right that all of these are approximations that start to break down for large numbers of parameters. That being said, the differences are generally small in practice. $\endgroup$
    – Closed Limelike Curves
    Oct 24, 2021 at 16:05
  • 1
    $\begingroup$ If you want to deal with this in a completely principled way that takes into account fractional degrees of freedom and works even for very small sample sizes, you can use leave-one-out cross-validation instead -- AIC and AICc are both asymptotic approximations of leave-one-out cross-validation using K-L divergence loss. WAIC handles fractional degrees of freedom directly, but breaks down for small samples just as much as AIC does. $\endgroup$
    – Closed Limelike Curves
    Oct 24, 2021 at 16:08
  • 1
    $\begingroup$ @Carl You can definitely compare nonlinear, weighted least squares heteroscedastic models using AIC -- a model that doesn't have heteroscedasticity included will usually be a poor fit to heteroscedastic data and this will show up in the log-likelihood term of AIC. The WLS estimator is just a maximum likelihood estimator that assumes variables follow Gaussian dists with different variances. $\endgroup$
    – Closed Limelike Curves
    Nov 1, 2021 at 1:29
3
$\begingroup$

It is not meaningful to ask the question whether AIC is better than BIC. Even though these two different model selection criteria look superficially similar they were each designed to solve fundamentally different problems. So you should choose the model selection criterion which is appropriate for the problem you have.

AIC is a formula estimates the expected value of twice the negative log likelihood of test data using a correctly specified probability model whose parameters were obtained by fitting the model to training data. That is, AIC estimates expected cross validation error using a negative log likelihood error. That is, $AIC \approx E\{-2 \log \prod_{i=1}^n p(x_i | \hat{\theta}_n)\}$ Where $x_1, \ldots, x_n$ are test data, $\hat{\theta}_n$ is estimated using training data, and $E\{ \}$ denotes the expectation operator with respect to the iid data generating process which generated both the training and test data.

BIC on the other hand is not designed to estimate cross validation error. BIC estimates twice the negative logarithm of the likelihood of the observed data given the model. This likelihood is also called the marginal likelihood it is computed by integrating the likelihood function weighted by a parameter prior $p(\theta)$ over the parameter space. That is, $ BIC \approx -2 \log \int [\prod_{i=1}^n p( x_i | \theta) ] p(\theta)d\theta$.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Some proponents of AIC versus BIC are so enamored of their opinions that they remind me of Democrats versus Republicans in the US. The question posited is a practical one as these armed camps often review scientific journal articles, and indeed a more relevant question is whether maximum likelihood is appropriate at all in the circumstances in which it tends to be applied. $\endgroup$
    – Carl
    Jan 31, 2019 at 22:53
  • $\begingroup$ BTW (+1) for contributing to discussion. Would like to see more about whether either AIC or BIC is applicable to when they tend to be used, but that is, admittedly, a separate question. $\endgroup$
    – Carl
    Jan 31, 2019 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.