8
$\begingroup$

In the Wikipedia entry for Akaike information criterion, we read under Comparison with BIC (Bayesian information criterion) that

...AIC/AICc has theoretical advantages over BIC...AIC/AICc is derived from principles of information; BIC is not...BIC has a prior of 1/R (where R is the number of candidate models), which is "not sensible"...AICc tends to have practical/performance advantages over BIC...AIC is asymptotically optimal...BIC is not asymptotically optimal...the rate at which AIC converges to the optimum is...the best possible.

In the AIC talk section, there are numerous comments about the biased presentation of the comparison with BIC section. One frustrated contributor protested that the whole article "reads like a commercial for cigarettes."

In other sources, for example this thesis appendix the tenor of the claims for AIC seem more realistic. Thus, as a service to the community, we ask:

Q: Are there circumstances in which BIC is useful and AIC is not?

$\endgroup$
4
$\begingroup$

According to Wikipedia, the AIC can be written as follows: $$ 2k - 2 \ln(\mathcal L) $$ As the BIC allows a large penalization for complex models there are situations in which the AIC will hint that you should select a model that is too complex, while the BIC is still useful. The BIC can be written as follows: $$ -2 \ln(\mathcal L) + k \ln(n) $$ So the difference is that the BIC penalizes for the size of the sample. If you do not want to penalize for the sample there

A quick explanation by Rob Hyndman can be found here: Is there any reason to prefer the AIC or BIC over the other? He writes:

  • AIC is best for prediction as it is asymptotically equivalent to cross-validation.
  • BIC is best for explanation as it allows consistent estimation of the underlying data generating process.**

Edit: One example can be found in Time Series analysis. In VAR models the AIC (as well as its corrected version the AICc) often take to many lags. Therefore one should primarily look at the BIC when choosing the number of lags of a VAR Modell. For further information you can read chapter 9.2 from Forecasting- Principles and Practice by Rob J. Hyndman and George Athana­sopou­los.

$\endgroup$
  • $\begingroup$ Can you add more, please? In particular, cannot BIC be used to converge on an appropriate prior given its post-hoc? I appreciate the answer, thanks. BTW, "too" complex not "2". Strangely limited not temporal concept of "prediction" seems limited to predicting only in the sense of interpolation of values from a nearly identical range of withheld values. Usually the word prediction would apply to extrapolation beyond the range of an observed time series, which is not what either cross-validation or AIC are especially good at. Maybe the term "predicted interpolation" should be used. $\endgroup$ – Carl Sep 30 '16 at 15:00
  • $\begingroup$ The bold text is a one to one citation from Rob Hyndman, who is a famous statistics professor from Australia. I think by "prediction" he means "inference". So the AIC would be more useful for inferential statistics while the BIC would be more useful for descriptive statistics. $\endgroup$ – Ferdi Oct 1 '16 at 12:39
  • $\begingroup$ Yes, prolific as well. Still, what I am asking for is one good example of what AIC cannot do that BIC can. $\endgroup$ – Carl Oct 1 '16 at 15:40
  • 1
    $\begingroup$ @Ferdi, no, definitely "prediction" does not mean "inference" in that blog post. "Prediction" is "prediction", or "forecasting" where you don't care whether your model is "correct" (in some sense) as long as it forecasts well. Following that post, it seems that BIC is the preferred one for inference. $\endgroup$ – Richard Hardy Nov 28 '16 at 9:50
  • $\begingroup$ Thank you for your reply. Prediction or Forecast is "inferring" from observed data on "non-observed data". $\endgroup$ – Ferdi Dec 1 '16 at 9:40
1
$\begingroup$

It is not meaningful to ask the question whether AIC is better than BIC. Even though these two different model selection criteria look superficially similar they were each designed to solve fundamentally different problems. So you should choose the model selection criterion which is appropriate for the problem you have.

AIC is a formula estimates the expected value of twice the negative log likelihood of test data using a correctly specified probability model whose parameters were obtained by fitting the model to training data. That is, AIC estimates expected cross validation error using a negative log likelihood error. That is, $AIC \approx E\{-2 \log \prod_{i=1}^n p(x_i | \hat{\theta}_n)\}$ Where $x_1, \ldots, x_n$ are test data, $\hat{\theta}_n$ is estimated using training data, and $E\{ \}$ denotes the expectation operator with respect to the iid data generating process which generated both the training and test data.

BIC on the other hand is not designed to estimate cross validation error. BIC estimates twice the negative logarithm of the likelihood of the observed data given the model. This likelihood is also called the marginal likelihood it is computed by integrating the likelihood function weighted by a parameter prior $p(\theta)$ over the parameter space. That is, $ BIC \approx -2 \log \int [\prod_{i=1}^n p( x_i | \theta) ] p(\theta)d\theta$.

$\endgroup$
  • $\begingroup$ Some proponents of AIC versus BIC are so enamored of their opinions that they remind me of Democrats versus Republicans in the US. The question posited is a practical one as these armed camps often review scientific journal articles, and indeed a more relevant question is whether maximum likelihood is appropriate at all in the circumstances in which it tends to be applied. $\endgroup$ – Carl Jan 31 '19 at 22:53
  • $\begingroup$ BTW (+1) for contributing to discussion. Would like to see more about whether either AIC or BIC is applicable to when they tend to be used, but that is, admittedly, a separate question. $\endgroup$ – Carl Jan 31 '19 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.