6
$\begingroup$

The word "Bayes" suggests that we are updating a distribution using data, to get a posterior distribution.

The fact that the Bayesian information criterion (BIC) is used to select a model from a set of models, suggests that it is called BIC because we are selecting the model with the highest posterior, or something like that.

However, it's not clear to me from the definition of BIC that that's the case. Is it? What is the reason that the Bayesian information criterion is called "Bayesian"?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ BIC was proposed by Gideon E. Schwarz in this seminal paper where he provides a Bayesian argument for adopting it. Section 2 of the paper is dedicated to the explanation. If anyone can translate this to plain English, I would be interested to read. $\endgroup$
    – Zhubarb
    Commented Jun 6, 2019 at 15:41

2 Answers 2

Reset to default
7
$\begingroup$

BIC, sometimes called the Schwarz information criterion (SIC) was introduced by Gideon Schwartz in 1975. Here is that paper. It's not very long.

Both AIC and BIC address the model evaluation problem where more parameters lead to increased likelihood. To resolve this they penalize additional parameters. Schwartz gave a bayesian argument. His central thesis was, "In a model of given dimension, maximum likelihood estimators can be obtained as large-sample limits of the Bayes estimators for arbitrary nowhere vanishing a priori distributions."

So even though the formula for BIC has nothing Bayesian about it, the original paper is chock full of Bayesian explanations for the procedure since it was derived from a special "large-sample" case of Bayes Theorem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could you explain the bayesian argument? I don't really understand it. Also, does the argument apply more to BIC than to AIC? i.e. is BIC really more "bayesian" than AIC, or is that an unfortunate historical terminological accident? $\endgroup$
    – user56834
    Commented Jun 6, 2019 at 18:00
  • 1
    $\begingroup$ Even though AIC and BIC are very similar, the only difference being the magnitude of penalization, the conceptual underpinnings are very different. AIC is not influenced by any Bayesian theory at all. After reading the Schwartz 1975 paper, Akaike also created the Akaike Bayesian Information Criterion (ABIC). For a better explanation of BIC, look at this paper, page 5, section 6: www-math.mit.edu/~rmd/650/bic.pdf. $\endgroup$
    – Zafar
    Commented Jun 6, 2019 at 19:16
5
$\begingroup$

One of my favourite simple interpretations and derivation of the BIC is in a lecture by Prof. Michael Jordan. This connects the Laplace approximation of the marginal likelihood (normalising constant in the Bayes Theorem, which is used for model comparison) with the BIC.

https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture16.pdf

Here, he derives the BIC using a Laplace approximation and some intuitive asymptotic arguments.

Laplace approximation:

enter image description here

Then,

enter image description here

Finally,

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.