# Why is the bayesian information criterion called that way?

The word "Bayes" suggests that we are updating a distribution using data, to get a posterior distribution.

The fact that the Bayesian information criterion (BIC) is used to select a model from a set of models, suggests that it is called BIC because we are selecting the model with the highest posterior, or something like that.

However, it's not clear to me from the definition of BIC that that's the case. Is it? What is the reason that the Bayesian information criterion is called "Bayesian"?

• BIC was proposed by Gideon E. Schwarz in this seminal paper where he provides a Bayesian argument for adopting it. Section 2 of the paper is dedicated to the explanation. If anyone can translate this to plain English, I would be interested to read. Commented Jun 6, 2019 at 15:41

BIC, sometimes called the Schwarz information criterion (SIC) was introduced by Gideon Schwartz in 1975. Here is that paper. It's not very long.

Both AIC and BIC address the model evaluation problem where more parameters lead to increased likelihood. To resolve this they penalize additional parameters. Schwartz gave a bayesian argument. His central thesis was, "In a model of given dimension, maximum likelihood estimators can be obtained as large-sample limits of the Bayes estimators for arbitrary nowhere vanishing a priori distributions."

So even though the formula for BIC has nothing Bayesian about it, the original paper is chock full of Bayesian explanations for the procedure since it was derived from a special "large-sample" case of Bayes Theorem.

• Could you explain the bayesian argument? I don't really understand it. Also, does the argument apply more to BIC than to AIC? i.e. is BIC really more "bayesian" than AIC, or is that an unfortunate historical terminological accident? Commented Jun 6, 2019 at 18:00
• Even though AIC and BIC are very similar, the only difference being the magnitude of penalization, the conceptual underpinnings are very different. AIC is not influenced by any Bayesian theory at all. After reading the Schwartz 1975 paper, Akaike also created the Akaike Bayesian Information Criterion (ABIC). For a better explanation of BIC, look at this paper, page 5, section 6: www-math.mit.edu/~rmd/650/bic.pdf. Commented Jun 6, 2019 at 19:16

One of my favourite simple interpretations and derivation of the BIC is in a lecture by Prof. Michael Jordan. This connects the Laplace approximation of the marginal likelihood (normalising constant in the Bayes Theorem, which is used for model comparison) with the BIC.

https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture16.pdf

Here, he derives the BIC using a Laplace approximation and some intuitive asymptotic arguments.

Laplace approximation:

Then,

Finally,