Is there a prior probability distribution associated with BIC (Bayesian Information Criterion)?

I ask this because in the R package BAS there is a linear modeling function that can be called as follows:

bas.lm(y ~ x, data = my_dataframe, prior = 'BIC')

and the documentation is not clear on how the priors are set given the 'BIC' argument.

The posterior distributions come out to be normal looking (student-T actually I think). They could for example look something like this: posteriors

In summary, I am wondering what prior distributions are being used for the parameters?

  • $\begingroup$ The BIC is only Bayesian in denomination, as it does not use a prior distribution. The asymptotic expansion of the Bayes factor is such that the impact of the prior distribution is only second-order and BIC stops at this order. $\endgroup$ – Xi'an Sep 10 '16 at 9:36

I cannot offer you a definite answer for that particular package, but in general, the Bayesian Information Criterion is a large-sample approximation to the Bayes factor. As such, as the likelihood dominates the prior in large samples, the effect of the prior is neglected.

Here is a somewhat heuristic derivation.

We can obtain a large sample normal approximation to the log-likelihood $l(\theta|y)$ in the multiparameter case. A multivariate Taylor expansion gives \begin{eqnarray} l(\theta|y)&\approx&l(\hat\theta|y)+(\theta-\hat\theta)'\frac{\partial l(\hat\theta|y)}{\partial\theta}-\frac{n}{2}(\theta-\hat\theta)'V^{-1}(\theta-\hat\theta)\notag\\ &=& l(\hat\theta|y)-\frac{n}{2}(\theta-\hat\theta)'V^{-1}(\theta-\hat\theta) \end{eqnarray} where $$ V=\left(-\frac{1}{n}\sum_j\frac{\partial^2 l(\hat\theta|y_j)}{\partial\theta\partial\theta'}\right)^{-1}=\left(-\frac{1}{n}\frac{\partial^2 l(\hat\theta|y)}{\partial\theta\partial\theta'}\right)^{-1} $$ Let $$m_i(y)=\int f_{i}\left(y|\theta_{i},M_i\right)\pi_{i}\left(\theta_{i}|M_i\right)d\theta_{i}$$

Substituting the exponential of the approximate expression for $l(\theta|y)$ gives \begin{eqnarray*} m_i(y)&\approx&L(\hat\theta_i|y)\int\exp\left(-\frac{n}{2}(\theta_i-\hat\theta_i)'V_i^{-1}(\theta_i-\hat\theta_i)\right)\pi_i(\theta_i|M_i)d\theta_i\\ &\approx&L(\hat\theta_i|y)\pi_i(\hat\theta_i|M_i)\int\exp\left(-\frac{n}{2}(\theta_i-\hat\theta_i)'V_i^{-1}(\theta_i-\hat\theta_i)\right)d\theta_i, \end{eqnarray*} where the second approximation arises because the $\exp$ part dominates around $\hat\theta_i$ (look for Laplace approximation to get a rigorous argument here).

As the integrand now is a kernel of a multivariate normal distribution with covariance matrix $V_i/n$, carrying out the integration gives \begin{eqnarray*} m_i(y)&\approx&L(\hat\theta_i|y)\pi_i(\hat\theta_i|M_i)(2\pi)^{d_i/2}|n^{-1}V_i|^{1/2}\\ &=&L(\hat\theta_i|y)\pi_i(\hat\theta_i|M_i)(2\pi)^{d_i/2}n^{-d_i/2}|V_i|^{1/2}, \end{eqnarray*} where $d_i$ is the number of parameters in $\theta_i$.

The log-Bayes factor for models 1 and 2 can now be approximated by \begin{eqnarray*} \log(B_{12})&\approx&\left[\log\left(\frac{L_1(\hat\theta_1|y)}{L_2(\hat\theta_2|y)}\right)-\frac{d_1-d_2}{2}\log(n)\right]\\ &&+\left[\log\left(\frac{\pi_1(\hat\theta_1|M_1)}{\pi_2(\hat\theta_2|M_2)}\right)+\frac{1}{2}\log\frac{|V_1|}{|V_2|}+\frac{d_1-d_2}{2}\log(2\pi)\right] \end{eqnarray*} The second squared brackets can be neglected for large $n$. The first term is the log-likelihood ratio, which is large if the data favors $M_1$. The second term penalizes larger models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.