Given the basic marginal likelihood identity:

$$ ln \; m(y)=ln \; p(y|\theta^*)+ln\; \pi(\theta^*)-ln \pi(\theta^*|y) $$

is there a way to derive from this the Bayesian Information Criterion?


1 Answer 1


As proposed by Chib (1995), the marginal likelihood can be computed from the marginal likelihood identity: $$m(y)=\frac{\phi(y|\theta^*)\pi(\theta^*)}{\pi(\theta^*|y)}$$ where $\theta^*$ can be any admissible value. The natural logarithm of this equation presents a computationally convenient expression: $$log \; m(y) = log\; \pi(\theta^*)+log \; \phi(y|\theta^*)-log \; \pi(\theta^*|y)$$ This expression can also be seen as something corrected by $-log\; \pi(\theta^*|y)$. The assumption on the posterior density: $\int_A \pi_\alpha(\theta|\alpha)\; d\theta \xrightarrow{n\to \infty} \int_A N(\theta|\mu, \Sigma) \; d\theta$ which can also be scale by $\sqrt{n}(\theta-\hat{\theta})$ will converge to a standard normal. Therefore we have: $\theta|y\sim N(\hat{\theta}, \frac{1}{n}\nu_\infty)$ where $\nu_\infty=\left(ln \; \frac{1}{n}\frac{d^2 log \; p(y|\theta}{d\theta d\theta} \right)^{-1}$ is a bounded term $Op(1)$. We can use this $\theta|y$ put it in the identity: $$m(y)\approx\frac{\pi(\theta)\phi(y|\theta)}{N(\theta|\hat{\theta},\frac{1}{n} \nu_\infty)}$$ and evaluate at MLE: $$m(y)\approx\frac{\pi(\hat{\theta_n})\phi(y|\hat{\theta}_n)}{(2\pi)^{-\frac{1}{2}}|\frac{1}{n}\nu_\infty|^{-\frac{1}{2}} e^0}=\frac{\pi(\hat{\theta_n})\phi(y|\hat{\theta}_n)}{(2\pi)^{-\frac{1}{2}}\left(\frac{1}{n}\right)^{-\frac{d}{2}}|\nu_\infty|^{-\frac{1}{2}} }$$ given that $|kA|=k^d|A|$ Given that $\pi(\hat{\theta_n})$, $(2\pi)^{-\frac{1}{2}}$ and $|\nu_\infty|^{-\frac{1}{2}}$ are $Op(1)$, then we have: $$log \; m(y)= log \; \phi(y|\hat{\theta_n})-\frac{d}{2} log \; n + Op(1)$$ which is the BIC ($m(y)=BIC[1+Op(1)]$) which includes a penalty so that high dimensional model not necessarily are better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.