# Can BIC be Used for Hypothesis Testing

Define the Bayesian information criterion as $$\mathrm{BIC} = {-2 \cdot \ln{\hat L} + k \cdot (\ln(n) - \ln(2 \pi))}$$ (I do not drop the constant, $- \ln(2 \pi)$, to avoid issues when equating to the marginal likelihood)

Given data $Y$ and a model $H_i$, the approximate relationship between the marginal likelihood $P(Y|H_i)$ and $\mathrm{BIC}_i$ is $$-2 \cdot \ln P(Y|H_i) \approx \mathrm{BIC}_i$$ which seems to imply $$P(Y|H_i) \approx \exp\bigg(\frac{\mathrm{BIC}_i}{-2}\bigg)$$

Given a null and alternative model, $H_0$ and $H_1$ respectively, the Bayesian hypothesis test for the probability of the alternative conditional on the data could be computed as $$P(H_1|Y)=\frac{P(Y|H_1)}{P(Y|H_0)+P(Y|H_1)}$$ where the prior probability, $P(H_i)=1/2$ for $i=1,2$. My question is when, if ever, is it okay to approximate $$P(H_1|Y) \approx \frac{\exp\bigg(\frac{\mathrm{BIC_1}}{-2}\bigg)}{\exp\bigg(\frac{\mathrm{BIC_0}}{-2}\bigg)+\exp\bigg(\frac{\mathrm{BIC_1}}{-2}\bigg)}$$ for Bayesian hypothesis testing. Despite the simplicity of the above equation I have rarely seen it used in practice which makes me doubt it's reliability as an approximation.

• Note that when doing the above calculation at the end, whether you drop constants is immaterial. – Glen_b Jan 28 '16 at 0:48
• – Christoph Hanck Mar 18 '19 at 18:45

You can construct just such an asymptotic approximation, but note that you can rewrite it in terms of the difference from (say) $BIC_0$ (or indeed any convenient constant). This can help avoid problems with underflow or overflow when exponentiating numbers which might be very far from 0.

Note further that (using a similar approach to the one you used) it generalizes to a larger collection of alternative models than just two.

I wouldn't call it "hypothesis testing", though; to my mind it's nearer to Bayesian model selection, but it occurs more often in a related but slightly different context. (Don't mind me, though, other people have referred to it or something very like it as hypothesis testing, you can probably find several examples among the references in the links below, and elsewhere.)

It (or a slightly re-written form of it) is an approximation I've seen often (I guess it depends on which things you read), and does produce an approximate posterior probability of the models under consideration (under a particular set of assumptions).

It occurs particularly often in the context of discussions of model averaging or model uncertainty, where rather than choosing a particular model and conditioning on that choice, all* of the models are weighted by their posterior probability, in order (for example) to produce a distribution of predictions.

* or sometimes just a subset of the models with the highest posterior probabilities, often as an approximation of an overall, but sometimes extremely large set. (see also Occam's window)

If you search on Bayesian model averaging and BIC you should be able to turn up quite a few references (names like Hoeting, Raftery or Madigan are on quite a few of the papers, but many other authors write on this stuff); if you can't find any I can point some out.

Just as one example, in Raftery , equation 35, he uses just such an expression as you have above but generalized to $k$ models.

Try these links, which have a number of papers that do something along the lines of what you describe (for the first link, I can't get the original to load so I have gone to the last version at archive.org):

https://web.archive.org/web/20150925053749/http://www2.research.att.com/~volinsky/bma.html

http://www.stat.washington.edu/raftery/Research/bma.html

(not all links at those pages will necessarily be what you're after, but each will have many papers that relate to it.)

 Raftery, A.E. (1995).
"Bayesian model selection in social research (with Discussion)."
Sociological Methodology, 25, 111-196.

• Maybe this is touching the vein you were alluding to. Is the probability of the hypothesis given the data, or the more common frequentist analogous inverse, a necessary part of hypothesis testing? Doesn't hypothesis testing boil down to the decision made? In that case couldn't you call a BIC_1 < BIC_0 decision rule a type of hypothesis testing? – russellpierce Feb 8 '16 at 17:23