# What is the connection between credible regions and Bayesian hypothesis tests?

In frequentist statistics, there is a close connection between confidence intervals and tests. Using inference about $\mu$ in the $\rm N(\mu,\sigma^2)$ distribution as an example, the $1-\alpha$ confidence interval $$\bar{x}\pm t_{\alpha/2}(n-1)\cdot s/\sqrt{n}$$ contains all values of $\mu$ that aren't rejected by the $t$-test at the significance level $\alpha$.

Frequentist confidence intervals are in this sense inverted tests. (Incidentally, this means that we can interpret the $p$-value as the smallest value of $\alpha$ for which the null value of the parameter would be included in the $1-\alpha$ confidence interval. I find that this can be a useful way to explain what $p$-values really are to people who know a bit of statistics.)

Reading about the decision-theoretic foundation of Bayesian credible regions, I started to wonder whether there is a similar connection/equivalence between credible regions and Bayesian tests.

Is there a general connection? If there is no general connection, are there examples where there is a connection? If there is no general connection, how can we see this?

-
A related question which I've been wondering about - could someone point me to a paper they consider to be the "gold standard" or "canonical example" of Bayesian hypothesis testing used on a real problem, rather than a toy example. I've never really understood Bayesian hypothesis testing and I think I'd find a good example of its use instructive. –  Patrick Caldon Jul 10 '12 at 2:45
@PatrickCaldon I doubt there is a "golden paper" on this because Bayesian hypothesis testing is formulated in a decision-theoretic framework (therefore it is too wide to be captured in a single paper). The book mentioned in MånsT's answer provides a good material, Berger's books and talks might also be of interest. –  user10525 Jul 10 '12 at 11:26

I managed to come up with an example where a connection exists. It seems to depend heavily on my choice of loss function and the use of composite hypotheses though.

I start with a general example, which is then followed by a simple special case involving the normal distribution.

## General example

For an unknown parameter $\theta$, let $\Theta$ be the parameter space and consider the hypothesis $\theta\in\Theta_0$ versus the alternative $\theta\in\Theta_1=\Theta\backslash\Theta_0$.

Let $\varphi$ be a test function, using the notation in Xi'an's The Bayesian Choice (which is sort of backwards to what I at least am used to), so that we reject $\Theta_0$ if $\varphi=0$ and accept $\Theta_0$ if $\varphi=1$. Consider the loss function $$L(\theta,\varphi) = \begin{cases} 0, & \mbox{if } \varphi=\mathbb{I}_{\Theta_0}(\theta) \\ a_0, & \mbox{if } \theta\in\Theta_0 \mbox{ and }\varphi=0\\ a_1, & \mbox{if } \theta\in\Theta_1 \mbox{ and }\varphi=1. \end{cases}$$ The Bayes test is then $$\varphi^\pi(x)=1\quad \rm if\quad P(\theta\in\Theta_0|x)\geq a_1(a_0+a_1)^{-1}.$$

Take $a_0=\alpha\leq 0.5$ and $a_1=1-\alpha$. The null hypothesis $\Theta_0$ is accepted if $\rm P(\theta\in\Theta_0|x)\geq 1-\alpha$.

Now, a credible region $\Theta_c$ is a region such that $\rm P(\Theta_c|x)\geq 1-\alpha$. Thus, by definition, if $\Theta_0$ is such that $\rm P(\theta\in\Theta_0|x)\geq 1-\alpha$, $\Theta_c$ can be a credible region only if $\rm P(\Theta_0\cap\Theta_c|x)>0$.

We accept the null hypothesis if an only if each $1-\alpha$-credible region contains a non-null subset of $\Theta_0$.

## A simpler special case

To better illustrate what kind of test we have in the above example, consider the following special case.

Let $x\sim \rm N(\theta,1)$ with $\theta\sim \rm N(0,1)$. Set $\Theta=\mathbb{R}$, $\Theta_0=(-\infty,0]$ and $\Theta_1=(0,\infty)$, so that we wish to test whether $\theta\leq 0$.

Standard calculations give $$\rm P(\theta\leq 0|x)=\Phi(-x/\sqrt{2}),$$ where $\Phi(\cdot)$ is the standard normal cdf.

Let $z_{1-\alpha}$ be such that $\Phi(z_{1-\alpha})=1-\alpha$. $\Theta_0$ is accepted when $-x/\sqrt{2}>z_{1-\alpha}$.

This is equivalent to accepting when $x\leq \sqrt{2}z_{\alpha}.$ For $\alpha=0.05$, $\Theta_0$ is therefore rejected when $x>-2.33$.

If instead we use the prior $\theta\sim \rm N(\nu,1)$, $\Theta_0$ is rejected when $x>-2.33-\nu$.

The above loss function, where we think that falsely accepting the null hypothesis is worse than falsely rejecting it, may at first glance seem like a slightly artifical one. It can however be of considerable use in situations where "false negatives" can be costly, for instance when screening for dangerous contagious diseases or terrorists.

The condition that all credible regions must contain a part of $\Theta_0$ is actually a bit stronger than what I was hoping for: in the frequentist case the correspondence is between a single test and a single $1-\alpha$ confidence interval and not between a single test and all $1-\alpha$ intervals.

-
+1 I would use credibility region instead of credibility interval. –  user10525 Jul 9 '12 at 11:40
Thanks @Procrastinator! I've edited the answer and changed it to "region" while I was at it. I mostly work with HPD regions of unimodal posteriors, so I tend to think of confidence regions as intervals. :) –  MånsT Jul 9 '12 at 18:20

Michael and Fraijo suggested that simply checking whether the parameter value of interested was contained in some credible region was the Bayesian equivalent of inverting confidence intervals. I was a bit skeptical about this at first, since it wasn't obvious to me that this procedure really resulted in a Bayesian test (in the usual sense).

As it turns out, it does - at least if you're willing to accept a certain type of loss functions. Many thanks to Zen, who provided references to two papers that establish a connection between HPD regions and hypothesis testing:

I'll try to summarize them here, for future reference. In analogue with the example in the original question, I'll treat the special case where the hypotheses are $$H_0: \theta\in\Theta_0=\{\theta_0\}\qquad \mbox{and}\qquad H_1: \theta\in\Theta_1=\Theta\backslash \Theta_0,$$ where $\Theta$ is the parameter space.

Pereira & Stern proposed a method for testing said hypotheses without having to put prior probabilities on $\Theta_0$ and $\Theta_1$.

Let $\pi(\cdot)$ denote the density function of $\theta$ and define $$T(x)=\{ \theta:\pi(\theta|x)>\pi(\theta_0|x)\}.$$

This means that $T(x)$ is a HPD region, with credibility $P(\theta\in T(x)|x)$.

The Pereira-Stern test rejects $\Theta_0$ when $P(\theta\notin T(x)|x)$ is "small" ($<0.05$, say). For a unimodal posterior, this means that $\theta_0$ is far out in the tails of the posterior, making this criterion somewhat similar to using p-values. In other words, $\Theta_0$ is rejected at the $5~\%$ level if and only if it is not contained the in $95~\%$ HPD region.

Let the test function $\varphi$ be $1$ if $\Theta_0$ is accepted and $0$ if $\Theta_0$ is rejected. Madruga et al. proposed the loss function $$L(\theta,\varphi,x) = \begin{cases} a(1-\mathbb{I}(\theta\in T(x)), & \mbox{if } \varphi(x)=0 \\ b+c\mathbb{I}(\theta\in(T(x)), & \mbox{if } \varphi(x)=1, \end{cases}$$ with $a,b,c>0$.

Minimization of the expected loss leads to the Pereira-Stern test where $\Theta_0$ is rejected if $P(\theta\notin T(x)|x)<(b+c)/(a+c).$

So far, all is well. The Pereira-Stern test is equivalent to checking whether $\theta_0$ is in an HPD region and there is a loss function that generates this test, meaning that it is founded in decision theory.

The controversial part though is that the loss function depends on $x$. While such loss functions have appeared in the literature a few times, they don't seem to be generally accepted as being very reasonable.

For further reading on this topic, see a list of papers that cite the Madruga et al. article.

Update October 2012:

I wasn't completely satisfied with the above loss function, as its dependence on $x$ makes the decision-making more subjective than I would like. I spent some more time thinking about this problem and ended up writing a short note about it, posted on arXiv earlier today.

Let $q_{\alpha}(\theta|x)$ denote the posterior quantile function of $\theta$, such that $P(\theta\leq q_{\alpha}(\theta|x))=\alpha$. Instead of HPD sets we consider the central (equal-tailed) interval $(q_{\alpha/2}(\theta|x),q_{1-\alpha/2}(\theta|x))$. To test $\Theta_0$ using this interval can be justified in the decision-theoretic framework without a loss function that depends on $x$.

The trick is to reformulate the problem of testing the point-null hypothesis $\Theta_0=\{\theta_0\}$ as a three-decision problem with directional conclusions. $\Theta_0$ is then tested against both $\Theta_{-1}=\{\theta:\theta<\theta_0\}$ and $\Theta_{1}=\{\theta:\theta>\theta_0\}$.

Let the test function $\varphi=i$ if we accept $\Theta_i$ (note that this notation is the opposite of that used above!). It turns out that under the weighted $0-1$ loss function $$L_2(\theta,\varphi) = \begin{cases} 0, & \mbox{if } \theta\in\Theta_i\mbox{ and }\varphi=i, \quad i\in \{-1,0,1\}, \\ \alpha/2, & \mbox{if } \theta\notin\Theta_0 \mbox{ and }\varphi=0,\\ 1, & \mbox{if } \theta\in\Theta_{i}\cup\Theta_0 \mbox{ and }\varphi=-i,\quad i\in\{-1,1\},\end{cases}$$ the Bayes test is to reject $\Theta_0$ if $\theta_0$ is not in the central interval.

This seems like a quite reasonable loss function to me. I discuss this loss, the Madruga-Esteves-Wechsler loss and testing using credible sets further in the manuscript on arXiv.

-
(I'm marking this as a community wiki) –  MånsT Jul 26 '12 at 12:21
When you say "To arrive at the Pereira-Stern test we must minimize the expected posterior loss", well, actually we do that in any Bayesian decision procedure. The difference here is that the loss function depends on data (as you pointed out), which is not standard. Normaly we have $L : \{Parameter\; Space\} \times \{Actions\} \to \mathbb{R}$. –  Zen Jul 26 '12 at 12:51
@Zen: Yes, of course, I phrased that wrongly. Thanks for pointing that out. :) –  MånsT Jul 26 '12 at 13:23
@MånsT: (+1) This is an interesting answer. I very much respect the fact you chose to mark this as CW in this instance, but I wish you wouldn't have. :-) –  cardinal Jul 26 '12 at 13:29
OK, e-mail sent! –  Zen Sep 7 '12 at 16:27

I believe the paper http://ba.stat.cmu.edu/vol03is01.php can clarify most of our discussion here.

-
Thank you, @Carlos! The link doesn't seem to work right now, but I guess that it leads to your 2008 paper in Bayesian Analysis with Stern and Wechsler. I found that a very interesting read! –  MånsT Jul 26 '12 at 14:14
Dear MånsT: Bayesian Analysis moved to Project Euclid. Prof. Carlos paper is here: projecteuclid.org/… –  Zen Jul 26 '12 at 21:15

I coincidentally read your arXiv paper prior to coming to this question and already wrote a blog entry on it (scheduled to appear on October, 08). To sum up, I find your construction of theoretical interest, but also think it is too contrived to be recommended, esp. as it does not seem to solve the point-null hypothesis Bayesian testing problem, which traditionally requires to put some prior mass on the point-null parameter value.

To wit, the solution you propose above (in the October update) and as Theorem 2 in your arXiv paper is not a valid test procedure in that $\varphi$ takes three values, rather than the two values that correspond to accept/reject. Similarly, the loss function you use in Theorem 3 (not reproduced here) amounts to testing a one-sided hypothesis, $H_0: \theta\le\theta_0$, rather than a point-null hypothesis $H_0: \theta= \theta_0$.

My major issue however is that it seems to me that both Theorem 3 and Theorem 4 in your arXiv paper are not valid when $H_0$ is a point-null hypothesis, i.e. when $\Theta_0=\{\theta_0\}$, with no prior mass.

-
Thanks (+1) for your comments! I very much look forward to reading your blog post. :) As you point out, Theorems 3 and 4 are concerned with composite hypotheses only. The $1-\alpha/2$ in Theorem 2 is a misprint. It should read $\alpha/2$, in which case $\varphi=0$ when $\alpha/2<\mbox{min}(P(\Theta_{-1}),P(\Theta_1))$, which happens when $\theta_0$ is in the the credible interval. I'll change this in the arXiv manuscript as soon as possible! –  MånsT Oct 4 '12 at 22:40
You are right (+1!), I was thinking of the inequality the other way! In the arXiv document, the central inequality is written the wrong way. i.e. one should accept $H_0$ iff –  Xi'an Oct 5 '12 at 6:49
That's good to hear :) The updated manuscript (with Thm 2 corrected) will be on arXiv on Monday. I'll make the assumption that $\Theta_0$ is not point-null in Thm 4 explicit as well. –  MånsT Oct 5 '12 at 6:51
Just make sure to clarify the proof of Theorem 2 in the arXiv document: the displayed inequality is written the wrong way. i.e. one should accept $H_0$ iff $P(\theta\in\Theta_i|x)>\alpha/2$, not the opposite! –  Xi'an Oct 5 '12 at 6:58

You can use a credible interval (or HPD region) for Bayesian hypothesis testing. I don't think it is common; though, to be fair I do not see much nor do I use formal Bayesian Hypothesis testing in practice. Bayes factors are occasionally used (and in Robert's "Bayesian Core" somewhat lauded) in hypothesis testing set up.

-
Cheers @Fraijo! Could you perhaps elaborate a bit on how your answer differ from that of Michael Chernick? –  MånsT Jul 10 '12 at 11:15
I do not think the use of Bayes factors for testing hypothesis is "occasional", see for example this reference. –  user10525 Jul 10 '12 at 11:19
@MånsT in his follow up the process Michael describes seems to be a Bayes Factor test. Essentially you create two models with different priors based on your hypothesis and then compare the the probability of the data set based on those priors. The reference Procrasinator posted gives a quick review of this. –  Fraijo Jul 10 '12 at 16:31
@Procrastinator I said occasional only because in my industry I see few people using Bayesian methods, let alone using Bayesian methods for testing hypothesis. Personally I use Bayes factors to check my models for sensitivity to the prior, which I suppose is a form of hypothesis testing. –  Fraijo Jul 10 '12 at 16:37
@Fraijo: I realise that I wasn't clear enough. I'm familiar with the general theory of Bayesian testing, it's decision theoretic foundation and Bayes factors. What I wondered was if the tests using credible regions that you mentioned in your answer in any way differed from those that Michael proposed (in his answer, not in the comments following it). –  MånsT Jul 10 '12 at 20:01
show 1 more comment

A credible region is just a region where the integral of the posterior density over the region is a specified probability e.g. 0.95. One way to form a Bayesian hypothesis test is to see whether or not the null hypothesized value(s) of the parameter(s) fall in the credible region. In this way we can have a similar 1-1 correspondence between hypothesis tests and credible regions just like the frequentists do with confidence intervals and hypothesis tests. But this is not the only way to do hypothesis testing.

-
Are this kind of ad hoc Bayesian tests often used in practice? –  MånsT Jul 5 '12 at 14:16
@MansT I don't think so. I think that usually Bayesians put prior odds on the null hypothesis being true and then based on the data construct posterior odds. If the posterior odds are storngly against the null hypothesis then it is rejected. I am not the best person to ask though since I do not do Bayesian inference very often. –  Michael Chernick Jul 5 '12 at 14:31
The test described by Michael is credited to Lindley by Zellner in his book on Bayesian econometrics. –  Zen Jul 10 '12 at 2:11
Thanks @Zen (+1). The procedure is described in Section 10.2 of Zellners book, if anyone is interested. It seems that it can be useful when the prior information is "vague or diffuse". This is an interesting approach (+1 to @Michael), but it doesn't seem to be founded in Bayesian decision theory (other than in a roundabout way). –  MånsT Jul 10 '12 at 8:06
Dear MånsT, take a look at these papers: mdpi.org/entropy/papers/e1040099.pdf w.ime.usp.br/~jstern/miscellanea/citacoes/swtest1.pdf –  Zen Jul 10 '12 at 17:42