5
$\begingroup$

I was told that the framework of Bayesian Statistics has no concept of statistical hypothesis testing or confidence intervals.

How does this make sense? Bayesian statistics only says that we iteratively update our prior belief based on evidence. Consider the following example I got from my textbook where we use Bayesian statistics to model the behaviour of some variable C.

t e [0, 20] C ~ Poi(lambda) lambda = { lambda1, if t>10; lambda2, otherwise } lambda1 ~ exp(alpha1) lambda2 ~ exp(alpha2) Run MCMC to compute posteriors of lambda1 and lambda 2

The parameter lambda takes on two different behaviours: one before 10 seconds and another after 10 seconds.

Now we'll do a t-test on to check whether the distributions of lambda1 are statistically different from lambda2 to check whether these two distributions are indeed different. I do not know whether doing this t-test makes sense, I was told we cannot do statistical testing in a Bayesian framework. We are supposed to use visualizations and reject them ourselves.

Can anyone confirm whether this is true? Is statistical testing invalid from a Bayesian perspective?

$\endgroup$
  • 1
    $\begingroup$ A quick search for "bayesian hypothesis testing" in your search engine of choice should prove enlightening. $\endgroup$ – Alexis Feb 20 '16 at 1:39
  • $\begingroup$ Yeah, it seems to give a rather ad-hoc answer. Most results say "no" but I'm hoping for a clear answer by a human. $\endgroup$ – user46925 Feb 20 '16 at 1:56
  • 1
    $\begingroup$ It may help you to more rigorously define what "statistically different" actually means in this context ( I do not know what it means here). I also don't understand your model since you have some latent variable $t$ that is not conditioned on anything. As for the more general question regarding statistical testing in a Bayesian setting; there is no concrete answer. Since different people will still have different definitions as to what constitutes "statistical hypothesis testing" you would need to define , much more rigorously, exactly what type of testing methodology you have in mind. $\endgroup$ – Zachary Blumenfeld Feb 20 '16 at 2:15
  • 2
    $\begingroup$ You can have things that are tests and intervals, but they have a somewhat different interpretation (e.g. a Bayesian interval isn't a confidence interval in the sense that it guarantees a coverage probability; it does something else). $\endgroup$ – Glen_b Feb 20 '16 at 3:51
  • 1
    $\begingroup$ What are the hypotheses that you are attempting to test via a t-test? Are you trying to test $H_0:\;\alpha_1 =\alpha_2$ vs $H_1:\;\alpha_1 \neq \alpha_2$. If so, are these hyper-parameters estimated in the model or are they given a priori? To say "Bayesian Statistics has no concept of statistical hypothesis testing" requires a specific (frequentist) definition of hypothesis testing that not everyone will agree on. To better understand this, it would be helpful to work your example in both a Bayesian and frequentist setting and compare the results. $\endgroup$ – Zachary Blumenfeld Feb 20 '16 at 3:56
9
$\begingroup$

I am surprised at the textbook statement as testing hypotheses and comparing models are a most fundamental feature of Bayesian analysis, with a wide variety of possible resolutions that exposes the multiple and sometimes incompatible facets of the problem.

(excerpt from our book, Bayesian essentials with R, Chapter 2, p.29:)

For the null and alternative hypotheses $$ H_0:\ \theta \in > \Theta_0\text{ and }H_a:\ \theta \in \Theta_1 $$ and under the loss function $$ L_{a_0,a_1} (\theta ,d) = \begin{cases} a_0 & \hbox{if}\quad \theta \in \Theta_0\quad\hbox{and}\quad d=0\,, \cr a_1 & \hbox{if}\quad \theta \in \Theta_1\quad\hbox{and}\quad d=1\,, \cr 0 & \hbox{otherwise.} \cr \end{cases} $$ where $d=0$ denotes the rejection of $H_0$, the Bayes optimal decision associated with a prior $\pi$ is given by $$ \delta^\pi(x) = \begin{cases} 1 & \hbox{if}\quad \mathbb{P}^\pi(\theta \in \Theta_0|x)>a_1\big/{a_0+a_1}, \cr 0 & \hbox{otherwise.}\cr \end{cases} $$ For this class of losses, the null hypothesis $H_0$ is rejected when the posterior probability of $H_0$ is too small, the acceptance level $a_1/(a_0+a_1)$ being determined by the choice of $(a_0,a_1)$.

The Bayesian paradigm allows for testing and model comparison, to a larger extent than other statistical paradigms, I would say. What may sound at first like a drawback is that all aspects of this decision have to be spelled out, from the specification of the sampling models under the null and under the alternative hypotheses (which explains why I cannot spell out a strict distinction between hypothesis testing and model choice), to the construction of prior distributions on the parameters of both sampling models, to prior weights on the prior likelihood of both hypotheses, to the impact of selecting the "wrong "model".

Outside this Neyman-Pearson decision framework, there are further Bayesian resolutions of the testing issue, like

  1. the substitute Bayes factor$$\dfrac{\mathbb{P}^\pi(\theta \in \Theta_0|x)}{\mathbb{P}^\pi(\theta \in \Theta_1|x)}\Big/\dfrac{\mathbb{P}^\pi(\theta \in \Theta_0)}{\mathbb{P}^\pi(\theta \in \Theta_1)}$$that avoids selecting the prior weights but which are not free from foundational drawbacks;
  2. information criteria like BIC, DIC, WAIC and Aitkin's integrated likelihood;
  3. score functions and related information approaches;
  4. posterior predictive assessments like the posterior $p$-value of Gelman et al. and others;
  5. Evans' relative belief;
  6. divergence criteria like ABC$\mu$;
  7. model averaging;
  8. embedding models like our mixture representation.
$\endgroup$
  • 1
    $\begingroup$ I just ordered the book through Amazon. $\endgroup$ – Antoni Parellada Feb 28 '16 at 19:21
7
$\begingroup$

No.

Bayesian statistics has a concept of hypothesis testing. From Wagenmakers and Grünweld:

A Bayesian hypothesis test (Jeffreys, 1961) proceeds by contrasting two quantities: the probability of the observed data $D$ given $H_{0}$ (i.e., $\theta = \frac{1}{2}$) and the probability of the observed data $D$ given $H_{1}$ (i.e., $\theta \ne \frac{1}{2}$). The ratio $B_{01} = p(D|H_{0})/p(D|H_{1})$ is the Bayes factor, and it quantifies the evidence that the data provide for $H_{0}$ vis-à-vis $H_{1}$.

Eric-Jan Wagenmaker and Peter Grünweld. 2006. A Bayesian Perspective on Hypothesis Testing A Comment on Killeen (2005). Psychological Science. 17(7):641–642.

$\endgroup$
  • $\begingroup$ Is it statistically invalid to apply a traditional t-test as I did in my example? $\endgroup$ – user46925 Feb 20 '16 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy