3
$\begingroup$

Suppose we have a hypothesis test: $$H_0: \theta≥\theta_0 ~~~ vs~~~ H_1:\theta<\theta_0$$

With the observation $X$, the p-value is calculated by $p = P(X|H_0)$. Which means the sum of probability for less or equal likely events.

The p-value tells us the likelihood of this set of observation to happen given the Null hypothesis. But usually, what we really want to know is how likely my hypothesis is right, given the observed data. Which, I think, should be $P(H_0|X)$.

Then I recall the Bayes' Formula, where I can derive $P(H_0|X)$ from $P(X|H_0)$. $$P(H_0|X) = \frac{P(X|H_0) ~P(H_0)}{P(X)}$$

So in this perspective, p value is merely a term in the formula. Am I wrong? If I'm right, why don't we use Bayes' Formula to fix our hypothesis test?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Welcome to Cross Validated and a +1 from me. I once wondered this, and I saw two issues.

  1. That definition of a p-value is not quite right.

$$p=P(X\ge x \vert H_o)$$

(Or something similar for a two-sided test)

With that in mind, you are not quite flipping around the conditioning to derive to posterior probability of the null hypothesis. We want to know the posterior probability of $H_0$ after looking at the data, but that p-value bring in that “or more extreme” business.

  1. What are the probabilities of the null hypothesis and the observed test statistic? I have yet to come up with a better answer than to say that both have a probability of zero. In that case, our fraction has a $\frac{0}{0}$ term.

Finally, even in a true Bayesian setting, the probability of a single point like $\mu=0$ is zero for a continuous posterior distribution, so I’m not convinced that this kind of Bayesian p-value is at all what we want. Bayesian inference can be wonderful for a lot, but it doesn’t magically make continuous distributions have positive probability measure for individual points.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer! And yes, I have noticed the problems you raised. But what if I treat $X≥x$ as an event, let's call it $A$. Then can I transform the problem into $P(H_0|A) = \frac{P(A|H_0)P(H_0)}{P(A)}$? In which case, I think, maybe we can avoid the probability zero. $\endgroup$
    – PZH
    Dec 27, 2021 at 9:31
  • 1
    $\begingroup$ 1) I still think $P(H_0)=0$, so you’d wind up with a posterior probability of zero. // 2) What would be $P(A)?$ // 3) I do not see $P(H_0\vert A)$ as particularly interesting. We use $A$ almost like a trick in frequentist statistics in order to calculate a positive p-value instead of going to, for example, an F-distribution and finding that the probability of our test statistic is zero. However, in your framework, we want to know $P(H_0\vert\text{our data})$, not $P(H_0\vert\text{our data…or data we did not see but might be more extreme}))$, which your new formula says. $\endgroup$
    – Dave
    Dec 27, 2021 at 9:43
  • $\begingroup$ @PZH Is there anything from my comment you would like edited into the answer? $\endgroup$
    – Dave
    Dec 27, 2021 at 9:50
  • $\begingroup$ Thanks for your patience again and answer accepted, Dave. I understand where the problem is and I will go check more about Bayesian inference! $\endgroup$
    – PZH
    Dec 27, 2021 at 9:53
  • $\begingroup$ While $\mu=0$ may have zero probability, it provides a convenient and accurate approximation to the more relevant case that $|\mu| < \epsilon$. There is a paper in Statistical Science by Berger and Delampady, late 1980s, about this. So it is fine to consider a non-zero prior probability for $\mu=0$,: It is a convenient approximation to the case where there is the same non-zero probability on $|\mu| < \epsilon$. $\endgroup$ Dec 27, 2021 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.