3
$\begingroup$

In hypothesis testing, one usually calculates $p$-value by assuming that null hypothesis holds. In wiki page of $p$-value, first sentence verifies my statement above. To write $p$-value using mathematical notation, I think it is the following $$p-value = P( observation| H_0 \text{ is true }).$$ But in reality, one is interested in its reverse, i.e. $$P( H_0 \text{ is true }| observation ).$$ To connect the 2 quantities, one can use Bayes' rule: $$P(H_0 \text{ is true }| observation) = \frac{P( observation| H_0 \text{ is true }) P(H_0 \text{ is true }) }{P(observation)}.$$ Does my understanding above hold?

I seldom see reasoning above involving $p$-value. So it must be false somewhere in my reasoning. But I could not find it.

$\endgroup$
  • $\begingroup$ Maybe you should check out what Bayes Factors are. Your formula is not yet correct. $\endgroup$ – LuckyPal Aug 26 at 9:03
  • $\begingroup$ @LuckyPal my bad. Edited. $\endgroup$ – Idonknow Aug 26 at 9:07
  • $\begingroup$ ""$\text{p-value} = P(\text{observation}| H_0 \text{ is true })$" -- no, the p-value is not the probability of what you observed given H0 is true. $\endgroup$ – Glen_b Aug 26 at 12:30
3
$\begingroup$

The approach that you describe is actually used in a nature publication as an illustration for the (often unknown) complexity involved when interpreting standard p-values. See here: Nuzzo, 2014.

But I agree with Dayne: this approach is not justied by the frequentist paradigm, since in the frequentist sense parameters are not random. If you go Bayesian, then this perfectly legit. However, Bayesians have developed other nice tools that are preferred, like Bayes Factors or posterior predictive p-values.

$\endgroup$
  • $\begingroup$ Great! Would love to know how Bayesians handle this. $\endgroup$ – Dayne Aug 26 at 11:26
  • $\begingroup$ @Dayne as far as I am aware, the majority of recent research uses Bayes Factors. A good introduction paper could be Held and Ott (2018): annualreviews.org/doi/full/10.1146/… $\endgroup$ – LuckyPal Aug 26 at 12:01
1
$\begingroup$

Good thinking, but here's the problem in your explanation:

"$H_0$ is true" is not a random variable. In the frequentist approach, parameters are (un)known fixed numbers. So the hypothesis is either right or wrong. Therefore the idea of

$P(H_0$ is true $|$ observation$)$ is itself incorrect.

Edit: Based on another answer below, one more thing comes to mind. Even if you consider parameters as random (i.e., Bayesian approach), you can relate $P(H_0$ is true $|$ observation$)$ to MLE. In MLE you maximize something subject to parameters. Basically rather than assuming some value of the parameter in $H_0$, you just assume that distribution comes from a particular family (for example, you may assume distribution is Normal). And then you maximize likelihood function subject to parameters. So an MLE can be though of as finding the $H_0$ for which $P(H_0$ is true $|$ observation$)$ is maximum among various possible hypotheses.

$\endgroup$
  • $\begingroup$ Then I think the other half of this is that, to a Bayesian, the data aren’t a random variable, right? $\endgroup$ – Dave Aug 26 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.