5
$\begingroup$

Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is. [1]

[1] https://www.nature.com/news/scientific-method-statistical-errors-1.14700

I am having trouble to understand this text, especially this passage:

The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place.

Why can the P value not work backward? Is that not the point of P value? If the probability of the observed data is very extreme under assumption of nullhypothesis, we reject the nullhypothesis and assume the alternative hypothesis to be true, or am I having a mistake in thinking?

$\endgroup$
2
  • 1
    $\begingroup$ This type of question has a long history. Check this out: royalsocietypublishing.org/doi/10.1098/rsbl.2019.0174 $\endgroup$ – Brant Inman Feb 21 at 16:22
  • 2
    $\begingroup$ "The underlying reality" is refering to the fact that you never get away from assuming the null hypothesis is true (no matter what the p value is). $\endgroup$ – Alexis Feb 21 at 21:47
7
$\begingroup$

If your data is $\mathcal{D}$, and your hypothesis $H_0$ then the p-value is $ p = \mathbb{P}(\mathcal{D}\mid H_0)$.

The $p$ value tells you the following:

If $H_0$ is true, how likely is the data I'm currently observing ?

So if $p$ is very low, it only means the data cannot easily happen in a world in which $H_0$ is true. This does not mean $H_0$ is wrong: the data itself $\mathcal{D}$ could be wrong. You have a choice: you either reject the theory $H_0$ or the data $\mathcal{D}$. If you instead want to compute $\mathbb{P}(H_0\mid\mathcal{D})$, you need to apply Bayes' rule $$ \mathbb{P}(H_0\mid\mathcal{D}) \propto \mathbb{P}(\mathcal{D}\mid H_0)\mathbb{P}(H_0) =p\,\mathbb{P}(H_0)$$ "Working backwards" means that $\mathbb{P}(H_0\mid\mathcal{D}) = \mathbb{P}(\mathcal{D}\mid H_0)$ which is wrong in almost every scenario. To apply the formula, you need to compute $\mathbb{P}(H_0)$, this is what is meant by

The odds that a real effect was there in the first place

You however never have the value of $\mathbb{P}(H_0)$.

$\endgroup$
2
  • $\begingroup$ Excellent answer. $\mathbb{P}(H_0\mid\mathcal{D})$ made it all clear. Thanks $\endgroup$ – Long Nguyen Feb 21 at 19:07
  • $\begingroup$ "the data itself $\mathcal{D}$ could be wrong" -- even if the data is right (i.e., any measurement errors are already accounted for in the low $p$), you could still be justified in concluding that $H_0$ is true, if $\mathbb{P}(H_0)$ is sufficiently close to 1. So I disagree that "you either reject the theory $H_0$ or the data $\mathcal{D}$." Obligatory xkcd $\endgroup$ – nanoman Feb 22 at 3:54
0
$\begingroup$

Answering without equations:

p value is a measure of surprise. Given that the null hypothesis is true (Design of the experiment) what is the chances that you stumble upon a value at least this extreme in your data.

You compute p on the test data. You never can include the actual data. If you can get the actual data there is no need of hypothesis testing itself in the first place. So the test data even is a random sample and it can come from any part of the actual distribution. Given this randomness you cannot use the p value that you computed say that there was just a p*100% chance of his result being a false alarm.

Hope this answers your question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.