Understand a statement about P value

Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is. [1]

I am having trouble to understand this text, especially this passage:

The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place.

Why can the P value not work backward? Is that not the point of P value? If the probability of the observed data is very extreme under assumption of nullhypothesis, we reject the nullhypothesis and assume the alternative hypothesis to be true, or am I having a mistake in thinking?

• This type of question has a long history. Check this out: royalsocietypublishing.org/doi/10.1098/rsbl.2019.0174 Feb 21, 2021 at 16:22
• "The underlying reality" is refering to the fact that you never get away from assuming the null hypothesis is true (no matter what the p value is). Feb 21, 2021 at 21:47

If your data is $$\mathcal{D}$$, and your hypothesis $$H_0$$ then the p-value is $$p = \mathbb{P}(\mathcal{D}\mid H_0)$$.

The $$p$$ value tells you the following:

If $$H_0$$ is true, how likely is the data I'm currently observing ?

So if $$p$$ is very low, it only means the data cannot easily happen in a world in which $$H_0$$ is true. This does not mean $$H_0$$ is wrong: the data itself $$\mathcal{D}$$ could be wrong. You have a choice: you either reject the theory $$H_0$$ or the data $$\mathcal{D}$$. If you instead want to compute $$\mathbb{P}(H_0\mid\mathcal{D})$$, you need to apply Bayes' rule $$\mathbb{P}(H_0\mid\mathcal{D}) \propto \mathbb{P}(\mathcal{D}\mid H_0)\mathbb{P}(H_0) =p\,\mathbb{P}(H_0)$$ "Working backwards" means that $$\mathbb{P}(H_0\mid\mathcal{D}) = \mathbb{P}(\mathcal{D}\mid H_0)$$ which is wrong in almost every scenario. To apply the formula, you need to compute $$\mathbb{P}(H_0)$$, this is what is meant by

The odds that a real effect was there in the first place

You however never have the value of $$\mathbb{P}(H_0)$$.

• Excellent answer. $\mathbb{P}(H_0\mid\mathcal{D})$ made it all clear. Thanks Feb 21, 2021 at 19:07
• "the data itself $\mathcal{D}$ could be wrong" -- even if the data is right (i.e., any measurement errors are already accounted for in the low $p$), you could still be justified in concluding that $H_0$ is true, if $\mathbb{P}(H_0)$ is sufficiently close to 1. So I disagree that "you either reject the theory $H_0$ or the data $\mathcal{D}$." Obligatory xkcd Feb 22, 2021 at 3:54