Understand a statement about P value

Question

Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is. [1]

[1] https://www.nature.com/news/scientific-method-statistical-errors-1.14700

I am having trouble to understand this text, especially this passage:

The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place.

Why can the P value not work backward? Is that not the point of P value? If the probability of the observed data is very extreme under assumption of nullhypothesis, we reject the nullhypothesis and assume the alternative hypothesis to be true, or am I having a mistake in thinking?

Answer 1

If your data is $\mathcal{D}$, and your hypothesis $H_0$ then the p-value is $ p = \mathbb{P}(\mathcal{D}\mid H_0)$.

The $p$ value tells you the following:

If $H_0$ is true, how likely is the data I'm currently observing ?

So if $p$ is very low, it only means the data cannot easily happen in a world in which $H_0$ is true. This does not mean $H_0$ is wrong: the data itself $\mathcal{D}$ could be wrong. You have a choice: you either reject the theory $H_0$ or the data $\mathcal{D}$. If you instead want to compute $\mathbb{P}(H_0\mid\mathcal{D})$, you need to apply Bayes' rule $$ \mathbb{P}(H_0\mid\mathcal{D}) \propto \mathbb{P}(\mathcal{D}\mid H_0)\mathbb{P}(H_0) =p\,\mathbb{P}(H_0)$$ "Working backwards" means that $\mathbb{P}(H_0\mid\mathcal{D}) = \mathbb{P}(\mathcal{D}\mid H_0)$ which is wrong in almost every scenario. To apply the formula, you need to compute $\mathbb{P}(H_0)$, this is what is meant by

The odds that a real effect was there in the first place

You however never have the value of $\mathbb{P}(H_0)$.

Answer 2

Answering without equations:

p value is a measure of surprise. Given that the null hypothesis is true (Design of the experiment) what is the chances that you stumble upon a value at least this extreme in your data.

You compute p on the test data. You never can include the actual data. If you can get the actual data there is no need of hypothesis testing itself in the first place. So the test data even is a random sample and it can come from any part of the actual distribution. Given this randomness you cannot use the p value that you computed say that there was just a p*100% chance of his result being a false alarm.

Hope this answers your question.