Most scientists would look at his original P value of 0.01 and say that there was just a 1% chance of his result being a false alarm. But they would be wrong. The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place. To ignore this would be like waking up with a headache and concluding that you have a rare brain tumour — possible, but so unlikely that it requires a lot more evidence to supersede an everyday explanation such as an allergic reaction. The more implausible the hypothesis — telepathy, aliens, homeopathy — the greater the chance that an exciting finding is a false alarm, no matter what the P value is. [1]
[1] https://www.nature.com/news/scientific-method-statistical-errors-1.14700
I am having trouble to understand this text, especially this passage:
The P value cannot say this: all it can do is summarize the data assuming a specific null hypothesis. It cannot work backwards and make statements about the underlying reality. That requires another piece of information: the odds that a real effect was there in the first place.
Why can the P value not work backward? Is that not the point of P value? If the probability of the observed data is very extreme under assumption of nullhypothesis, we reject the nullhypothesis and assume the alternative hypothesis to be true, or am I having a mistake in thinking?