I know this topic was explained many times, but I need help with just a concrete wording.
p-value is the probability of obtaining at least as extreme data (or test statistic for convenience), calculated UNDER the true null hypothesis.
That is - "no effect", "by chance" (nothing else by chance could create the effect) was assumed, and then we looked at the collected data, to see, if they "support" or "match" that claim or not. In other words, it's Probability(data | true null hypothesis).
Let's play with words about the p-value:
Probability, that as extreme or more extreme data were collected provided that H0 was actually true.
Probability, that as extreme or more extreme data were collected provided that nothing else but chance "operated" or "acted"
Probability, that the observed data can be explained by chance alone (we assume that nothing happened, but such data arrived)
The result is explainable by chance alone.
At which point it becomes incorrect? To me, all sound exactly the same. I don't feel the subtle differences. By chance = nothing acts = true H0.
At the same time, the article: Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations says:
#2 The P value for the null hypothesis is the probability that chance alone produced the observed association; for example, if the P value for the null hypothesis is 0.08, there is an 8 % probability that chance alone produced the association.
No! This is a common variation of the first fallacy and it is just as false. To say that chance alone produced the observed association is logically equivalent to asserting that every assumption used to compute the P value is correct, including the null hypothesis. Thus to claim that the null P value is the probability that chance alone produced the observed association is completely backwards: The P value is a probability computed assuming chance was operating alone. The absurdity of the common backwards interpretation might be appreciated by pondering how the P value, which is a probability deduced from a set of assumptions (the statistical model), can possibly refer to the probability of those assumptions.
OK, so saying, that p-value is probability of obtaining the data "by chance" is NOT the same, as saying that "such or more extreme data were obtained assuming that only chance operated"?
At the same time, a book that I am just reading, "Understanding Regression Analysis: A Conditional Distribution Approach" by Westfall & Ariar, which is very strict about the misunderstanding of p-values, claims, that p answers the question: "Is this result explainable by chance alone?"
Google Books shows the following, but I can hardly understand how these subtle wordings are correct, if the "by chance alone" is indicated as WRONG in the cited article as misinterpretation?
and from page 84:
When the only reason for a difference between statistical estimates is chance alone, and not any systematic effect, then that difference is said to be explained by chance alone.
Definition of “Explainable by chance alone” When a difference between statistical estimates is within a typical range of differences that are explained by chance alone, then that difference is said to be explainable by chance alone
Another book "AP Statistics Premium: With 9 Practice Tests" by Sternstein says:
All the authors are PhD in statistics, all say similar (to me) things. Who is then correct!?
I know, that p-value is a conditional probability about the DATA, not the hypothesis. We assume true H0 and check the data we got under this assumptions.
Isn't then THE SAME as saying that "how probable was to get such data under the true H0" = "how probable was to get such data only by chance" = "were the data explainable by chance alone"?
Please help, where's this subtle moment I miss?