2
$\begingroup$

I know this topic was explained many times, but I need help with just a concrete wording.

p-value is the probability of obtaining at least as extreme data (or test statistic for convenience), calculated UNDER the true null hypothesis.

That is - "no effect", "by chance" (nothing else by chance could create the effect) was assumed, and then we looked at the collected data, to see, if they "support" or "match" that claim or not. In other words, it's Probability(data | true null hypothesis).

Let's play with words about the p-value:

  1. Probability, that as extreme or more extreme data were collected provided that H0 was actually true.

  2. Probability, that as extreme or more extreme data were collected provided that nothing else but chance "operated" or "acted"

  3. Probability, that the observed data can be explained by chance alone (we assume that nothing happened, but such data arrived)

  4. The result is explainable by chance alone.

At which point it becomes incorrect? To me, all sound exactly the same. I don't feel the subtle differences. By chance = nothing acts = true H0.

At the same time, the article: Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations says:

#2 The P value for the null hypothesis is the probability that chance alone produced the observed association; for example, if the P value for the null hypothesis is 0.08, there is an 8 % probability that chance alone produced the association.

No! This is a common variation of the first fallacy and it is just as false. To say that chance alone produced the observed association is logically equivalent to asserting that every assumption used to compute the P value is correct, including the null hypothesis. Thus to claim that the null P value is the probability that chance alone produced the observed association is completely backwards: The P value is a probability computed assuming chance was operating alone. The absurdity of the common backwards interpretation might be appreciated by pondering how the P value, which is a probability deduced from a set of assumptions (the statistical model), can possibly refer to the probability of those assumptions.

OK, so saying, that p-value is probability of obtaining the data "by chance" is NOT the same, as saying that "such or more extreme data were obtained assuming that only chance operated"?

At the same time, a book that I am just reading, "Understanding Regression Analysis: A Conditional Distribution Approach" by Westfall & Ariar, which is very strict about the misunderstanding of p-values, claims, that p answers the question: "Is this result explainable by chance alone?"

Google Books shows the following, but I can hardly understand how these subtle wordings are correct, if the "by chance alone" is indicated as WRONG in the cited article as misinterpretation?

  1. enter image description here

and from page 84:

When the only reason for a difference between statistical estimates is chance alone, and not any systematic effect, then that difference is said to be explained by chance alone.

and 85

Definition of “Explainable by chance alone” When a difference between statistical estimates is within a typical range of differences that are explained by chance alone, then that difference is said to be explainable by chance alone

Another book "AP Statistics Premium: With 9 Practice Tests" by Sternstein says:

enter image description here

All the authors are PhD in statistics, all say similar (to me) things. Who is then correct!?

I know, that p-value is a conditional probability about the DATA, not the hypothesis. We assume true H0 and check the data we got under this assumptions.

Isn't then THE SAME as saying that "how probable was to get such data under the true H0" = "how probable was to get such data only by chance" = "were the data explainable by chance alone"?

Please help, where's this subtle moment I miss?

Improve this question
$\endgroup$
9
  • $\begingroup$ Hypothesis testing involves assuming a (idealised) statistical model and defining a null hypothesis. Your second paragraph is the appropriate definition. A p-value can be regarded as useful for addressing the question: Is the data consistent with the null hypothesis? $\endgroup$
    – Graham Bornholt
    Sep 12, 2022 at 4:09
  • $\begingroup$ Although instead of saying "UNDER the true null hypothesis", it would be better to say "assuming the null hypothesis (and the associated model) is correct". $\endgroup$
    – Graham Bornholt
    Sep 12, 2022 at 4:10
  • 1
    $\begingroup$ Your third paragraph is incorrect. Why do you say 'no effect'? Why do you say 'by chance'? $\endgroup$
    – Graham Bornholt
    Sep 12, 2022 at 4:22
  • $\begingroup$ Thank you. I say "no effect" because that's typically the statement of the null hypothesis. This is equivalent to say "by chance", which means - "only random changes did this, no the effect specified by the alternative hypothesis". This is repeated by all books, please note. The only difference is the use of words "probability of". $\endgroup$
    – Khalim321
    Sep 12, 2022 at 5:07
  • 1
    $\begingroup$ The phrase "explainable by chance alone" is a great way to present insignificant results to the lay public as well as to scientists. It simply means "capable of being explained by chance alone." It certainly does not imply that the results are explained by chance alone. And yes, if the null model is the true model, then any difference we see is in fact explained by chance alone. Of course, we usually do not know if the null model is the correct one, so that's we we say that insignificant results are explainable by chance alone. $\endgroup$
    – BigBendRegion
    Apr 12, 2023 at 21:53

1 Answer 1

Reset to default
0
$\begingroup$

Let's say that we did a t-test of any continuous real world measure was zero. No continuous measure is really zero to the millionth digit of 0.0000000... but the null hypothesis states it is zero no matter how many digits are in regard.

So we know beforehand, that the (two-tailed) null hypothesis is wrong! Bear with me, we test against the null of which we know, that is wrong. Surprisingly, there is no logic flaw in that.

When you bare that in mind, you can hopefully see how

there is an 8 % probability that chance alone produced the association.

Is wrong, because we know from before, that the probability of that is zero.

On the other hand

saying that "such or more extreme data were obtained assuming that only chance operated"?

Does not give the null any actual probability but just makes a counterfactual assumption.

I hope this point of view makes a difference (not necessarily all differences) visible.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Perfect, thank you! Now I understand it. In fact, the h0 could be also true a priori, like in non-inferiority studies, where true H0 is indeed possible, but the frequentist framework states only it can be 0% or 100%, thus no probability statement can be made about it, only about the sampling process, that "samples under this hypothesis". So if an author of a book says equates "by chance" with h0, then any statement starting from "the probability than only chance did something" is equal to claiming "the probability of the null hypothesis", which is here meaningless. I got it! Thank you. $\endgroup$
    – Khalim321
    Sep 12, 2022 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.