I understand that the $p$-value is the conditional probability of observing the test statistic or something more extreme given that the null hypothesis is true. I have read the great explanation by @user28 in this post: What is the meaning of p values and t values in statistical tests? However, do large $p$-values say anything? Does a larger $p$-value lend greater support to the null hypothesis? If I set rejection region to be $<0.05$, then does it make a difference if I get $p$-value $0.06$ or $0.99$? (After all, $0.05$ is arbitrary, and $0.06$ is so close to being rejected that if I arbitrarily set $0.05$ as $0.1$ instead, the null hypothesis would have been rejected.) Can one make any statistical use of a non-rejecting $p$-value?
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ One example is mentioned at the end of this answer: Fisher's re-examination of Mendel's pea experiments. $\endgroup$– whuber ♦Jun 23, 2014 at 14:37
-
1$\begingroup$ You cannot conclude that a test statistic supports the null hypothesis when the calculation of that statistic is conditional on the assumption that the null is true. That is to say, if $E$ and $H_0$ are events, you cannot say $H_0$ is true if $\Pr[E \mid H_0]$ is "large." $\endgroup$– heropupJun 23, 2014 at 16:04
-
$\begingroup$ @heropup By the same reasoning we can't say that $H_0$ is false if $p$-value is very small? $\endgroup$– ado sarDec 18, 2022 at 20:14
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
How you should 'use' the p-value depends on how you have designed your study with regard to the analyses you will run. I discuss two different philosophies about p-values in my answer here: When to use Fisher and Neyman-Pearson framework? You may find it helpful to read that. If you have, for example, run a power analysis and intend to use the p-value to make a final decision, you should not use close to the line ('marginally significant') as a meaningful category. It is fine to use a different alpha than $0.05$ (such as $0.10$), but once you decided on it and set your study up accordingly, you should stick with it.
In addition, you cannot use a large p-value as evidence for the null hypothesis. I discussed that idea in my answer here: Why do statisticians say a non-significant result means "you cannot reject the null" as opposed to accepting the null hypothesis? Reading that answer may be helpful to you as well.
answered Jun 23, 2014 at 15:00
$\begingroup$
$\endgroup$
4
In my view, everything boils down to assumptions, that is, how well the models fits them. If it does agree with all of them, treat the p-value as a probability. Then you can compare p-value of 0.06 with 0.99 by concluding which of the two are more likely. Also, a lot depend on circumstances: in some cases, marginal significance shouldn't be ignored, because as you stated, rejection region can be set rather arbitrarily. But if the models satisfies the assumptions, then you should not seek to reject your hypothesis to some arbitrary level but rather investigate, how likely is the outcome that you got.
-
1$\begingroup$ It is problematic to (mis)interpret a p-value as a probability in this sense. You seem to be applying a Bayesian-like approach but without the usual care needed to consider and defend a prior distribution. What justification can you provide to support your recommendations? $\endgroup$– whuber ♦Jun 23, 2014 at 14:40
-
$\begingroup$ well, I was having a linear regression in mind satisfying all the conventional assumptions of the models and as a result regression coefficients having normal distributions. I suppose in this case it would be safe to treat p-value as a probability, wouldn't it? $\endgroup$– SarunasJun 23, 2014 at 14:51
-
1$\begingroup$ No, it would not. There is a lot of debate about p-values, but practically everybody who has entered into it (on either side) has noted that this use of p-values is invalid. Comparing p-values does not determine whether one model is more or less likely than another. $\endgroup$– whuber ♦Jun 23, 2014 at 14:54
-
1$\begingroup$ Yes, you are right. I started googling and two source made things more clear: Hubbard, R. (2004). Alphabet soup: Blurring the distinctions between p's and α's in psychological research. Theory and Psychology, 14 (3), 295-327; Hubbard, R., & Lindsay, R.M. (2008). Why P values are not a useful measure of evidence in statistical significance testing. Theory and Psychology, 18 (1), 69-88. Obviously, I had an incorrect view of the concept! $\endgroup$– SarunasJun 23, 2014 at 15:16
$\begingroup$
$\endgroup$
It is true that the acceptance range for the p-value of a hypothesis test is rather arbitrary, but nevertheless a lower p-value means that the test result can be accepted with more certainty, because the p-value essentially defines the confidence interval for the estimate, so a narrower confidence interval should be regarded more significant for the test.
