Skip to main content
Cross Validated

Return to Answer

Cut incorrect part of answer, per comments
Source Link
virtuolie
virtuolie
  • 642
  • 4
  • 11

Suppose you collect data and compute a $p$-value without first selecting a Type I error rate. Then, observing the $p$-value, you select a rate specifically so that it will be statistically significant. If you then proclaim, "My result is statistically significant!" that statement alone contains no inferentially-useful information. What makes that statement informative, in the general case, is having chosen the error rate in advance$-$or, equivalently, the pre-existence of a default error rate of 5%. For purposes of the null hypothesis statistical test (NHST), one does not reject at $p = .049$ and "super-reject" at $p = .01$.

Put another way, unless one is using an exact test, the sample $p$-value is itself a random variable, an estimate of a parameter with a variance. Having set the error rate at 5% in advance, we are guaranteed to reject 5% of the time under a true null (in the long run and when all other assumptions are met). But, there is a distribution around the observed $p$-value, so sometimes that rejection will occur by observing $p = .049$ and sometimes it will occur by observing $p = .01$.

Hence, observing $p = .01$ does not imply one would only observe this random sample 1% of the time under a true null. Rather, it means we would observe it 1% of the time +/- some (typically large) margin of error. To avoid this sort of less useful/more confusing interpretation, simply reject the null with the nominal error rate, whether the observed error rate is just below the nominal or really far below it.

That said, there are other uses for the $p$-value beyond NHST's, which may use the exact observed value. For example, the $p$-value of a $t$-test is a fit statistic for the model where population $t$ has a non-central $t$-distribution. As with $R^2$, one may compare different $p$-values for different models applied to the same data, e.g., for $t$-tests with different experimental hypotheses of different non-centrality parameters. This makes $p$ equivalent to a test of the effect size, as another answer to your question implies. (However, this is rarely the interpretation intended for tables of multi-starred estimates.)

Suppose you collect data and compute a $p$-value without first selecting a Type I error rate. Then, observing the $p$-value, you select a rate specifically so that it will be statistically significant. If you then proclaim, "My result is statistically significant!" that statement alone contains no inferentially-useful information. What makes that statement informative, in the general case, is having chosen the error rate in advance$-$or, equivalently, the pre-existence of a default error rate of 5%. For purposes of the null hypothesis statistical test (NHST), one does not reject at $p = .049$ and "super-reject" at $p = .01$.

Put another way, unless one is using an exact test, the sample $p$-value is itself a random variable, an estimate of a parameter with a variance. Having set the error rate at 5% in advance, we are guaranteed to reject 5% of the time under a true null (in the long run and when all other assumptions are met). But, there is a distribution around the observed $p$-value, so sometimes that rejection will occur by observing $p = .049$ and sometimes it will occur by observing $p = .01$.

Hence, observing $p = .01$ does not imply one would only observe this random sample 1% of the time under a true null. Rather, it means we would observe it 1% of the time +/- some (typically large) margin of error. To avoid this sort of less useful/more confusing interpretation, simply reject the null with the nominal error rate, whether the observed error rate is just below the nominal or really far below it.

That said, there are other uses for the $p$-value beyond NHST's, which may use the exact observed value. For example, the $p$-value of a $t$-test is a fit statistic for the model where population $t$ has a non-central $t$-distribution. As with $R^2$, one may compare different $p$-values for different models applied to the same data, e.g., for $t$-tests with different experimental hypotheses of different non-centrality parameters. This makes $p$ equivalent to a test of the effect size, as another answer to your question implies. (However, this is rarely the interpretation intended for tables of multi-starred estimates.)

Suppose you collect data and compute a $p$-value without first selecting a Type I error rate. Then, observing the $p$-value, you select a rate specifically so that it will be statistically significant. If you then proclaim, "My result is statistically significant!" that statement alone contains no inferentially-useful information. What makes that statement informative, in the general case, is having chosen the error rate in advance$-$or, equivalently, the pre-existence of a default error rate of 5%. For purposes of the null hypothesis statistical test (NHST), one does not reject at $p = .049$ and "super-reject" at $p = .01$.

That said, there are other uses for the $p$-value beyond NHST's, which may use the exact observed value. For example, the $p$-value of a $t$-test is a fit statistic for the model where population $t$ has a non-central $t$-distribution. As with $R^2$, one may compare different $p$-values for different models applied to the same data, e.g., for $t$-tests with different experimental hypotheses of different non-centrality parameters. This makes $p$ equivalent to a test of the effect size, as another answer to your question implies. (However, this is rarely the interpretation intended for tables of multi-starred estimates.)

Source Link
virtuolie
virtuolie
  • 642
  • 4
  • 11

Suppose you collect data and compute a $p$-value without first selecting a Type I error rate. Then, observing the $p$-value, you select a rate specifically so that it will be statistically significant. If you then proclaim, "My result is statistically significant!" that statement alone contains no inferentially-useful information. What makes that statement informative, in the general case, is having chosen the error rate in advance$-$or, equivalently, the pre-existence of a default error rate of 5%. For purposes of the null hypothesis statistical test (NHST), one does not reject at $p = .049$ and "super-reject" at $p = .01$.

Put another way, unless one is using an exact test, the sample $p$-value is itself a random variable, an estimate of a parameter with a variance. Having set the error rate at 5% in advance, we are guaranteed to reject 5% of the time under a true null (in the long run and when all other assumptions are met). But, there is a distribution around the observed $p$-value, so sometimes that rejection will occur by observing $p = .049$ and sometimes it will occur by observing $p = .01$.

Hence, observing $p = .01$ does not imply one would only observe this random sample 1% of the time under a true null. Rather, it means we would observe it 1% of the time +/- some (typically large) margin of error. To avoid this sort of less useful/more confusing interpretation, simply reject the null with the nominal error rate, whether the observed error rate is just below the nominal or really far below it.

That said, there are other uses for the $p$-value beyond NHST's, which may use the exact observed value. For example, the $p$-value of a $t$-test is a fit statistic for the model where population $t$ has a non-central $t$-distribution. As with $R^2$, one may compare different $p$-values for different models applied to the same data, e.g., for $t$-tests with different experimental hypotheses of different non-centrality parameters. This makes $p$ equivalent to a test of the effect size, as another answer to your question implies. (However, this is rarely the interpretation intended for tables of multi-starred estimates.)