For the purpose of this post, accept that there is some critical threshold below which a finding is declared to be statistically significant.

The p-value is a statistic because it is a function of the data.

Statistics are random variables. We can put confidence intervals around statistics.

Why don't we put confidence intervals around p-values? Or, equivalently, why don't we test the hypothesis that the observed p-value is below 0.05, say?

One answer could be that it would be hard to calculate analytically, but bootstrapping could solve that issue.

  • 5
    $\begingroup$ p-value is a characteristic of a given sample, not an estimate of something in the population to be able to bear some error. So p-value is an exact number, not an interval. p-value is interval only in Monte Carlo testing - because in such a testing not full information from the sample is used to compute p. $\endgroup$
    – ttnphns
    Mar 8 '13 at 5:56
  • 2
    $\begingroup$ The p-value depends on the null hypothesis in addition to the data, whereas the data do not depend on the null hypothesis. $\endgroup$
    – Alexis
    Jun 19 '15 at 17:15
  • 1
    $\begingroup$ Very relevant thread: stats.stackexchange.com/questions/254595 - see discussions in the comments. $\endgroup$
    – amoeba
    Aug 18 '17 at 16:30

Among other things, you might want to read: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

  • $\begingroup$ This is a bare link and does not describe the bare connection to the question. Though it is good reading. $\endgroup$
    – Erik
    Mar 8 '13 at 8:34
  • $\begingroup$ I generally agree that one-sentence answers are bad, but that's all I have. I can't explain it myself, but know of a paper by a famous statistician who seems to address the point. $\endgroup$
    – Wayne
    Mar 8 '13 at 12:47
  • $\begingroup$ @Wayne, I'm aware of the paper, thanks. But Gelman and Stern are arguing that estimates of a mean, say, between two samples may not be statistically different even though one of the estimates is significant while the other is not. I'm looking at a different issue: Why no confidence intervals around a p-value for a particular sample? $\endgroup$
    – Charlie
    Mar 8 '13 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.