5
$\begingroup$

For the purpose of this post, accept that there is some critical threshold below which a finding is declared to be statistically significant.

The p-value is a statistic because it is a function of the data.

Statistics are random variables. We can put confidence intervals around statistics.

Why don't we put confidence intervals around p-values? Or, equivalently, why don't we test the hypothesis that the observed p-value is below 0.05, say?

One answer could be that it would be hard to calculate analytically, but bootstrapping could solve that issue.

Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ p-value is a characteristic of a given sample, not an estimate of something in the population to be able to bear some error. So p-value is an exact number, not an interval. p-value is interval only in Monte Carlo testing - because in such a testing not full information from the sample is used to compute p. $\endgroup$
    – ttnphns
    Commented Mar 8, 2013 at 5:56
  • 2
    $\begingroup$ The p-value depends on the null hypothesis in addition to the data, whereas the data do not depend on the null hypothesis. $\endgroup$
    – Alexis
    Commented Jun 19, 2015 at 17:15
  • 1
    $\begingroup$ Very relevant thread: stats.stackexchange.com/questions/254595 - see discussions in the comments. $\endgroup$
    – amoeba
    Commented Aug 18, 2017 at 16:30

1 Answer 1

Reset to default
2
$\begingroup$

Among other things, you might want to read: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is a bare link and does not describe the bare connection to the question. Though it is good reading. $\endgroup$
    – Erik
    Commented Mar 8, 2013 at 8:34
  • $\begingroup$ I generally agree that one-sentence answers are bad, but that's all I have. I can't explain it myself, but know of a paper by a famous statistician who seems to address the point. $\endgroup$
    – Wayne
    Commented Mar 8, 2013 at 12:47
  • $\begingroup$ @Wayne, I'm aware of the paper, thanks. But Gelman and Stern are arguing that estimates of a mean, say, between two samples may not be statistically different even though one of the estimates is significant while the other is not. I'm looking at a different issue: Why no confidence intervals around a p-value for a particular sample? $\endgroup$
    – Charlie
    Commented Mar 8, 2013 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.