8
$\begingroup$

The $p$-value is used to report how strongly we can presume against an hypothesis. As is clear, this $p$ value is itself estimated from data and if new data where collected in the same conditions, the new $p$ value will very unlikely be the same.

Halsey, Curran-Everett, Vowler & Drummond (2015) in a commentary to Nature Methods showed that the uncertainty surrounding a $p$-value can be fairly large. In a reply, Lazzeroni, Lu & Belitskaya-Lévy (2016, same journal) gave an example of an observed $p$ value of 0.049 whose confidence interval goes from 0.00000008 to 0.99.

My question is: do we know the sampling distribution of $p$ values? According to the latter, it does not depend on sample size (and presumably on the sample's standard deviation as all these are used to "standardize" the test statistic). Presumably, it might depend on the test procedure?

I know that if $H_0$ is true, the distribution of $p$-values is uniform over the range 0 to 1 (but can't remember where I learned this). As $H_0$ is more and more inadequate, the distribution of $p$-values becomes peaked, leaning over the 0% probabilities (for left-tail tests).

It is fairly easy with bootstrap to get a visual representation of the distribution of the $p$-values. However, a more satisfying answer would be to have a formula (closed-form is even better) so that we can know exactly what characteristics affect that distribution, and henceforth, the width of the confidence interval.

Do you know of such a formula, or if it is even possible to have one?

Improve this question
$\endgroup$
19
  • 2
    $\begingroup$ I believe what you want would be a prediction interval for future p-values constructed under the same conditions as the original p-value? Perhaps you do mean confidence interval rather than prediction interval, but talking about a confidence interval for an observed value is very confusing to me. Whether you meant prediction or confidence interval, I'm pretty sure you want to specify that the interval refers to the mean of future p-values from future studies. $\endgroup$
    – Cliff AB
    Commented Jan 4, 2017 at 20:17
  • 2
    $\begingroup$ @Cliff If you accept that there is a sampling distribution of p-values (which seems uncontroversial), then the fact that p-values are bounded implies this sampling distribution has an expectation. Its expectation evidently is a property of the underlying distribution within the context of a specific model and specific test statistic. Given that, it looks like this expectation could reasonably viewed as a property of the distribution itself, permitting one to apply all the conventional concepts of estimate, estimator, and confidence interval. $\endgroup$
    – whuber ♦
    Commented Jan 4, 2017 at 20:27
  • 3
    $\begingroup$ Halsey et al paper that OP mentioned and the reasoning behind it is discussed at great length in this recent thread: stats.stackexchange.com/questions/250269 - which I would say is perhaps even a duplicate (@whuber). The general conclusion of that thread is Halsey et al (who borrow their claims from the earlier work by Cumming) are sloppy and do not state their assumptions. I strongly dislike their paper. $\endgroup$
    – amoeba
    Commented Jan 4, 2017 at 20:31
  • 2
    $\begingroup$ @whuber Yes, I agree. Still it might be useful for the OP to read those discussions. $\endgroup$
    – amoeba
    Commented Jan 4, 2017 at 20:44
  • 3
    $\begingroup$ @Amoeba Be careful: one does not construct a CI for a statistic; a CI refers to a parameter. In classical situations (Z tests, t tests, etc) there is a one-to-one correspondence between the statistic and the p-value. To the extent a statistic can estimate something (typically an effect size), a fortiori a p-value must be estimating something, too. But what it might estimating has nothing to do with how one constructs a CI. A plausible candidate for its estimand is the expected p-value (for a given model, given statistic, and given effect size). The chief difficulty, it seems to me, $\endgroup$
    – whuber ♦
    Commented Jan 4, 2017 at 20:58

1 Answer 1

Reset to default
3
$\begingroup$

The problem is that a p value is not an estimate of a parameter so the idea of a confidence interval does not apply. It also does not make sense to talk about the uncertainty surrounding a p value. The p value is certain; the conclusion you draw from it is not.

Improve this answer
$\endgroup$
10
  • 6
    $\begingroup$ You appear to be denying the premises of the question, including the point of view that the p-value is uncertain. That's going to be controversial, because it's well known--and intuitively obvious--that when an experiment is repeated a different p-value is almost sure to arise. You might find the thread at stats.stackexchange.com/questions/181611 somewhat relevant. $\endgroup$
    – whuber ♦
    Commented Jan 4, 2017 at 20:09
  • $\begingroup$ Hello @David, glad to see you on StackExchange. Although I agree with you that in general, p is not a parameter, I am sure we could image a world in which populations are characterized by a parameter $\pi$. In this world, all the samples would have a constant size and all the sampling method will be constant as well. In this improbable world (if you allow the pun), $\pi$ is a parameter, and $p \equiv \hat\pi$, probably the best, unbiased estimate of $\pi$. Hence, if I frame my question relative to this world, can we have a confidence interval around an observed $p$? $\endgroup$
    – Denis Cousineau
    Commented Jan 4, 2017 at 20:12
  • $\begingroup$ Hi @Denis. That makes a lot of sense. However, I think the critique of significance testing (others make) that since p values differ across replications they are not informative is incorrect. Of course different replications will provide different degrees of conclusiveness about the direction of an effect (I assume the effect is almost never 0). That doesn't bear on the conclusiveness of a given study. $\endgroup$
    – David Lane
    Commented Jan 4, 2017 at 20:27
  • 2
    $\begingroup$ Sure, as @David says, $p$ are informative. They are just variable. If we could get some confidence interval, and find that the whole interval is very narrow and close to zero, that would add additional strength to a conclusion. $\endgroup$
    – Denis Cousineau
    Commented Jan 4, 2017 at 20:32
  • 1
    $\begingroup$ @whuber Of course the p value is uncertain before you do the experiment. However, the important uncertainty is the direction of the effect, not the p value. The p value is a tool to guide inference, not the object of the inference. That's why it doesn't make sense to say the data provide evidence for a significant effect. $\endgroup$
    – David Lane
    Commented Jan 4, 2017 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.