As we know, $p$-values are uniformly distributed under $H_0$. This urges me to ask if this constitutes a valid (re-)definition of the $p$-value.

$p$-value - A statistic with a uniform distribution on $[0, 1]$ under $H_0$.

Intuitively it makes sense for continuous distributions, but for discrete distributions, some adjustments are to be made. For example, this patched version might work,

$p$-value - A statistic $X$ such that $\forall c \in \left\{P(X = c|H_0) > 0\right\}$, $P(X \le c|H_0) = c$.

I'm asking this question because I have some trouble understanding the "power" of a hypothesis test. If this argument holds true, then I can continue to define the power as a function of its corresponding $p$-value statistic, which would be cool.

  • 1
    $\begingroup$ For discrete distributions (e.g., a $\chi^2$ test or a binomial test), only certain p-values are possible (see: Comparing and contrasting, p-values, significance levels and type I error). $\endgroup$ – gung - Reinstate Monica Mar 31 '19 at 1:29
  • $\begingroup$ An uniform r.v. independent to your test will always satisfies your definition. $\endgroup$ – Francis Mar 31 '19 at 1:41
  • $\begingroup$ @Francis Actually this fact is part of my motivation! I mean, technically it can be considered a $p$-value; it just have extremely low power. I even think it would be a good idea to make it a reference point with “zero power”, and define the power of other $p$-values with respective to it. $\endgroup$ – nalzok Mar 31 '19 at 1:48
  • 1
    $\begingroup$ P-value under the null hypothesis is UNIF(0,1) only for test statistics that are continuous and exact. $\endgroup$ – BruceET Mar 31 '19 at 6:17
  • $\begingroup$ @BruceET Thanks for pointing that out, but I wonder if it’s fair to say the $p$-value of discrete/approximate statistics have an “approximately” UNIF(0, 1) distribution? Like, for discrete distributions, I might define a similar construct with a generalized inverse. $\endgroup$ – nalzok Mar 31 '19 at 6:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.