6
$\begingroup$

It is well known that the distribution of the p-value in a statistical test is $U(0,1)$ under the null. Now, consider the following implementation of the two-samples Kolmogorov Smirnov test on normal samples:

K = 1e4
n = 100
p.val = sapply(1:K,FUN = function(x) ks.test(rnorm(n),rnorm(n),exact=TRUE)$p.value)
mean(p.val<0.05)
plot(p.val,main = 'Plot of p.val')
hist(p.val,freq=FALSE)

p.val is far from being uniformly distributed and takes only discrete values. In addition, the proportion of p.val < 0.05 does not seem to be 0.05, but slightly less, around 0.037 during my testings. enter image description here

What is going on here? In the 1 sample case, we get the results as expected though, this problems is specific to the two-samples case.

EDIT: The problem lies in the fact that the p-value is not uniformly distributed under null in general. Thanks @whuber for pointing it out.

Let $\boldsymbol{X}^n = (X_1, \dots, X_n)$ be a vector of observation. In general a statistical test at level $\alpha \in (0,1)$ has the form:

\begin{equation} \phi_{\alpha} = 1_{T(\boldsymbol{X}^n) \in \mathcal{R}_{\alpha}} \end{equation} Where $T$ is a real functional of the observations and $\mathcal{R}_{\alpha}$ is the critical region, i.e. $\mathcal{R}_{\alpha} \subset \mathbb{R}$ such that $\mathbb{P}(T(\boldsymbol{X}^n) \in \mathcal{R}_{\alpha}) \leq \alpha$ under the null hypothesis. Now let $P$ be the p-value of the test, i.e. by definition: \begin{equation} P = \inf \{x \in (0,1) | \phi_{x} = 1\}. \end{equation} Then for any $\alpha \in (0,1)$ we have: \begin{align} \mathbb{P}(P \leq \alpha) &= \mathbb{P}(\inf \{x \in (0,1) | \phi_{x} = 1\} \leq \alpha)\\ & = \mathbb{P}(\phi_{\alpha} = 1). \end{align} The last equation is because the two events $\{ \inf \{x \in (0,1) | \phi_{x} = 1\} \leq \alpha \}$ and $\{ \phi_{\alpha} = 1 \}$ are equal. But under the null, the only guarantee is that $\mathbb{P}(\phi_{\alpha} = 1) \leq \alpha$, i.e. that $\mathbb{P}(P \leq \alpha) \leq \alpha$, hence $P$ is not uniform in general.

In order to get uniformly distributed p-values, the familly $\{\phi_{\alpha}|0\leq \alpha \leq 1\}$ must be exactly of level $\alpha$ (in the sense the the type I error must be exactly $\alpha$) the distribution of the test statistic must be absolutely continuous and the null must be simple. Because the 2-samples KS test has discret test statistic, the p-value cannot be uniformly distributed.

$\endgroup$
  • $\begingroup$ Interesting. I thought this might be due to Monte Carlo noise, but upping K to 1,000,000 returned the same shaped distribution as the histogram on the right. $\endgroup$ – Mark White Oct 15 '19 at 13:19
  • 5
    $\begingroup$ What you believe is "well known" just isn't true. The distribution of the p-value is not necessarily uniform. The two most salient exceptions are (1) discrete sampling distributions and (2) composite null hypotheses. The KS statistic is discrete--its values must be multiples of $1/n$--and therefore, for smallish datasets, is always visibly non-uniform. $\endgroup$ – whuber Oct 15 '19 at 13:54
  • $\begingroup$ Take a look at plot(ecdf(p.val));abline(0,1,col=2) to see the connection between the distribution of p-values in the discrete case and a uniform distribution. $\endgroup$ – Glen_b Oct 16 '19 at 0:19
  • 1
    $\begingroup$ Since the issue is discrete distributions of test statistics rather than anything specific to the K-S in particular, I think this is a duplicate of: stats.stackexchange.com/questions/153249/… $\endgroup$ – Glen_b Oct 16 '19 at 0:20
1
$\begingroup$

It is not surprising that when dealing with a two sample test with a fairly small sample size (100), the test statistic (and thus the p-value) takes on discrete values. Increasing n (and decreasing K to reduce computational time) leads to a less discontinuous distribution.

As whuber pointed out in the comments, the fact that the p-value is obligately discrete leads to unusually high histogram bars. The apparent skew in the distribution is an artefact of the bins used to construct it and the discrete values possible for the test statistic in the low n setting.

K = 1000
n = 1000
p.val = sapply(1:K,FUN = function(x) ks.test(rnorm(n),rnorm(n),exact=TRUE)$p.value)
mean(p.val<0.05)
#> [1] 0.055
plot(p.val,main = 'Plot of p.val')

hist(p.val,freq=FALSE, breaks = "FD")

Created on 2019-10-15 by the reprex package (v0.3.0)

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ The histogram in the question confusingly looks like it depicts a skewed distribution. It does not. The problem lies in its construction: namely, the appearance of many zero-height bars. One's eye takes in the outline--the upper envelope--of the other bars, which (greatly) distorts the perception of the actual probabilities and the shape of the distribution. $\endgroup$ – whuber Oct 15 '19 at 14:19
  • 1
    $\begingroup$ That's a great observation! You are absolutely correct; if the height of the histogram bars were redistributed the distribution would be uniform. $\endgroup$ – alan ocallaghan Oct 15 '19 at 14:21
  • 1
    $\begingroup$ As the left hand plot in the question shows, it's not truly uniform. That's because the p-values necessarily are (nonlinear) transformations of the KS statistic and therefore are not uniformly spaced. Making up for that are the relative frequencies (impossible to estimate in that graphic): p-values that are further from their neighbors occur slightly more often. Thus, in a kind of local average density sense, the p-value distribution is uniform. That might be more clearly seen by constructing a kernel density estimate from the p-values. $\endgroup$ – whuber Oct 15 '19 at 14:42
  • 1
    $\begingroup$ You're right, that was a poor choice of words on my behalf. The non-uniformity apparent is just due to the smoothing parameters (bin location and width) chosen. If one were to construct a histogram-type plot using bins centered at the possible p-values, with widths adjusted based on the interval between values, it would appear uniform. The same is true for my plot, but with a higher number of possible values it's not visible with Freedman-Diaconis binning. $\endgroup$ – alan ocallaghan Oct 15 '19 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.