10
$\begingroup$

I am using the Kolmogorov–Smirnov two-sample test to compare distributions, and I noticed a $p$-value is frequently reported as the test statistic. How is this $p$-value determined? I know it's the probability of obtaining a result at least as large as the one obtained, but how is this $p$-value determined given this is a nonparametric test? That is, we can't assume Gaussian fluctuations in the distribution and compute the $p$-value using a $t$-test.

Thanks!

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ The Kolmogorov-Smirnov statistic (over the class of distributions of continuous random variables) is distribution-free. So, the distribution of the test statistic does not depend on the underlying distribution of the data (under the null hypothesis). $\endgroup$
    – cardinal
    Oct 24, 2011 at 19:31
  • 2
    $\begingroup$ @Cardinal's point is made in a comment in the Wikipedia entry. Note that the distribution of the test statistic is asymptotic (that is, valid when the smaller sample size is itself large); it likely does depend on the common underlying distribution for small samples. $\endgroup$
    – whuber ♦
    Oct 24, 2011 at 21:15
  • $\begingroup$ @whuber: I fear I don't quite understand your comment and I don't want to misinterpret it. Certainly, the distribution in finite samples will not be exactly the same as the asymptotic distribution, but that does not prevent the statistic from being distribution-free for every fixed sample size $n$ (really $(n_1,n_2)$ since the sizes may differ). [cont] $\endgroup$
    – cardinal
    Oct 25, 2011 at 0:27
  • 6
    $\begingroup$ @whuber: ...Let $X_i \sim F$ and $Y_i \sim G$ be independent iid sequences. Then $n \hat{F}_n(x) = |\{i: X_i \leq x\}| = |\{i: F(X_i) \leq F(x)\}|$ and $n \hat{G}_n(x) = |\{i: Y_i \leq x \}| = |\{i: G(Y_i) \leq G(x)\}|$. So, with the aforementioned assumption that $F$ and $G$ are continuous distributions, under the null hypothesis $F=G$, we see that $\sup |\hat{F}_n(x) - \hat{G}_n(x)|$ is equal in distribution to the same statistic obtained from two independent $\mathcal U(0,1)$ samples of the same size. $\endgroup$
    – cardinal
    Oct 25, 2011 at 0:33
  • 1
    $\begingroup$ @whuber: I think these are two separate, but subtly different, effects. In some sense, we like asymptotics precisely because they (often) give us a distribution-free statistic "in the limit" (by virtue of the CLT). So, the fact that the reported $p$-value is independent of the distribution assumption is not all that remarkable. One might then ask, what is the point of a distribution-free statistic if I cannot (easily) compute its distribution for a given sample size and, instead, must rely on an asymptotic approximation? What one seems to gain is a version of uniform convergence. $\endgroup$
    – cardinal
    Oct 25, 2011 at 1:19

2 Answers 2

Reset to default
12
$\begingroup$

Under the null hypothesis, the asymptotic distribution of the two-sample Kolmogorov–Smirnov statistic is the Kolmogorov distribution, which has CDF

$$\operatorname{Pr}(K\leq x)=\frac{\sqrt{2\pi}}{x}\sum_{i=1}^\infty e^{-(2i-1)^2\pi^2/(8x^2)} \>.$$

The $p$-values can be calculated from this CDF - see Section 4 and Section 2 of the Wikipedia page on the Kolmogorov–Smirnov test.

You seem to be saying that a non-parametric test statistic shouldn't have a distribution - that's not the case - what makes this test non-parametric is that the distribution of the test statistic does not depend on what continuous probability distribution the original data come from. Note that the KS test has this property even for finite samples as shown by @cardinal in the comments.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ (+1) I might suggest a small tweak to your last sentence. The test statistic is distribution-free even in finite samples (though it won't be the same as the asymptotic distribution). So, this distribution-free property is what makes the test statistic nonparametric. Note that there are lots of examples where the asymptotic distribution does not depend on the underlying continuous distribution (just think about the CLT), so, unless I'm mistaken, I don't believe that's the core feature here. :) $\endgroup$
    – cardinal
    Feb 21, 2012 at 13:30
  • $\begingroup$ I made the correction but the more I think about it the more I wonder how you know that the statistic truly doesn't depend on the original distribution of the data in finite samples - can you say any more about this @cardinal? $\endgroup$
    – Macro
    Feb 21, 2012 at 14:04
  • $\begingroup$ Sure. See the fourth comment (my third one) to the question above. $\endgroup$
    – cardinal
    Feb 21, 2012 at 14:21
  • $\begingroup$ I see! very cool and simple - thanks cardinal $\endgroup$
    – Macro
    Feb 21, 2012 at 15:26
  • $\begingroup$ No one has addressed the distribution in small samples, where we can directly calculate the permutation distribution of the statistic. If we have $m$ $X$ labels and $n$ $Y$ labels we can write down all the possible orders of them (corresponding to the values all arranged from smallest to largest) and it's possible to compute the two-sample K-S statistic directly from that. In practice the algorithm to find a p-value can be made more sophisticated than just to write out all possibilities (either way the amount of calculation grows quickly but the asymptotic distribution comes in fairly quickly) $\endgroup$
    – Glen_b
    Sep 20, 2018 at 0:02
0
$\begingroup$

The p-value of,say 0.80, implies that 80% of samples of size n of samples from the population, will have a D statistic less than the one obtained from the test. This is calculated based on the D-statistic of KS test, which measures the maximum distance between the CDFs of theoretical and empirical distribution, for the given distribution against which the sample is evaluated.

Note that only the value D*SQRT(sample size) has a kolmogrov distribution and not D itself. If you want to manually calculate p value given D value, you can refer the published tables available in the internet for kolomogrov distribution. This is also the value given in packages like R

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is not a clearly explained answer. $\endgroup$
    – Michael R. Chernick
    Feb 10, 2019 at 1:58
  • $\begingroup$ It is a continuation of previous answer posted by Macro above. Unlike what has been believed by many, the p-value calculated by R package is perfect. It means if you take every possible sample of given size from the population and compare it with theoretical distribution, the value of [maximum distance D *SQRT(sample size)] calculated against each sample, will have a kolomogrov distrbution. For a given D-statistic, the R package gives the value of probability that the sample of given difference belongs to the theoretical population, 0.8 means only 20% will have higher D $\endgroup$
    – Murugesan Narayanaswamy
    Feb 11, 2019 at 4:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.