6
$\begingroup$

How is the p-value calculated in Kolmogorov-Smirnov test? What's the formula for it? I suspect it's derived based on the definition of p-value. The value of test statistic is the largest differnece between the two CDFs, is that correct? So essentially, we are asking what's the probability of getting such a big or bigger difference in empirical CDFs assuming the null hypothesis is true (both data sets come from the same distribution). And by answering this question we obtain the formula for p-value in KS-test (I haven't seen the formula yet, though). So what is the formula for p-value and how is it derived?

$\endgroup$
0
8
$\begingroup$

This is a good question because while intuitively relatively straight-forward (ie. what is the chance that the value of the Kolmogorov-Smirnov D statistic would be as large or larger than observed?) the calculation of the said probability is not.

First let me point that one should differentiate between two-sided and one-sided tests. I will focus on the two-sided test as it is more commonly used.

OK, let's say that the distance between two CDFs is $D_{m,n}$ where $m$ and $n$ are the sample sizes concerned. The $p$-value we want is $P(D_{m,n} \geq D_{Obs} | H_0)$. $D_{Obs}$ is what we observe and $H_0$ is that the two population distributions are identical, ie. we have samples from the same population. Clearly the crux of the derivation of the exact null probability distribution of $D_{m,n}$. Here is where Smirnov comes in the story; he proved that [1]:

$lim_{n,m \rightarrow \infty} P( D_{Obs} \geq \sqrt{\frac{mn}{m+n}} D_{m,n} ) = L(D_{Obs})$ where: $L(D_{Obs}) = 1 - 2 \Sigma_i^\infty(-1)^{i-1}e^{2i^2 D_{obs}^2}$.

Most of the time people actually do not work with $D_{Obs}$ directly but use certain rescaling parameters that reflecting the influence of $n$ and $m$. It also goes without saying that you are not to evaluate the above infinite series directly. Certain implementations sum the first 100 points but even then you need to be mindful about numerical precision issues. There is really nice freely accessible paper from the Journal of Statistical Software on Evaluating Kolmogorov’s Distribution. In the past one relied on empirically derived tables more (eg. [2]).

I based my answer on the English language papers mentioned and in the book Nonparametric Statistical Inference by Gibbons and Chakraborti (Chapt. 6).

[1]: Smirnov, N. V. (1939), Estimate of deviation between empirical distribution functions in two independent samples (in Russian), Bulletin of Moscow University, 2, 3–16.

[2]: Frank J. Massey, Jr. (1951), The Kolmogorov-Smirnov Test for Goodness of Fit, Journal of the American Statistical Association, Vol. 46, No. 253, pp. 68- 78

$\endgroup$
8
  • $\begingroup$ You say that $P(D_{m,n} \geq D_{Obs} | H_0)$ is the p-value, fine. Just making sure - you didn't say what it's equal to in your answer? I'm a beginner in statistics and can't quite see the relation between the $P(D_{m,n} \geq D_{Obs} | H_0)$ formula and the other two that have been proven by Smirnov you've mentioned. $\endgroup$ May 4 '15 at 12:04
  • $\begingroup$ Out of curiosity, do people ever use a confidence interval based on a permutation test? Seems like that would both be an easier calculation and eliminate any issues with convergence to the limit... $\endgroup$
    – Ben Kuhn
    May 4 '15 at 15:46
  • $\begingroup$ @user4205580: The p-value is equal to the chance that the value of the Kolmogorov-Smirnov D statistic would be as large or larger than observed. $P(D_{m,n} \geq D_{Obs}) = 1 - P(D_{m,n} < D_{Obs})$. Yes, there are some parameters in place ($m$,$n$) but those do not affect the proportionality.) $\endgroup$
    – usεr11852
    May 5 '15 at 2:44
  • $\begingroup$ @BenKuhn: I think people do that all the time if one thinks of Monte Carlo sampling as a permutation test. $\endgroup$
    – usεr11852
    May 5 '15 at 2:47
  • $\begingroup$ @usεr11852 Actually, what is $L(D_{obs})$? $\endgroup$ May 5 '15 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.