How is the p-value calculated in Kolmogorov-Smirnov test? What's the formula for it? I suspect it's derived based on the definition of p-value. The value of test statistic is the largest differnece between the two CDFs, is that correct? So essentially, we are asking what's the probability of getting such a big or bigger difference in empirical CDFs assuming the null hypothesis is true (both data sets come from the same distribution). And by answering this question we obtain the formula for p-value in KS-test (I haven't seen the formula yet, though). So what is the formula for p-value and how is it derived?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
8
This is a good question because while intuitively relatively straight-forward (ie. what is the chance that the value of the Kolmogorov-Smirnov D statistic would be as large or larger than observed?) the calculation of the said probability is not.
First let me point that one should differentiate between two-sided and one-sided tests. I will focus on the two-sided test as it is more commonly used.
OK, let's say that the distance between two CDFs is $D_{m,n}$ where $m$ and $n$ are the sample sizes concerned. The $p$-value we want is $P(D_{m,n} \geq D_{Obs} | H_0)$. $D_{Obs}$ is what we observe and $H_0$ is that the two population distributions are identical, ie. we have samples from the same population. Clearly the crux of the derivation of the exact null probability distribution of $D_{m,n}$. Here is where Smirnov comes in the story; he proved that [1]:
$lim_{n,m \rightarrow \infty} P( D_{Obs} \geq \sqrt{\frac{mn}{m+n}} D_{m,n} ) = L(D_{Obs})$ where: $L(D_{Obs}) = 1 - 2 \Sigma_i^\infty(-1)^{i-1}e^{2i^2 D_{obs}^2}$.
Most of the time people actually do not work with $D_{Obs}$ directly but use certain rescaling parameters that reflecting the influence of $n$ and $m$. It also goes without saying that you are not to evaluate the above infinite series directly. Certain implementations sum the first 100 points but even then you need to be mindful about numerical precision issues. There is really nice freely accessible paper from the Journal of Statistical Software on Evaluating Kolmogorov’s Distribution. In the past one relied on empirically derived tables more (eg. [2]).
I based my answer on the English language papers mentioned and in the book Nonparametric Statistical Inference by Gibbons and Chakraborti (Chapt. 6).
[1]: Smirnov, N. V. (1939), Estimate of deviation between empirical distribution functions in two independent samples (in Russian), Bulletin of Moscow University, 2, 3–16.
[2]: Frank J. Massey, Jr. (1951), The Kolmogorov-Smirnov Test for Goodness of Fit, Journal of the American Statistical Association, Vol. 46, No. 253, pp. 68- 78
answered May 4, 2015 at 8:16
-
$\begingroup$ You say that $P(D_{m,n} \geq D_{Obs} | H_0)$ is the p-value, fine. Just making sure - you didn't say what it's equal to in your answer? I'm a beginner in statistics and can't quite see the relation between the $P(D_{m,n} \geq D_{Obs} | H_0)$ formula and the other two that have been proven by Smirnov you've mentioned. $\endgroup$ Commented May 4, 2015 at 12:04
-
$\begingroup$ Out of curiosity, do people ever use a confidence interval based on a permutation test? Seems like that would both be an easier calculation and eliminate any issues with convergence to the limit... $\endgroup$– Ben KuhnCommented May 4, 2015 at 15:46
-
$\begingroup$ @user4205580: The p-value is equal to the chance that the value of the Kolmogorov-Smirnov
Dstatistic would be as large or larger than observed. $P(D_{m,n} \geq D_{Obs}) = 1 - P(D_{m,n} < D_{Obs})$. Yes, there are some parameters in place ($m$,$n$) but those do not affect the proportionality.) $\endgroup$ Commented May 5, 2015 at 2:44 -
$\begingroup$ @BenKuhn: I think people do that all the time if one thinks of Monte Carlo sampling as a permutation test. $\endgroup$ Commented May 5, 2015 at 2:47
-
$\begingroup$ @usεr11852 Actually, what is $L(D_{obs})$? $\endgroup$ Commented May 5, 2015 at 7:33