I'm trying to understand the output of the Kolmogorov-Smirnov test function (two samples, two sided). Here is a simple test.

x <- c(1,2,2,3,3,3,3,4,5,6)
y <- c(2,3,4,5,5,6,6,6,6,7)
z <- c(12,13,14,15,15,16,16,16,16,17)


#   Two-sample Kolmogorov-Smirnov test
#data:  x and y
#D = 0.5, p-value = 0.1641
#alternative hypothesis: two-sided
#Warning message:
#In ks.test(x, y) : cannot compute exact p-value with ties


#Two-sample Kolmogorov-Smirnov test

#data:  x and z
#D = 1, p-value = 9.08e-05
#alternative hypothesis: two-sided
#Warning message:
#In ks.test(x, z) : cannot compute exact p-value with ties


#Two-sample Kolmogorov-Smirnov test

#data:  x and x
#D = 0, p-value = 1
#alternative hypothesis: two-sided
#Warning message:
#In ks.test(x, x) : cannot compute exact p-value with ties

There are a few things I don't understand here.

  1. From the help, it seems that the p-value refers to the hypothesis var1=var2. However, here that would mean that the test says (p<0.05):

    a. Cannot say that X = Y;

    b. Can say that X = Z;

    c. Cannot say that X = X (!)

Besides appearing that x is different from itself (!), it is also quite strange to me that x=z, as the two distributions have zero overlapping support. How is that possible?

  1. According to the definition of the test, D should be the maximum difference between the two probability distributions, but for instance in the case (x,y) it should be D = Max|P(x)-P(y)| = 4 (in the case when P(x), P(y) aren't normalized) or D=0.3 (if they are normalized). Why D is different from that?

  2. I have intentionally made an example with many ties, as the data I'm working with have lots of identical values. Why does this confuse the test? I thought it calculated a probability distribution that should not be affected by repeated values. Any idea?

  • 1
    $\begingroup$ I think your conclusions are reversed, based on the p-values, you can say X=X and cannot say X=Z $\endgroup$ – user3661376 May 17 at 15:01

The KS test is premised on testing the "sameness" of two independent samples from a continuous distribution (as the help page states). If that is the case then the probability of ties should be astonishingly small (also stated). The test statistic is the maximum distance between the ECDF's of the two samples. The p-value is the probability of seeing a test statistic as high or higher than the one observed if the two samples were drawn from the same distribution. (It is not the "probability that var1 = var2". And furthermore, 1-p_value is NOT the that probability either.) High p-values say you cannot claim statistical support for a difference, but low p-values are not evidence of sameness. Low p-values can occur with low sample sizes (as your example provides) or the presence of interesting but small differences, e.g. superimposed oscillatory disturbances. If you are working with situations with large numbers of ties it suggests you may need to use a test that more closely fits your data situation.

My explanation of why ties were a violation of assumptions was not a claim that ties invalidated the results. The statistical properties of the KS test in practice are relatively resistant or robust to failure of that assumption. The main problem with the KS test as I see is that it is excessively general and as a consequence is under-powered to identify meaningful differences of an interesting nature. The KS test is a very general test and has rather low power for more specific hypotheses.

On the other hand, I also see the KS-test (or the "even more powerful" Anderson Darling or Lillefors(sp?) test) used to test "normality" in situations where such a test is completely unwarranted, such as test for the normality of variables being used as predictors in a regression model before the fit. One might legitimately want to be testing the normality of the residuals since that is what is assumed in the modeling theory. Even then modest departures from normality of the residuals do not generally challenge the validity of the results. Persons would be better of using robust methods to check for important impact of "non-normality" on conclusions about statistical significance.

Perhaps you should consult with a local statistician? It might assist you in defining the statistical question a bit more precisely and therefore have a better chance of identifying a difference if one actually exists. That would be avoidance of a "type II error": failing to support a conclusion of difference when such a difference is present.

| cite | improve this answer | |
  • 1
    $\begingroup$ Very nice. This may be enlightening: Kolmogorov-Smirnov with discrete data: What is proper use of dgof::ks.test in R? $\endgroup$ – Stephan Kolassa Jul 5 '16 at 19:17
  • $\begingroup$ I have tested the same examples both with dgof::ks.test(x,y,simulate.p.value=TRUE, B=1000) and Matching::ks.boot(x,y, nboots=1000) (sekhon.berkeley.edu/matching/ks.boot.html). Both D and the calculated p-value are absolutely identical in both cases. This makes me think that perhaps KS is not so bad, even when one has several ties and the method is not guaranteed to work? The reason why I like KS is that is not parametric, i.e. I don't need to assume a distribution for the samples. $\endgroup$ – Nonancourt Jul 6 '16 at 11:46
  • $\begingroup$ However, I still cannot make sense of the values of D. I thought it might be a prefactor as sqrt(m*n/(m+n)) as here, but that would make D(x,y) = sqrt(100/20)*0.3=0.67, which is still different. $\endgroup$ – Nonancourt Jul 6 '16 at 11:53

To compute the D (from ks.test code):


    Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.5, p-value = 0.1641
alternative hypothesis: two-sided

alternative <- "two.sided"
x <- x[!is.na(x)]
n <- length(x)
  y <- y[!is.na(y)]
  n.x <- as.double(n)
  n.y <- length(y)
  w <- c(x, y)
  z <- cumsum(ifelse(order(w) <= n.x, 1/n.x, -1/n.y))
  z <- z[c(which(diff(sort(w)) != 0), n.x + n.y)] #exclude ties
  STATISTIC <- switch(alternative, two.sided = max(abs(z)), 
                      greater = max(z), less = -min(z))

[1] 0.5
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.