# Why do I get strange results from the Kolmogorov–Smirnov two sample test?

I'm trying to understand the output of the Kolmogorov-Smirnov two sample test. I have two columns, test1 and test2. I'm having some difficulty in understanding the interpretation of the counterintuitive results I am getting.

structure(list(test1 = c(0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07,
0.06, 0.13, 0.14, 2, 0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26,
0.14, 0.19, 0.38, 7, 1, 1, 1, 1, 2, 1, 0.74, 0.53, 1, 1, 0.32,
0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 0.03, 0.04, 0.01, 0.01,
0.01, 0.01, 0.01, 0.004, 0.01, 0.95), test2 = c(1, 1, 1, 2, 1,
0.74, 0.53, 1, 1, 0.32, 0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02,
0.03, 0.04, 0.01, 0.01, 0.01, 0.01, 0.01, 0.004, 0.01, 0.95,
0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 0.06, 0.13, 0.14, 2,
0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 0.14, 0.19, 0.38, 7,
1)), row.names = c(NA, -50L), class = c("tbl_df", "tbl", "data.frame"
))


I did run the two sample KS test as follows:

ks.test(data$$test1, data$$test2)


Why does the test result indicate that the two distributions are similar?

Two-sample Kolmogorov-Smirnov test

data:  Data$$test1 and Data$$test2
D = 0, p-value = 1
alternative hypothesis: two-sided


If I plot test1 and test2, visual inspection obviously shows that the two distributions are not similar.

> plot(Data$$test1, type="l",col="red") > lines(Data$$test2,col="green")


• What does a histogram of each dataset look like? If I make a vector x=rnorm(10) then run plot(x, type="l") and plot(x[10:1], type="l") I will get two very different plots because of the type of plot. Their histograms will be identical. the ks test would return a value of 1 as well. Commented Jun 10, 2020 at 16:47
• The histograms are identical indeed. Why is there such a difference between the line plots and histograms? I thought the KS test focuses on the distribution as laid down in the line plot. Commented Jun 10, 2020 at 16:51
• A distribution is best visualised by a histogram (at least, for univariate data). A line plot is displaying the data by the order it was entered into R (which might perhaps correspond to the order it was observed). When we don't take "time" into account x and x[10:1] are identical Commented Jun 10, 2020 at 16:57
• The K-s test compares ECDFs, not 'line plots'. Your plots show data in order of listing, which is different btw the two samples. They are not directly relevant to the K-S test. See ECDFs in my Answer. Commented Jun 10, 2020 at 18:08

The distributions are not only similar, they are identical.

Your two columns contain the same values, only shifted circularly. The KS test is not any sort of paired test; it's comparing two distributions for which the order does not matter.

• But that means it ignores the location (mean) of the distribution, correct? I am aiming to test the similarity of both the shape and location of the two distributions as shown in the line plots. Commented Jun 10, 2020 at 16:52
• @Jonathan It doesn't look like you are plotting distributions, you are just plotting the data. The distribution of the sample (or an approximation of the underlying distribution) is from a histogram of the data. Commented Jun 10, 2020 at 16:54

Outline for 'troubleshooting' test and data:

test1 = c(0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07,
0.06, 0.13, 0.14, 2, 0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26,
0.14, 0.19, 0.38, 7, 1, 1, 1, 1, 2, 1, 0.74, 0.53, 1, 1, 0.32,
0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 0.03, 0.04, 0.01, 0.01,
0.01, 0.01, 0.01, 0.004, 0.01, 0.95)

test2 = c(1, 1, 1, 2, 1,
0.74, 0.53, 1, 1, 0.32, 0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02,
0.03, 0.04, 0.01, 0.01, 0.01, 0.01, 0.01, 0.004, 0.01, 0.95,
0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 0.06, 0.13, 0.14, 2,
0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 0.14, 0.19, 0.38, 7, 1)

ks.test(test1, test2)

Two-sample Kolmogorov-Smirnov test

data:  test1 and test2
D = 0, p-value = 1
alternative hypothesis: two-sided

Warning message:
In ks.test(test1, test2) : cannot compute exact p-value with ties


What is the source of the ties?

length(test1)
[1] 50
length(unique(test1))
[1] 32
length(test2)
[1] 50
length(unique(test2))
[1] 32


Each sample contains 18 duplicated observations.

Finally, the K-S test works by comparing empirical cumulative distribution functions (ECDFs). Here are the two ECDFs: one with orange dots, the other with blue +s. As @BryanKrause (+1) says, the two samples have identical observations---even if they are listed in different orders.

plot(ecdf(test1), col="orange")
lines(ecdf(test2), col="blue", pch="+")


In this instance, Warning notwithstanding, the K-S test has given exactly the correct P-value, which is $$1.$$

Here are sorted listings of the data, which make it easy to spot ties-- both within and between samples:

sort(test1)
[1] 0.004 0.010 0.010 0.010 0.010 0.010 0.010 0.020 0.030 0.030
[11] 0.040 0.040 0.060 0.070 0.090 0.100 0.100 0.110 0.130 0.130
[21] 0.140 0.140 0.160 0.170 0.180 0.190 0.210 0.230 0.260 0.260
[31] 0.290 0.320 0.330 0.350 0.380 0.430 0.530 0.560 0.740 0.950
[41] 1.000 1.000 1.000 1.000 1.000 1.000 1.000 2.000 2.000 7.000
sort(test2)
[1] 0.004 0.010 0.010 0.010 0.010 0.010 0.010 0.020 0.030 0.030
[11] 0.040 0.040 0.060 0.070 0.090 0.100 0.100 0.110 0.130 0.130
[21] 0.140 0.140 0.160 0.170 0.180 0.190 0.210 0.230 0.260 0.260
[31] 0.290 0.320 0.330 0.350 0.380 0.430 0.530 0.560 0.740 0.950
[41] 1.000 1.000 1.000 1.000 1.000 1.000 1.000 2.000 2.000 7.000


Also, data summaries are identical, so there is no difference in sample means as you claim.

summary(test1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.0040  0.0625  0.1850  0.5167  0.5525  7.0000
summary(test2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.0040  0.0625  0.1850  0.5167  0.5525  7.0000