1
$\begingroup$

I'm trying to understand the output of the Kolmogorov-Smirnov two sample test. I have two columns, test1 and test2. I'm having some difficulty in understanding the interpretation of the counterintuitive results I am getting.

structure(list(test1 = c(0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 
0.06, 0.13, 0.14, 2, 0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 
0.14, 0.19, 0.38, 7, 1, 1, 1, 1, 2, 1, 0.74, 0.53, 1, 1, 0.32, 
0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 0.03, 0.04, 0.01, 0.01, 
0.01, 0.01, 0.01, 0.004, 0.01, 0.95), test2 = c(1, 1, 1, 2, 1, 
0.74, 0.53, 1, 1, 0.32, 0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 
0.03, 0.04, 0.01, 0.01, 0.01, 0.01, 0.01, 0.004, 0.01, 0.95, 
0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 0.06, 0.13, 0.14, 2, 
0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 0.14, 0.19, 0.38, 7, 
1)), row.names = c(NA, -50L), class = c("tbl_df", "tbl", "data.frame"
))

I did run the two sample KS test as follows:

ks.test(data$test1, data$test2)

Why does the test result indicate that the two distributions are similar?

Two-sample Kolmogorov-Smirnov test

data:  Data$test1 and Data$test2
D = 0, p-value = 1
alternative hypothesis: two-sided

If I plot test1 and test2, visual inspection obviously shows that the two distributions are not similar.

> plot(Data$test1, type="l",col="red")
> lines(Data$test2,col="green")

enter image description here

$\endgroup$
4
  • 1
    $\begingroup$ What does a histogram of each dataset look like? If I make a vector x=rnorm(10) then run plot(x, type="l") and plot(x[10:1], type="l") I will get two very different plots because of the type of plot. Their histograms will be identical. the ks test would return a value of 1 as well. $\endgroup$
    – jcken
    Commented Jun 10, 2020 at 16:47
  • $\begingroup$ The histograms are identical indeed. Why is there such a difference between the line plots and histograms? I thought the KS test focuses on the distribution as laid down in the line plot. $\endgroup$
    – Jonathan
    Commented Jun 10, 2020 at 16:51
  • $\begingroup$ A distribution is best visualised by a histogram (at least, for univariate data). A line plot is displaying the data by the order it was entered into R (which might perhaps correspond to the order it was observed). When we don't take "time" into account x and x[10:1] are identical $\endgroup$
    – jcken
    Commented Jun 10, 2020 at 16:57
  • $\begingroup$ The K-s test compares ECDFs, not 'line plots'. Your plots show data in order of listing, which is different btw the two samples. They are not directly relevant to the K-S test. See ECDFs in my Answer. $\endgroup$
    – BruceET
    Commented Jun 10, 2020 at 18:08

2 Answers 2

3
$\begingroup$

The distributions are not only similar, they are identical.

Your two columns contain the same values, only shifted circularly. The KS test is not any sort of paired test; it's comparing two distributions for which the order does not matter.

$\endgroup$
2
  • $\begingroup$ But that means it ignores the location (mean) of the distribution, correct? I am aiming to test the similarity of both the shape and location of the two distributions as shown in the line plots. $\endgroup$
    – Jonathan
    Commented Jun 10, 2020 at 16:52
  • $\begingroup$ @Jonathan It doesn't look like you are plotting distributions, you are just plotting the data. The distribution of the sample (or an approximation of the underlying distribution) is from a histogram of the data. $\endgroup$ Commented Jun 10, 2020 at 16:54
2
$\begingroup$

Outline for 'troubleshooting' test and data:

test1 = c(0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 
 0.06, 0.13, 0.14, 2, 0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 
 0.14, 0.19, 0.38, 7, 1, 1, 1, 1, 2, 1, 0.74, 0.53, 1, 1, 0.32, 
 0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 0.03, 0.04, 0.01, 0.01, 
 0.01, 0.01, 0.01, 0.004, 0.01, 0.95)

test2 = c(1, 1, 1, 2, 1, 
 0.74, 0.53, 1, 1, 0.32, 0.17, 0.11, 0.09, 0.1, 0.04, 0.03, 0.02, 
 0.03, 0.04, 0.01, 0.01, 0.01, 0.01, 0.01, 0.004, 0.01, 0.95, 
 0.43, 0.16, 0.18, 0.13, 0.1, 0.23, 0.07, 0.06, 0.13, 0.14, 2, 
 0.21, 0.29, 0.35, 0.33, 0.56, 0.26, 0.26, 0.14, 0.19, 0.38, 7, 1)

ks.test(test1, test2)

        Two-sample Kolmogorov-Smirnov test

data:  test1 and test2
D = 0, p-value = 1
alternative hypothesis: two-sided

Warning message:
In ks.test(test1, test2) : cannot compute exact p-value with ties

What is the source of the ties?

length(test1)
[1] 50
length(unique(test1))
[1] 32
length(test2)
[1] 50
length(unique(test2))
[1] 32

Each sample contains 18 duplicated observations.

Finally, the K-S test works by comparing empirical cumulative distribution functions (ECDFs). Here are the two ECDFs: one with orange dots, the other with blue +s. As @BryanKrause (+1) says, the two samples have identical observations---even if they are listed in different orders.

plot(ecdf(test1), col="orange")
 lines(ecdf(test2), col="blue", pch="+")

enter image description here

In this instance, Warning notwithstanding, the K-S test has given exactly the correct P-value, which is $1.$

Here are sorted listings of the data, which make it easy to spot ties-- both within and between samples:

sort(test1)
 [1] 0.004 0.010 0.010 0.010 0.010 0.010 0.010 0.020 0.030 0.030
[11] 0.040 0.040 0.060 0.070 0.090 0.100 0.100 0.110 0.130 0.130
[21] 0.140 0.140 0.160 0.170 0.180 0.190 0.210 0.230 0.260 0.260
[31] 0.290 0.320 0.330 0.350 0.380 0.430 0.530 0.560 0.740 0.950
[41] 1.000 1.000 1.000 1.000 1.000 1.000 1.000 2.000 2.000 7.000
sort(test2)
 [1] 0.004 0.010 0.010 0.010 0.010 0.010 0.010 0.020 0.030 0.030
[11] 0.040 0.040 0.060 0.070 0.090 0.100 0.100 0.110 0.130 0.130
[21] 0.140 0.140 0.160 0.170 0.180 0.190 0.210 0.230 0.260 0.260
[31] 0.290 0.320 0.330 0.350 0.380 0.430 0.530 0.560 0.740 0.950
[41] 1.000 1.000 1.000 1.000 1.000 1.000 1.000 2.000 2.000 7.000

Also, data summaries are identical, so there is no difference in sample means as you claim.

summary(test1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0040  0.0625  0.1850  0.5167  0.5525  7.0000 
summary(test2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0040  0.0625  0.1850  0.5167  0.5525  7.0000 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.