I was reading up on the KS test and I got a doubt why is it not okay to check with this test when there is a statistically significant difference between the two samples:

  1. Sample 1, taken from t=1 to 100 and
  2. Sample 2, taken from T=51 to 150

I'm having difficulty in reasoning about this. Any help would be highly appreciated.

  • 1
    $\begingroup$ Looks like observations T=51 to T=100 belong to both samples. The samples under comparison should not overlap. $\endgroup$ Feb 7 at 17:05
  • $\begingroup$ What do you mean by t=1 to 100? $\endgroup$
    – Dave
    Feb 7 at 17:26
  • $\begingroup$ Sorry, for the mnemonic, by 't' I define to mean the timepoints $\endgroup$
    – Ranji Raj
    Feb 7 at 17:51
  • $\begingroup$ @RichardHardy, if the time frames are overlapping, what essentially does this overlap signify? Is it okay to test the statistical test? If not, what can be the possible danger? $\endgroup$
    – Ranji Raj
    Feb 8 at 12:21
  • 1
    $\begingroup$ For overlapping samples, no test result can be interpreted in the usual way. (Think of a pair of samples where all observations but one overlap.) I am afraid the accepted answer ignores this and thus might be misleading. $\endgroup$ Feb 8 at 12:55

A more complete description of your data and objectives would be required for a full discussion of your question. (Above, @RichardHardy has mentioned that your samples seem to overlap. In a comment on this Answer, @whuber has mentioned that independence cannot be assumed among observations in a time series. You would need to address both of these issues.)

However, even in many situations with disjoint, independent random samples, I would use the 2-sample Kolmogorov-Smirnov test only if no other appropriate tests are available.

In particular, I would not use a Kolmogorov-Smirnov test in preference to a Welch t test to look for a difference in means. In many situations, the K-S test is too reluctant to declare differences. between two normal samples.

Differences in means, significance level: For example if I have independent, random samples of sizes 100 and 50 from the same normal population, the K-S test rejects a test at the nominal 5% level more like 4% of the time. By contrast, a Welch two-sample t test rejects very nearly the anticipated 5% of the time. In the simulations below with 100,000 tests of each type, one can expect about two place accuracy for rejection probabilities.

pv.ks = replicate(10^5, ks.test(rnorm(100),rnorm(50))$p.val)
mean(pv.ks <= .05)
[1] 0.04117
pv.wt = replicate(10^5, t.test(rnorm(100),rnorm(50))$p.val) 
mean(pv.wt <= .05)
[1] 0.04931

Differences in means, power. Even worse, for the same sample sizes as before, the K-S test has power only about 67% to detect a difference between sampling from $\mathsf{Norm}(0,1)$ for the larger sample and from $\mathsf{Norm}(.5,1)$ for the smaller sample. By contrast, the Welch t test has power about 81% for the same task.

pv.ks = replicate(10^5, ks.test(rnorm(100),rnorm(50,.5,1))$p.val)
mean(pv.ks <= .05)
[1] 0.66591
pv.wt = replicate(10^5, t.test(rnorm(100),rnorm(50,.5,1))$p.val) 
mean(pv.wt <= .05)
[1] 0.81416

Difference in variances, power. If we are looking at two normal samples of size 20 to detect a difference between variances $\sigma^2=1$ and $\sigma^2=4.$ then the usual F-test, implemented in R as var.test--hardly known for its good power--finds a difference with probability about 84%, but the K-S test is much worse with power below 12%.

pv.ks = replicate(10^5, ks.test(rnorm(20),rnorm(20,0,2))$p.val)
mean(pv.ks <= .05)
[1] 0.11434
pv.f = replicate(10^5, var.test(rnorm(20),rnorm(20,0,2))$p.val) 
mean(pv.f <= .05)
[1] 0.83552
  • 1
    $\begingroup$ Did you notice the samples are overlapping? As such, no test result can be interpreted in the usual way without adjustments. (Think of a pair of samples where all observations but one overlap.) $\endgroup$ Feb 8 at 12:53
  • 1
    $\begingroup$ Also pertinent is the likelihood (since these appear to be time series) that the observations are not independent. The p-value of any distributional test is therefore not to be trusted. $\endgroup$
    – whuber
    Feb 8 at 14:13
  • 1
    $\begingroup$ @whuber. In the simulations of my answer the samples are IID. In addition to your comment about possible time series there is the issue of possible overlap between OP's samples. I should have mentioned both difficulties in the first sentences of my answer. $\endgroup$
    – BruceET
    Feb 8 at 17:02
  • $\begingroup$ @RichardHardy. I saw your previous comment about overlap, I hope I have made it clear in my answer that samples in my simulations are disjoint, random, and independent. $\endgroup$
    – BruceET
    Feb 8 at 17:18
  • $\begingroup$ @BruceET, thanks. It might be the case that your answer is great on its own but not for the OP's question (for which it might even be misleading by diverting attention from the fact that none of the tests is applicable on OPs data as is). This is why I commented. Maybe a better idea would be to post a new question where your answer would fit perfectly, but remove it from here where it might be misleading. But this is just my 2 cents. $\endgroup$ Feb 8 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.