Kolmogorov-Smirnov test with dependent data

Let's say, I have 100 students, each of them have 2 scores: reading score and writing score, so basically I will have 2 vectors with the length of each is 100.

I want to test the distribution of reading score and writing score are the same or not, so I want to use Kolmogorov-Smirnov test, but is it okay to run the test with dependent data?

• Welcome to our site! I've removed the reference to R in the title, because your question is really completely software-agnostic - it's about the underlying statistical issue of whether the procedure is valid.. Nov 4 '15 at 15:58
• Did you find a way to test the distributions for these paired dependent samples? I am facing the same problem and not sure how to proceed. Apr 13 '18 at 10:10

the Kolmogorov-Smirnov test is only to be used if the samples are independent. However, here you can find a script that does the job in form of a permutation test (in R).

• Can you explain the permutation test you're suggesting? How does it work? Nov 4 '15 at 22:04
• No, actually not. When I was posting this answer, I was a bit in a rush and I thought a simple one-liner would be a little rude/insufficient. So, I quickly googled the problem and posted what I found without actually spending much time to check whether the proposed solution is indeed a good one. Nov 5 '15 at 8:07
• For future readers, the linked permuation test is deeply flawed: It calculates a CI for the ks statistics by randomly selecting about half of the data many times, and then comparing it to the ks statisic of the original two datasets. But since the Ks statistic is dependent on the number of observations used (and the resampled use only about half of the full dataset), the resulting pvalue will be way off. Sep 18 '18 at 8:19

The Kolmogorov-Smirnov test is designed for two independent samples. You need a test for paired observations. I do not think anyone has ever proposed a KS test analog for paired observations. Also, it is a very cautious test, that is, it has low power in many circumstances.

I suggest a paired nonparametric test, either a sign test or a Wilcoxon signed-rank test, both based on the differences between the two scores for each subject. The latter is more common and has strong power for observations that have a normal distribution and a wide range of non-normal distributions.

If the two measures are strongly correlated then it is possible that the sign test would be quite powerful, possibly more so than the Wilcoxon signed rank test. Both tests are based on permutations of the observations: the signed differences for the Wilcoxon signed rank test and only the signs of the differences for the sign test.

Friedman's test is usually used for three or more measurements. If there are only two measurements, it gives the same p-value results as the Wilcoxon test.

• So the two samples have to be independent, but can the observations within the sample be non-independent (related to one another)? Sep 4 '21 at 14:43
• Independence of observations is the usual foundation of the basic, commonly used statistical tests. You would have to tell us how observations might be correlated. Sep 4 '21 at 22:49
• For instance, I am comparing the similarity between observed and predicted precipitation (for each pixel in an area) from two different models (independent samples) by using the two sample KS Test. The predicted model predicts for each pixel independently, so the precipitation values of each pixel are indenpendent of another. However, since this is a time series data, and a precipation event for each day will be resposible for each pixel value, thus maybe (I am not sure) the obsevations in one sample are correlated. PS, I have seen multiple highly cited papers use the KS for such data analysis Sep 4 '21 at 23:07

I believe what you need is a (Related-Samples) Friedman's Two-way Analysis of Variance by Ranks (see this). It comes with the a pair of null hypotheses about the distributions you are comparing that you can reject/retain depending on the p value:

• H0(a): "The populations represented by the k conditions have the same distribution of scores."
• H0(b): "The population has the same distribution of scores on the different measures represented by the conditions."

Or as stated in (2): "For Friedman’s two-way analysis of variance by ranks, the null hypothesis states that the K repeated measures or matched groups come from the same population or from populations with the same median (1). Under the null hypothesis, the test assumes that the response variable has the same underlying continuous distribution;"

Not very relevant to R, but the aforementioned test is also what IBM offers in the (v24) SPSS statistical package for testing if k related samples have been drawn from the same population (see IBM SPSS v24 Documentation):

• "Compare Distributions. Friedman's 2-way ANOVA by ranks (k samples) produces a related samples test of whether k related samples have been drawn from the same population. You can optionally request multiple comparisons of the k samples, either All pairwise multiple comparisons or Stepwise step-down comparisons."

References

• (1) Siegel, S. (1956). Nonparametric statistics for the behavioral sciences.
• (2) Pereira, D. G., Afonso, A., & Medeiros, F. M. (2015). Overview of Friedman’s test and post-hoc analysis. Communications in Statistics-Simulation and Computation, 44(10), 2636-2653.