7
$\begingroup$

Let's say, I have 100 students, each of them have 2 scores: reading score and writing score, so basically I will have 2 vectors with the length of each is 100.

I want to test the distribution of reading score and writing score are the same or not, so I want to use Kolmogorov-Smirnov test, but is it okay to run the test with dependent data?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to our site! I've removed the reference to R in the title, because your question is really completely software-agnostic - it's about the underlying statistical issue of whether the procedure is valid.. $\endgroup$
    – Silverfish
    Nov 4, 2015 at 15:58
  • $\begingroup$ Did you find a way to test the distributions for these paired dependent samples? I am facing the same problem and not sure how to proceed. $\endgroup$
    – Confounded
    Apr 13, 2018 at 10:10

3 Answers 3

Reset to default
1
$\begingroup$

the Kolmogorov-Smirnov test is only to be used if the samples are independent. However, here you can find a script that does the job in form of a permutation test (in R).

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ Can you explain the permutation test you're suggesting? How does it work? $\endgroup$
    – Glen_b
    Nov 4, 2015 at 22:04
  • $\begingroup$ No, actually not. When I was posting this answer, I was a bit in a rush and I thought a simple one-liner would be a little rude/insufficient. So, I quickly googled the problem and posted what I found without actually spending much time to check whether the proposed solution is indeed a good one. $\endgroup$
    – userE
    Nov 5, 2015 at 8:07
  • 5
    $\begingroup$ For future readers, the linked permuation test is deeply flawed: It calculates a CI for the ks statistics by randomly selecting about half of the data many times, and then comparing it to the ks statisic of the original two datasets. But since the Ks statistic is dependent on the number of observations used (and the resampled use only about half of the full dataset), the resulting pvalue will be way off. $\endgroup$
    – Leander Moesinger
    Sep 18, 2018 at 8:19
1
$\begingroup$

The Kolmogorov-Smirnov test is designed for two independent samples. You need a test for paired observations. I do not think anyone has ever proposed a KS test analog for paired observations. Also, it is a very cautious test, that is, it has low power in many circumstances.

I suggest a paired nonparametric test, either a sign test or a Wilcoxon signed-rank test, both based on the differences between the two scores for each subject. The latter is more common and has strong power for observations that have a normal distribution and a wide range of non-normal distributions.

If the two measures are strongly correlated then it is possible that the sign test would be quite powerful, possibly more so than the Wilcoxon signed rank test. Both tests are based on permutations of the observations: the signed differences for the Wilcoxon signed rank test and only the signs of the differences for the sign test.

Friedman's test is usually used for three or more measurements. If there are only two measurements, it gives the same p-value results as the Wilcoxon test.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So the two samples have to be independent, but can the observations within the sample be non-independent (related to one another)? $\endgroup$
    – Arthur_Morgan
    Sep 4, 2021 at 14:43
  • 1
    $\begingroup$ Independence of observations is the usual foundation of the basic, commonly used statistical tests. You would have to tell us how observations might be correlated. $\endgroup$
    – David Smith
    Sep 4, 2021 at 22:49
  • $\begingroup$ For instance, I am comparing the similarity between observed and predicted precipitation (for each pixel in an area) from two different models (independent samples) by using the two sample KS Test. The predicted model predicts for each pixel independently, so the precipitation values of each pixel are indenpendent of another. However, since this is a time series data, and a precipation event for each day will be resposible for each pixel value, thus maybe (I am not sure) the obsevations in one sample are correlated. PS, I have seen multiple highly cited papers use the KS for such data analysis $\endgroup$
    – Arthur_Morgan
    Sep 4, 2021 at 23:07
0
$\begingroup$

I believe what you need is a (Related-Samples) Friedman's Two-way Analysis of Variance by Ranks (see this). It comes with the a pair of null hypotheses about the distributions you are comparing that you can reject/retain depending on the p value:

  • H0(a): "The populations represented by the k conditions have the same distribution of scores."
  • H0(b): "The population has the same distribution of scores on the different measures represented by the conditions."

Or as stated in (2): "For Friedman’s two-way analysis of variance by ranks, the null hypothesis states that the K repeated measures or matched groups come from the same population or from populations with the same median (1). Under the null hypothesis, the test assumes that the response variable has the same underlying continuous distribution;"

Not very relevant to R, but the aforementioned test is also what IBM offers in the (v24) SPSS statistical package for testing if k related samples have been drawn from the same population (see IBM SPSS v24 Documentation):

  • "Compare Distributions. Friedman's 2-way ANOVA by ranks (k samples) produces a related samples test of whether k related samples have been drawn from the same population. You can optionally request multiple comparisons of the k samples, either All pairwise multiple comparisons or Stepwise step-down comparisons."

References

  • (1) Siegel, S. (1956). Nonparametric statistics for the behavioral sciences.
  • (2) Pereira, D. G., Afonso, A., & Medeiros, F. M. (2015). Overview of Friedman’s test and post-hoc analysis. Communications in Statistics-Simulation and Computation, 44(10), 2636-2653.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Please add further details to expand on your answer, such as working code or documentation citations. $\endgroup$
    – Community Bot
    Aug 29, 2021 at 12:38
  • $\begingroup$ Please provide additional details in your answer. As it's currently written, it's hard to understand your solution. $\endgroup$
    – Community Bot
    Aug 29, 2021 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.