32
$\begingroup$

I'm having some difficulty in understanding the interpretation of the 2 sample KS test, and how it is different from a regular t test between 2 groups.

Lets say I have males and females doing some task and I collect some scores from that task. My ultimate goal is to determine whether males and females perform differently on that task

So one thing I could do is run a t test between the 2 groups. Another thing I could do is calculate the ECDF for males and females, plot them, and conduct the 2 sample KS test. I would get something like this:

enter image description here

K-S test

The null hypothesis for the KS test is that the 2 sets of continuous score distributions come from the same population

When conducting the KS test, I get: D = 0.18888, p-value = 0.04742

First, I want to check that my interpretation of the results is correct. Here, I would reject the null hypothesis and say that male and female score distributions come from different populations. Or in other words, the distribution of male and female scores are different from each other.

More specifically, males tend to have a higher probability of achieving lower scores on this task, and that is the difference between the 2 sexes as I interpret from the plot

T-test

Now a t test will test the difference between male and female means on the score variable.

Lets imagine the case where male performance is worse than females in this task. In that case, the distribution of male scores will center around a low mean, whereas female score distribution will center around a high mean. This scenario would be in line with the plot above, as males will have a higher probability of achieving lower scores

If the t test comes out to be significant, I would conclude that females score, on average, significantly higher than males. Or in population terms, female scores are drawn from a population whose mean is higher than the male population, which sounds very similar to the KS conclusion that they come from different populations.

What's the difference?

So the conclusion I would draw in both the KS and t test cases are the same. Males perform poorly relative to females. So what is the benefit of using one test over the other? Is there any new knowledge that you can gain from using the KS test?

The way I see it, males with a distribution centered around a low mean, and females centering around a high mean is what causes the significant t test. But by that very same fact, males will have a higher probability of scoring lower values, which would cause the plot to look like above and giving a significant KS test. So the results of both tests have the same underlying cause, but maybe one could argue that a KS test takes into account more than just the means of the distributions and also considers the shape of the distribution, but is it possible to parse out the cause of the significant KS test from just the test results?

So what is the value in running a KS test over a t test? And lets assume that I can meet the assumptions of the t test for this question

Improve this question
$\endgroup$
8
  • $\begingroup$ Classic t-test is by large inferior to Bayesian data analysis, check out John Kruschke's "Bayesian Estimation supersedes the t test" indiana.edu/~kruschke/BEST/BEST.pdf $\endgroup$
    – Vladislavs Dovgalecs
    Apr 21, 2016 at 3:42
  • 1
    $\begingroup$ I'm not sure how the KS test relates to Bayesian methods...? $\endgroup$
    – Simon
    Apr 21, 2016 at 3:52
  • $\begingroup$ Just stop using KS and t-test $\endgroup$
    – Vladislavs Dovgalecs
    Apr 21, 2016 at 5:20
  • 5
    $\begingroup$ @xeon If you're going to make such strong statements, you had better support them. Your advice would be of no use at picking up the kind of difference in the example in my answer. Why should one abandon an approach that clearly works at identifying this difference in distributions in favor of one that doesn't? $\endgroup$
    – Glen_b
    Apr 21, 2016 at 10:19
  • 2
    $\begingroup$ @Glen_b That's why a wrote a comment, not an answer. May be OP didn't read the paper which is great btw; I merely wanted to suggest it. But I agree that I made too strong statement and behaved little bit snob. I apologize for being bit rude. Will not happen anymore. $\endgroup$
    – Vladislavs Dovgalecs
    Apr 21, 2016 at 16:08

1 Answer 1

Reset to default
29
$\begingroup$

As an example of why you'd want to use the two sample Kolmogorov-Smirnov test:

Imagine that the population means were similar but the variances were very different. The Kolmogorov-Smirnov test could pick this difference up but the t-test cannot.

Or imagine that the distributions have similar means and sd's but the males have a bimodal distribution (red) while the females (blue) don't:

enter image description here

Do males and females perform differently? Yes -- the males tend to either score somewhere around 7.5-8 or 12.5-13, while the females tend more often to score more toward the middle (near 10 or so) but are much less clustered about that value than the two values the males tend to score near to.

So the Kolmogorov-Smirnov can find much more general kinds of difference in distribution than the t-test can.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Ah, make sense. Could I extend that logic and say that if a t-test is significant, then the KS test will also likely be significant, however it could be due to mean difference and/or any other difference in the distribution, thus making interpretation of the KS test difficult? So is a KS test only really useful in the event that there is no mean difference between 2 groups? $\endgroup$
    – Simon
    Apr 21, 2016 at 10:26
  • 1
    $\begingroup$ The t-test is more sensitive to differences in mean (particularly if the populations distributions are close to normal with similar standard deviation). The KS test can be harder to interpret, but I wouldn't agree with your last sentence. You could have a small difference in means that is accompanied by other differences; the t-test only has the difference in means to inform it, while the KS test can be informed by the other kinds of differences. Imagine the above example, but where there's a small shift in means as well; the t-test may not pick up the difference as easily as the KS test does. $\endgroup$
    – Glen_b
    Apr 21, 2016 at 10:37
  • $\begingroup$ @Glen_b: is it then right to say that K-S tests whether the distributions are equal while the t-test tests whether the distributions have the same mean? $\endgroup$
    – user83346
    Apr 21, 2016 at 17:52
  • $\begingroup$ @fcop Yes and no; given the assumptions, and under the null, the ordinary equal variance t-test is actually testing for identity of distributions as well -- it's the generality of the alternative (combined with the assumptions) that really makes them different. Of course we can (and generally do) use the tests when their assumptions don't quite apply and then we're more looking at their behavior under the null and alternative; the t-test will tend to be sensitive to a change in mean under the alternative, while the KS is somewhat sensitive to a very broad class of alternatives. $\endgroup$
    – Glen_b
    Apr 21, 2016 at 23:52
  • $\begingroup$ @Glen_b I realize that this post is > 4 years old, but what about Welch's t-test vs. Kolmogorov-Smirnov? Welch's t-test accounts for variance $\endgroup$
    – con
    Aug 26, 2020 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.