I'm having some difficulty in understanding the interpretation of the 2 sample KS test, and how it is different from a regular t test between 2 groups.
Lets say I have males and females doing some task and I collect some scores from that task. My ultimate goal is to determine whether males and females perform differently on that task
So one thing I could do is run a t test between the 2 groups. Another thing I could do is calculate the ECDF for males and females, plot them, and conduct the 2 sample KS test. I would get something like this:
The null hypothesis for the KS test is that the 2 sets of continuous score distributions come from the same population
When conducting the KS test, I get: D = 0.18888, p-value = 0.04742
First, I want to check that my interpretation of the results is correct. Here, I would reject the null hypothesis and say that male and female score distributions come from different populations. Or in other words, the distribution of male and female scores are different from each other.
More specifically, males tend to have a higher probability of achieving lower scores on this task, and that is the difference between the 2 sexes as I interpret from the plot
Now a t test will test the difference between male and female means on the score variable.
Lets imagine the case where male performance is worse than females in this task. In that case, the distribution of male scores will center around a low mean, whereas female score distribution will center around a high mean. This scenario would be in line with the plot above, as males will have a higher probability of achieving lower scores
If the t test comes out to be significant, I would conclude that females score, on average, significantly higher than males. Or in population terms, female scores are drawn from a population whose mean is higher than the male population, which sounds very similar to the KS conclusion that they come from different populations.
What's the difference?
So the conclusion I would draw in both the KS and t test cases are the same. Males perform poorly relative to females. So what is the benefit of using one test over the other? Is there any new knowledge that you can gain from using the KS test?
The way I see it, males with a distribution centered around a low mean, and females centering around a high mean is what causes the significant t test. But by that very same fact, males will have a higher probability of scoring lower values, which would cause the plot to look like above and giving a significant KS test. So the results of both tests have the same underlying cause, but maybe one could argue that a KS test takes into account more than just the means of the distributions and also considers the shape of the distribution, but is it possible to parse out the cause of the significant KS test from just the test results?
So what is the value in running a KS test over a t test? And lets assume that I can meet the assumptions of the t test for this question