From the docs

This is a two-sided test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution

This is a two-sided test for the null hypothesis that 2 independent samples have identical average (expected) values. This test assumes that the populations have identical variances by default.

Assuming that one uses the default assumption of identical variances, the second test seems to be testing for identical distribution as well. The only difference then appears to be that the first test assumes continuous distributions.

If that is the case, what are the differences between the two tests? What is the right interpretation if they have very different results? For example I have two data sets for which the p values are 0.95 and 0.04 for the ttest(tt_equal_var=True) and the ks test, respectively.

  • $\begingroup$ Your question is really about when to use the independent samples t-test and when to use the Kolmogorov-Smirnov two sample test; the fact of their implementation in scipy is entirely beside the point in relation to that issue (I'd remove that bit). I am curious that you don't seem to have considered the (Wilcoxon-)Mann-Whitney test in your comparison (scipy.stats.mannwhitneyu), which many people would tend to regard as the natural "competitor" to the t-test for suitability to similar kinds of problems. Is there a reason for that? Can you show the data sets for which you got dissimilar results? $\endgroup$
    – Glen_b
    Jul 16 '18 at 7:33
  • $\begingroup$ I was not aware of the W-M-W test. Had a read over it and it seems indeed a better fit. The data is truncated at 0 and has a shape a bit like a chi-square dist. $\endgroup$
    – chrise
    Jul 16 '18 at 8:36
  • $\begingroup$ When you say it's truncated at 0, ... can you elaborate? Are <0 recorded as 0 (censored/Winsorized) or are there simply no values that would have been <0 at all -- they're not observed/not in the sample (distribution is actually truncated)? $\endgroup$
    – Glen_b
    Jul 17 '18 at 3:48
  • $\begingroup$ The distribution naturally only has values >= 0 $\endgroup$
    – chrise
    Jul 17 '18 at 6:24
  • $\begingroup$ Ah. I wouldn't call that truncated at all. $\endgroup$
    – Glen_b
    Jul 17 '18 at 6:31

The two-sample t-test assumes that the samples are drawn from Normal distributions with identical variances*, and is a test for whether the population means differ.

* specifically for its level to be correct, you need this assumption when the null hypothesis is true. Confidence intervals would also assume it under the alternative.

However the t-test is somewhat level robust to the distributional assumption (that is, its significance level is not heavily impacted by moderator deviations from the assumption of normality), particularly in large samples. Further, it is not heavily impacted by moderate differences in variance. If the sample sizes are very nearly equal it's pretty robust to even quite unequal variances.

If the the assumptions are true, the t-test is good at picking up a difference in the population means.

(If the distribution is heavy tailed, the t-test may have low power compared to other possible tests for a location-difference.)

The two-sample Kolmogorov-Smirnov test attempts to identify any differences in distribution of the populations the samples were drawn from. It is distribution-free. This means that (under the null) you can have the samples drawn from any continuous distribution, as long as it's the same one for both samples. It is weaker than the t-test at picking up a difference in the mean but it can pick up other kinds of difference that the t-test is blind to.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.