t test vs goodness of fit test

Question

Suppose I have two groups of data. I want to know if their distributions and / or means are different or not. Is it appropriate to do a two-sample t-test? Or should I go for a 2 sample goodness of fit test like 2D-KS test?

Answer 1

The basic principle is to use each test for what it was designed to do. Here are specific examples:

Example 1: Use t test. If you're trying to see whether two normal samples have the same mean, then use a t test in preference to a K-S test. This is especially true if samples are small so that the K-S test has relatively poor power. In the example below, the t test rejects $H_0$ at the 5% level, and the K-S test does not.

set.seed(706)
x1 = rnorm(15, 100, 15)
x2 = rnorm(15, 120, 15)
t.test(x1,x2)$p.val
[1] 0.04862413

ks.test(x1,x2)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.4, p-value = 0.1844
alternative hypothesis: two-sided

mean(x1); sd(x1)
[1] 101.7776
[1] 20.05416
mean(x2); sd(x2)
[1] 115.5439
[1] 16.2465

The K-S test compares sample ECDFs. The K-S test statistic $D$ is the maximum vertical distance between the two ECDFs. For significance in small samples, $D$ must be fairly large. [There is a shift in means, but the vertical distance betwee the the ECDFs is never great--relative to the jump sizes at each observation.]

plot(ecdf(x1), col="blue", main="ECDFs of Two Samples")
  lines(ecdf(x2), col="orange")

Example 2: Use K-S test. If you're trying to distinguish between shapes of distributions that have similar means and variances, then use the K-S test in preference to the t test. In the example below, the K-S test rejects $H_0$ at the 5% level, and the t test does not.

set.seed(2020)
y1 = rnorm(85, 1, 1)  
y2 = rexp(85,1)

summary(y1);  sd(y1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-2.0388  0.4949  1.1188  1.1171  1.7206  4.2016 
[1] 1.094069

summary(y2);  sd(y2)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.008152 0.282684 0.764139 1.049091 1.570979 4.116282 
[1] 0.982902

t.test(y1,y2)$p.val
[1] 0.6702003

ks.test(y1,y2)

    Two-sample Kolmogorov-Smirnov test

data:  y1 and y2
D = 0.21176, p-value = 0.04395
alternative hypothesis: two-sided

plot(ecdf(y1), col="blue", main="ECDFs of Two Samples")
  lines(ecdf(y2), col="orange")