# t test vs goodness of fit test

Suppose I have two groups of data. I want to know if their distributions and / or means are different or not. Is it appropriate to do a two-sample t-test? Or should I go for a 2 sample goodness of fit test like 2D-KS test?

The basic principle is to use each test for what it was designed to do. Here are specific examples:

Example 1: Use t test. If you're trying to see whether two normal samples have the same mean, then use a t test in preference to a K-S test. This is especially true if samples are small so that the K-S test has relatively poor power. In the example below, the t test rejects $$H_0$$ at the 5% level, and the K-S test does not.

set.seed(706)
x1 = rnorm(15, 100, 15)
x2 = rnorm(15, 120, 15)
t.test(x1,x2)$p.val [1] 0.04862413 ks.test(x1,x2) Two-sample Kolmogorov-Smirnov test data: x1 and x2 D = 0.4, p-value = 0.1844 alternative hypothesis: two-sided mean(x1); sd(x1) [1] 101.7776 [1] 20.05416 mean(x2); sd(x2) [1] 115.5439 [1] 16.2465  The K-S test compares sample ECDFs. The K-S test statistic $$D$$ is the maximum vertical distance between the two ECDFs. For significance in small samples, $$D$$ must be fairly large. [There is a shift in means, but the vertical distance betwee the the ECDFs is never great--relative to the jump sizes at each observation.] plot(ecdf(x1), col="blue", main="ECDFs of Two Samples") lines(ecdf(x2), col="orange")  Example 2: Use K-S test. If you're trying to distinguish between shapes of distributions that have similar means and variances, then use the K-S test in preference to the t test. In the example below, the K-S test rejects $$H_0$$ at the 5% level, and the t test does not. set.seed(2020) y1 = rnorm(85, 1, 1) y2 = rexp(85,1) summary(y1); sd(y1) Min. 1st Qu. Median Mean 3rd Qu. Max. -2.0388 0.4949 1.1188 1.1171 1.7206 4.2016 [1] 1.094069 summary(y2); sd(y2) Min. 1st Qu. Median Mean 3rd Qu. Max. 0.008152 0.282684 0.764139 1.049091 1.570979 4.116282 [1] 0.982902 t.test(y1,y2)$p.val
[1] 0.6702003

ks.test(y1,y2)

Two-sample Kolmogorov-Smirnov test

data:  y1 and y2
D = 0.21176, p-value = 0.04395
alternative hypothesis: two-sided

plot(ecdf(y1), col="blue", main="ECDFs of Two Samples")
lines(ecdf(y2), col="orange")