0
$\begingroup$

Suppose I have two groups of data. I want to know if their distributions and / or means are different or not. Is it appropriate to do a two-sample t-test? Or should I go for a 2 sample goodness of fit test like 2D-KS test?

$\endgroup$
0
$\begingroup$

The basic principle is to use each test for what it was designed to do. Here are specific examples:

Example 1: Use t test. If you're trying to see whether two normal samples have the same mean, then use a t test in preference to a K-S test. This is especially true if samples are small so that the K-S test has relatively poor power. In the example below, the t test rejects $H_0$ at the 5% level, and the K-S test does not.

set.seed(706)
x1 = rnorm(15, 100, 15)
x2 = rnorm(15, 120, 15)
t.test(x1,x2)$p.val
[1] 0.04862413

ks.test(x1,x2)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.4, p-value = 0.1844
alternative hypothesis: two-sided

mean(x1); sd(x1)
[1] 101.7776
[1] 20.05416
mean(x2); sd(x2)
[1] 115.5439
[1] 16.2465

The K-S test compares sample ECDFs. The K-S test statistic $D$ is the maximum vertical distance between the two ECDFs. For significance in small samples, $D$ must be fairly large. [There is a shift in means, but the vertical distance betwee the the ECDFs is never great--relative to the jump sizes at each observation.]

plot(ecdf(x1), col="blue", main="ECDFs of Two Samples")
  lines(ecdf(x2), col="orange")

enter image description here

Example 2: Use K-S test. If you're trying to distinguish between shapes of distributions that have similar means and variances, then use the K-S test in preference to the t test. In the example below, the K-S test rejects $H_0$ at the 5% level, and the t test does not.

set.seed(2020)
y1 = rnorm(85, 1, 1)  
y2 = rexp(85,1)

summary(y1);  sd(y1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-2.0388  0.4949  1.1188  1.1171  1.7206  4.2016 
[1] 1.094069

summary(y2);  sd(y2)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.008152 0.282684 0.764139 1.049091 1.570979 4.116282 
[1] 0.982902

t.test(y1,y2)$p.val
[1] 0.6702003

ks.test(y1,y2)

    Two-sample Kolmogorov-Smirnov test

data:  y1 and y2
D = 0.21176, p-value = 0.04395
alternative hypothesis: two-sided

plot(ecdf(y1), col="blue", main="ECDFs of Two Samples")
  lines(ecdf(y2), col="orange")

enter image description here

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.