Two-Group Hypothesis Testing

Question

Will unpaired t-test, definitely reject the null hypothesis if there is a non-zero difference in the population means for the two populations from which the samples are taken?

Answer 1

NO

(You would benefit from reading about hypothesis test power.)

set.seed(2024)
N <- 6
x <- rnorm(N, 0, 1) # N(0, 1)
y <- rnorm(N, 1, 1) # N(1, 1)
t.test(x, y, var.equal = T)

I get a p-value of $0.802$, well below any reasonable $\alpha$-level for rejection.

Because the sample size of six per group is so small, the test has minimal power to reject. If you iterate this many times, the distribution of p-values is only slightly skewed toward rejection.

library(ggplot2)
set.seed(2024)
N <- 6
R <- 1000
p <- rep(NA, R)
for (i in 1:R){
  x <- rnorm(N, 0, 1) # N(0, 1)
  y <- rnorm(N, 1, 1) # N(1, 1)
  p[i] <- t.test(x, y, var.equal = T)$p.value
}
d <- data.frame(
  p = p,
  CDF = ecdf(p)(p)
)
ggplot(d, aes(x = p, y = CDF)) +
  geom_point() +
  geom_abline(slope = 1, intercept = 0) # U(0, 1) distribution under H0

Running ecdf(p)(0.05), we see that only $36.9\%$ of simulations lead to rejection at $\alpha = 0.05$.

However, upping the sample size will lead to considerably more rejections.

library(ggplot2)
set.seed(2024)
N <- 6
R <- 1000
p <- rep(NA, R)
for (i in 1:R){
  x <- rnorm(N, 0, 1) # N(0, 1)
  y <- rnorm(N, 1, 1) # N(1, 1)
  p[i] <- t.test(x, y, var.equal = T)$p.value
}
d6 <- data.frame(
  p = p,
  CDF = ecdf(p)(p),
  N = "6"
)
N <- 16
R <- 1000
p <- rep(NA, R)
for (i in 1:R){
  x <- rnorm(N, 0, 1) # N(0, 1)
  y <- rnorm(N, 1, 1) # N(1, 1)
  p[i] <- t.test(x, y, var.equal = T)$p.value
}
d16 <- data.frame(
  p = p,
  CDF = ecdf(p)(p),
  N = "16"
)
d <- rbind(d6, d16)
ggplot(d, aes(x = p, y = CDF, col = N)) +
  geom_point() +
  geom_abline(slope = 1, intercept = 0) # U(0, 1) distribution under H0
ecdf(p)(0.05)

I get rejections in $81.7\%$ of the iterations and a distribution of p-values with stronger skew toward rejection.

This starts to get at a property of the t-test called consistency: the power to reject increasing toward perfection as the sample size increases (assuming the null hypothesis is false and the alternative is true, as is the case in these simulations).