This is easier to discuss for t tests (with only two samples) than
with one-factor ANOVAs (typically with more than two samples).
Whether you're using the pooled or the Welch t test, look at the formulas for the denominator of the t statistic. There you see that a difference between
sample sizes $n_1 \ne n_2$ is taken into account. [In the 'balanced case' where $n_1 = n_2$ the formulas for the pooled and Welch t statistics can be shown to be identical (even though the degrees of freedom may differ).]
Maybe a couple of simulations showing what happens to many tests for different scenarios will help convince you that the type one error probabilities are as promised (often fixed at 5%) and that type two error probabilities are reasonable. [With 100,000 iterations, error probabilities should
be accurate to about two places.]
Pooled t test. equal sample sizes, null hypothesis true:
Let $\mu_1 = \mu_2 = 50,$ $\sigma_1 = \sigma_2 = 2,$ $n_1 = n_2 = 10.$
set.seed(1234)
pv = replicate(10^5, t.test(rnorm(10,50,2),rnorm(10,50,2),
var.eq=T)$p.value)
mean(pv <= .05)
[1] 0.04923 # aprx rejection rate 5%
Pooled t test. unequal sample sizes, null hypothesis true:
Now $n_1=8 \ne n_2 = 12.$
set.seed(1235)
pv = replicate(10^5, t.test(rnorm(8,50,2),rnorm(12,50,2),
var.eq=T)$p.value)
mean(pv <= .05)
[1] 0.05117 # aprx rejection rate 5%
Pooled t test. equal sample sizes, null hypothesis false,
Now $\mu_1 = 52, \mu_2 = 49.$ Because the null hypothesis is false, we are now interested in the power of the pooled t test $(1$ minus probability of type two error).
set.seed(1236)
pv = replicate(10^5, t.test(rnorm(10,52,2),rnorm(10,49,2),
var.eq=T)$p.value)
mean(pv <= .05)
[1] 0.887 # aprx power 89%
Pooled t test. unequal sample sizes, null hypothesis false,
Now $n_1 = 8 \ne \nu = 12.$ There are still twenty observations altogether, but the design is unbalanced. As expected, there is a slight decrease in power
because the unbalanced design is less efficient, but the
result is reasonable.
set.seed(1236)
pv = replicate(10^5, t.test(rnorm(8,52,2),rnorm(12,49,2),
var.eq=T)$p.value)
mean(pv <= .05)
[1] 0.87566 # aprx power 88%
In all four scenarios, the pooled variance in the denominator of
the 'pooled t test' has made necessary changes in the t statistic.
These changes ensure good behavior of the pooled t test when $n_1 \ne n_2.$
Note: However, the pooled t test can behave badly when population variances are unequal---especially so for unbalanced designs.]
For example, if $H_0$ is true, and variances differ, the what
ought to be a test at level 5% can become a test at level 10% (or higher),
when the larger variance is matched with the smaller sample size.
set.seed(1237)
pv = replicate(10^5, t.test(rnorm(8,50,3),rnorm(12,50,1),
var.eq=T)$p.value)
mean(pv <= .05)
[1] 0.10743 # nowhere near intended 5% Type I error
But a Welch 2-sample t test corrects for unequal variances and gives
satisfactory results.
set.seed(1238) # WELCH
pv = replicate(10^5, t.test(rnorm(8,50,3),rnorm(12,50,1))$p.value)
mean(pv <= .05)
[1] 0.0516 # nearly 5% signif level
