# How can I take into account the different sample sizes of the two groups that I am comparing with a t-test or analysis of variance?

I am computing a p-value using the aov function in R in order to detect if the means of two groups are significantly different. However, these two groups that I am comparing have different sample sizes (for example the first one has 90 data points, while the second one only 15). I suppose that the p-value that is computed in this way is not really fair... is there a way to take into account this problem ? Is this under the name of the "effect size"?

I find on statistics websites that there is the so called "Cohen's d" or "Hedges' G" coefficients that could be calculated. If I understood well, if these coefficients are low, the effect due to this difference is low, otherwise it is high ?

I would like to check if I interpret in a good way what I found and, possibly, if there are other methods to take into account this problem (like coefficients that I can easily compute with R).

• The math in a function like that takes care of this.
– Dave
Mar 16, 2021 at 10:55
• Which function ? Do you mean aov ? @Dave Mar 16, 2021 at 10:57
• “aov”, “t.test” in R
– Dave
Mar 16, 2021 at 10:58
• @Dave How ? Is there like an argument that I have to specify to compute some value which quantifies this effect or do you mean that it is included in the p-value result and I do not have to worry about this problem ? Mar 16, 2021 at 11:06

This is easier to discuss for t tests (with only two samples) than with one-factor ANOVAs (typically with more than two samples).

Whether you're using the pooled or the Welch t test, look at the formulas for the denominator of the t statistic. There you see that a difference between sample sizes $$n_1 \ne n_2$$ is taken into account. [In the 'balanced case' where $$n_1 = n_2$$ the formulas for the pooled and Welch t statistics can be shown to be identical (even though the degrees of freedom may differ).]

Maybe a couple of simulations showing what happens to many tests for different scenarios will help convince you that the type one error probabilities are as promised (often fixed at 5%) and that type two error probabilities are reasonable. [With 100,000 iterations, error probabilities should be accurate to about two places.]

Pooled t test. equal sample sizes, null hypothesis true: Let $$\mu_1 = \mu_2 = 50,$$ $$\sigma_1 = \sigma_2 = 2,$$ $$n_1 = n_2 = 10.$$

set.seed(1234)
pv = replicate(10^5, t.test(rnorm(10,50,2),rnorm(10,50,2),
var.eq=T)$p.value) mean(pv <= .05)  0.04923 # aprx rejection rate 5%  Pooled t test. unequal sample sizes, null hypothesis true: Now $$n_1=8 \ne n_2 = 12.$$ set.seed(1235) pv = replicate(10^5, t.test(rnorm(8,50,2),rnorm(12,50,2), var.eq=T)$p.value)
mean(pv <= .05)
 0.05117    # aprx rejection rate 5%


Pooled t test. equal sample sizes, null hypothesis false, Now $$\mu_1 = 52, \mu_2 = 49.$$ Because the null hypothesis is false, we are now interested in the power of the pooled t test $$(1$$ minus probability of type two error).

 set.seed(1236)
pv = replicate(10^5, t.test(rnorm(10,52,2),rnorm(10,49,2),
var.eq=T)$p.value) mean(pv <= .05)  0.887 # aprx power 89%  Pooled t test. unequal sample sizes, null hypothesis false, Now $$n_1 = 8 \ne \nu = 12.$$ There are still twenty observations altogether, but the design is unbalanced. As expected, there is a slight decrease in power because the unbalanced design is less efficient, but the result is reasonable. set.seed(1236) pv = replicate(10^5, t.test(rnorm(8,52,2),rnorm(12,49,2), var.eq=T)$p.value)
mean(pv <= .05)
 0.87566    # aprx power 88%


In all four scenarios, the pooled variance in the denominator of the 'pooled t test' has made necessary changes in the t statistic. These changes ensure good behavior of the pooled t test when $$n_1 \ne n_2.$$

Note: However, the pooled t test can behave badly when population variances are unequal---especially so for unbalanced designs.]

For example, if $$H_0$$ is true, and variances differ, the what ought to be a test at level 5% can become a test at level 10% (or higher), when the larger variance is matched with the smaller sample size.

set.seed(1237)
pv = replicate(10^5, t.test(rnorm(8,50,3),rnorm(12,50,1),
var.eq=T)$p.value) mean(pv <= .05)  0.10743 # nowhere near intended 5% Type I error  But a Welch 2-sample t test corrects for unequal variances and gives satisfactory results. set.seed(1238) # WELCH pv = replicate(10^5, t.test(rnorm(8,50,3),rnorm(12,50,1))$p.value)
mean(pv <= .05)
 0.0516   # nearly 5% signif level

• Thank you @BruceET ! However, I did not get really much the meaning of the codes that you wrote , sorry :( so I do not know if you did it in the codes, but is it correct to say that the lower the p-value is, the higher is the difference between the two means of the two groups in t-test? Supposing that we have equal variances of the two groups and different size, since (as you explained very kindly) the different size of the groups is taken into account in the t.test , I would say yes... Mar 16, 2021 at 20:15