Testing the difference between two samples with very different sizes using an un-paired t-test

Question

I want to test wether population A is significantly different from population B (at 5% confidence interval). I need to do it for each variable using an un-paired t-test, since my data are normally distributed (tested with the Kolmogorov-Smirnov test, kstest function), and that it comes from different conditions. I am using the ttest2 MATLAB function.

Anyway, the t-test rejected the NULL-hypothesis of the mean populations being equal for all the variables. This is surprsing for me, as from the boxplot I expected the populations to be quite similar, except for variable 4 (see boxplot below). I red that the power of the t-test is higher when the samples are many, but decreases if the size of the two samples differ a lot. In my case, POP A has 2724 samples, and POP B has 504728.

In the light of such, one uestion arises: am I performing the right test? Should I say that the samples are significantly different for all the studied variables?

Answer 1

I don't think it's a problem of unequal sample sizes. It's more about the hypothesis testing nature of a t-test which is based on standard errors. For large samples they can become very narrow as they are in the denominator of the formula $SE=\frac{SD}{\sqrt{N}}$. Unless you are interested in identifying tiny mean differences, a test for effect sizes which are based on standard deviations and are therefore sample size-independent might be more suitable. I think this post would also be of help.