1
$\begingroup$

I want to test wether population A is significantly different from population B (at 5% confidence interval). I need to do it for each variable using an un-paired t-test, since my data are normally distributed (tested with the Kolmogorov-Smirnov test, kstest function), and that it comes from different conditions. I am using the ttest2 MATLAB function.

Anyway, the t-test rejected the NULL-hypothesis of the mean populations being equal for all the variables. This is surprsing for me, as from the boxplot I expected the populations to be quite similar, except for variable 4 (see boxplot below). I red that the power of the t-test is higher when the samples are many, but decreases if the size of the two samples differ a lot. In my case, POP A has 2724 samples, and POP B has 504728.

In the light of such, one uestion arises: am I performing the right test? Should I say that the samples are significantly different for all the studied variables?

boxplot of my data

$\endgroup$
1
$\begingroup$

I don't think it's a problem of unequal sample sizes. It's more about the hypothesis testing nature of a t-test which is based on standard errors. For large samples they can become very narrow as they are in the denominator of the formula $SE=\frac{SD}{\sqrt{N}}$. Unless you are interested in identifying tiny mean differences, a test for effect sizes which are based on standard deviations and are therefore sample size-independent might be more suitable. I think this post would also be of help.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the fast and useful response. Would you suggest any reference to perform Effect Size analysis with MATLAB, and maybe suggest one test I could perform on my data in the light of what I wrote? $\endgroup$ – umbe1987 Jul 14 '15 at 12:47
  • 1
    $\begingroup$ For independent samples you could compute the Cohen's D effect sizes. A detailed way of interpreting them is presented here. I am not a MATLAB user, but I found this which I think is what you need. According to the comments Hedges' g is one manifestation of Cohen's D. $\endgroup$ – StevenP Jul 14 '15 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.