# How robust is the Independent Samples t Test when the distributions of the samples are non-normal?

I've read that the t test is "reasonably robust" when the distributions of the samples depart from normality. Of course, it's the sampling distribution of the differences that are important. I have data for two groups. One of the groups is highly skewed on the dependent variable. The sample size is quite small for both groups (n=33 in one and 45 in the other). Should I assume that, under these conditions, my t test will be robust to violations of the normality assumption?

-

Questions about robustness are very hard to answer well - because the assumptions may be violated in so many ways, and in each way to different degrees. Simulation work can only sample a very small portion of the possible violations.

Given the state of computing, I think it is often worth the time to run both a parametric and a non-parametric test, if both are available. You can then compare results.

If you are really ambitious, you could even do a permutation test.

What if Alan Turing had done his work before Ronald Fisher did his? :-).

-

@PeterFlom hit the nail dead on with his first sentence.

I'll try to give a rough summary of what studies I have seen (if you want links it could be a while):

Overall, the two sample t-test is reasonably power-robust to symmetric non-normality (the true type-I-error-rate is affected somewhat by kurtosis, the power is impacted mostly by that).

When the two samples are mildly skew in the same direction, the one-tailed t-test is no longer unbiased. The t-statistic is skewed oppositely to the distribution, and has much more power if the test is in one direction than if it's in the other. If they're skew in opposite directions, the type I error rate can be heavily affected.

Heavy skewness can have bigger impacts, but generally speaking, moderate skewness with a two-tailed test isn't too bad if you don't mind your test in essence allocating more of its power to one direction that the other.

In short - the two-tailed, two-sample t-test is reasonably robust to those kinds of things if you can tolerate some impact on the significance level and some mild bias.

There are many, many, ways for distributions to be non-normal, though, which aren't covered by those comments.

-
I am not sure it is correct to say it is reasonably power-robust! It is reasonable level-robust, the significance level will be roughly correct, but for example wilcoxon tests can have much higher power for alternatives reasonably close to normality to be difficult to detect. This also depends on factors such as if there is equal number of observations in each group: robustness is much more fragile in the unequal-n case! –  kjetil b halvorsen Oct 9 '12 at 16:08
@kjetilbhalvorsen The studies I've seen - including some simulation I've done myself (and I haven't looked at any for a good while; you may well have seen something I have not), the majority of the effect on power seemed to be mostly pushing the level up and down (which didn't affect the Wilcoxon). Given the generally good power properties of the Wilcoxon in these circumstances (particularly with heavy-tails), that's enough to have the Wilcoxon win on power - if you adjust the levels so they're similar, it surprised me how well the t- often did. –  Glen_b Oct 9 '12 at 20:32