# How robust is the independent samples t-test when the distributions of the samples are non-normal?

I've read that the t-test is "reasonably robust" when the distributions of the samples depart from normality. Of course, it's the sampling distribution of the differences that are important. I have data for two groups. One of the groups is highly skewed on the dependent variable. The sample size is quite small for both groups (n=33 in one and 45 in the other). Should I assume that, under these conditions, my t-test will be robust to violations of the normality assumption?

• "Of course, it's the sampling distribution of the differences that are important" - Differences in what? I was tempted to edit this out of the question as I fear it's misleading to future readers (and tangential to the main point). My first thought was it's a mistaken reference to a paired t-test, where we assume the differences between pairs is normal, but that doesn't apply in an independent samples test. We don't even have pairs to difference! Perhaps "difference in the means" is intended? The rest of the Q considers normality of the two samples, not any differences. Dec 31, 2014 at 15:59
• The question of how robust the t-test is to such violations is an important and legitimate one. But a related issue is that checking for violations in your data first, and only then deciding whether to apply a t-test or some alternative test, is not recommended. Such a multi-step procedure has uncertain operating characteristics. See this thread: A principled method for choosing between t test or non-parametric e.g. Wilcoxon in small samples Dec 31, 2014 at 16:06
• What's a credible source? (I take it we'd both agree there's no such thing as an official source). Are we looking at level-robustness or also power? And if 'also power' ... what kind of alternative are we talking about? Jan 2, 2015 at 1:14
• @Glen_b Sorry, the "official sources" bounty message is clearly more for StackOverflow! I just feel this thread is practically important (plus quite high-traffic & poor on Wikipedia) to merit a few citations. The "canonical answer" bounty template would be inappropriate as Peter Flom's answer shows clearly. I get the feeling there's a "common body of knowledge" on this topic - if I'd been asked this Q off-hand, my list would look much like Dallal's (I'd have added kurtosis, but not ventured that equal sample size protects vs general non-normality) Jan 2, 2015 at 14:18
• @Glen_b Your answer mines a similar vein so it seems there are some basic points widely known/accepted. My degree covered assumptions but not consequences of violation: my knowledge is drawn from diverse sources, bits and bobs scattered about ("stats for psychologists" type books can pay more attention to consequences than many stats theory texts) - otherwise I'd have posted an answer not a bounty! If anyone knows a decent one-page summary in a good textbook, that'd do me fine. If it's a couple of papers with simulation results, that's fine too. Anything future readers could refer to and cite. Jan 2, 2015 at 14:21

Questions about robustness are very hard to answer well - because the assumptions may be violated in so many ways, and in each way to different degrees. Simulation work can only sample a very small portion of the possible violations.

Given the state of computing, I think it is often worth the time to run both a parametric and a non-parametric test, if both are available. You can then compare results.

If you are really ambitious, you could even do a permutation test.

What if Alan Turing had done his work before Ronald Fisher did his? :-).

• Peter, you've inspired me to write historical fiction to answer precisely that question!
– Sycorax
Dec 31, 2014 at 16:06

@PeterFlom hit the nail dead on with his first sentence.

I'll try to give a rough summary of what studies I have seen (if you want links it could be a while):

Overall, the two sample t-test is reasonably power-robust to symmetric non-normality (the true type-I-error-rate is affected somewhat by kurtosis, the power is impacted more by that).

When the two samples are mildly skew in the same direction, the one-tailed t-test is no longer unbiased. The t-statistic is skewed oppositely to the distribution, and has much more power if the test is in one direction than if it's in the other. If they're skew in opposite directions, the type I error rate can be heavily affected.

Heavy skewness can have bigger impacts, but generally speaking, moderate skewness with a two-tailed test isn't too bad if you don't mind your test in essence allocating more of its power to one direction that the other.

In short - the two-tailed, two-sample t-test is reasonably robust to those kinds of things if you can tolerate some impact on the significance level and some mild bias.

There are many, many, ways for distributions to be non-normal, though, which aren't covered by those comments.

• I am not sure it is correct to say it is reasonably power-robust! It is reasonable level-robust, the significance level will be roughly correct, but for example wilcoxon tests can have much higher power for alternatives reasonably close to normality to be difficult to detect. This also depends on factors such as if there is equal number of observations in each group: robustness is much more fragile in the unequal-n case! Oct 9, 2012 at 16:08
• @kjetilbhalvorsen The studies I've seen - including some simulation I've done myself (and I haven't looked at any for a good while; you may well have seen something I have not), the majority of the effect on power seemed to be mostly pushing the level up and down (which didn't affect the Wilcoxon). Given the generally good power properties of the Wilcoxon in these circumstances (particularly with heavy-tails), that's enough to have the Wilcoxon win on power - if you adjust the levels so they're similar, it surprised me how well the t- often did. Oct 9, 2012 at 20:32

@PeterFlom has already mentioned that simulation studies can never cover all scenarios and possibilities and therefore cannot lead to a definite answer. However, I still find it useful to actually explore an issue like this by conducting some simulations (this also happens to be exactly the type of exercise that I like to use when introducing the idea of Monte Carlo simulation studies to students). So, let's actually try this out. I'll use R for this.

The Code

n1 <- 33
n2 <- 45
mu1 <- 0
mu2 <- 0
sd1 <- 1
sd2 <- 1

iters <- 100000
p1 <- p2 <- p3 <- p4 <- p5 <- rep(NA, iters)

for (i in 1:iters) {

### normal distributions
x1 <- rnorm(n1, mu1, sd1)
x2 <- rnorm(n2, mu2, sd2)
p1[i] <- t.test(x1, x2)$p.value ### both variables skewed to the right x1 <- (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1 x2 <- (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2 p2[i] <- t.test(x1, x2)$p.value

### both variables skewed to the left
x1 <- -1 * (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1
x2 <- -1 * (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2
p3[i] <- t.test(x1, x2)$p.value ### first skewed to the left, second skewed to the right x1 <- -1 * (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1 x2 <- (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2 p4[i] <- t.test(x1, x2)$p.value

### first skewed to the right, second skewed to the left
x1 <- (rchisq(n1, df=1) - 1)/sqrt(2)      * sd1 + mu1
x2 <- -1 * (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2
p5[i] <- t.test(x1, x2)$p.value } print(round((apply(cbind(p1, p2, p3, p4, p5), 2, function(p) mean(p <= .05))), 3))  Explanation 1. First we set the group size (n1 and n2), the true group means (mu1 and mu2), and the true standard deviations (sd1 and sd2). 2. Then we define the number of iterations to run and set up vectors to store the p-values in. 3. Then I simulate data under 5 scenarios: 1. Both distributions are normal. 2. Both distributions are skewed to the right. 3. Both distributions are skewed to the left. 4. The first distribution is skewed to the left, the second to the right. 5. The first distribution is skewed to the right, the second to the left. Note that I am using chi-squared distributions for generating the skewed distributions. With one degree of freedom, those are heavily skewed distributions. Since the true mean and variance of a chi-squared distribution with one degree of freedom is equal to 1 and 2, respectively (see wikipedia), I rescale those distributions to first have mean 0 and standard deviation 1 and then rescale them to have the desired true mean and standard deviation (this could be done in one step, but doing it this way may be clearer). 4. In each case, I apply the t-test (Welch's version -- one could of course also consider Student's version that does assume equal variances in the two groups) and save the p-value to the vectors set up earlier. 5. Finally, once all iterations are complete, I compute for each vector how often the p-value is equal to or below .05 (i.e., the test is "significant"). This is the empirical rejection rate. Some Results 1. Simulating exactly as described above yields:  p1 p2 p3 p4 p5 0.049 0.048 0.047 0.070 0.070  So, when the skewness is in the same direction in both groups, the Type I error rate appears to be quite close to being well controlled (i.e., it is quite close to the nominal$\alpha = .05$). When the skewness is in opposite directions, there is some slight inflation in the Type I error rate. 2. If we change the code to mu1 <- .5, then we get:  p1 p2 p3 p4 p5 0.574 0.610 0.606 0.592 0.602  So, compared to the case where both distributions are normal (as assumed by the test), power actually appears to be slightly higher when the skewness is in the same direction! If you are surprised by this, you may want to rerun this a few times (of course, each time getting slightly different results), but the pattern will remain. Note that we have to be careful with interpreting the empirical power values under the two scenarios where the skewness is in opposite directions, since the Type I error rate is not quite nominal (as an extreme case, suppose I always reject regardless of what the data show; then I will always have a test with maximal power, but of course the test also has a rather inflated Type I error rate). One could start exploring a range of values for mu1 (and mu2 -- but what really matters is the difference between the two) and, more importantly, start changing the true standard deviations of the two groups (i.e., sd1 and sd2) and especially making them unequal. I also stuck to the sample sizes mentioned by the OP, but of course that could be adjusted as well. And skewness could of course take many other forms than what we see in a chi-squared distribution with one degree of freedom. I still think approaching things this way is useful, despite the fact that it cannot yield a definite answer. • Since we have an array of robust semi-parametric methods nowadays why is this discussion so worthwhile? Jan 6, 2015 at 13:27 • (+1) I think it might have been worth including the case where one sample was drawn from a skewed population and the other wasn't, as this was what the OP thought might be happening to their data. But it is nice to see an answer with explicit code. (A slight generalisation would actually allow a reader to investigate how well robust methods compare to the traditional t-test, which is a useful pedagogical exercise if you're trying to teach someone the dangers of applying a test whose assumptions have been violated...) Jan 6, 2015 at 16:30 In your situation, the t-test will likely be robust in terms of Type I error rate, but not Type II error rate. You would probably achieve more power through either a) a Kruskal-Wallis test, or b) a normalizing transformation prior to a t-test. I'm basing this conclusion on two Monte Carlo studies. In the first (Khan & Rayner, 2003), skew and kurtosis were indirectly manipulated via the parameters of the g-and-k distribution family, and the resulting power was examined. Importantly, the Kruskal- Wallis test's power was less damaged by non-normality, particularly for n>=15. A few caveats/qualifications about this study: Power was often hurt by high kurtosis, but it was less affected by skew. At first glance, this pattern might seem less relevant to your situation given that you noted a problem with skew, not kurtosis. However, I'm betting that excess kurtosis is also extreme in your case. Keep in mind that excess kurtosis will be at least as high as skew^2 - 2. (Let excess kurtosis equal the 4th standardized moment minus 3, so that excess kurtosis=0 for a normal distribution.) Note also that Khan and Rayner (2003) examined ANOVAs with 3 groups, but their results are likely to generalize to a two-sample t-test. A second relevant study (Beasley, Erikson, & Allison, 2009) examined both Type I and Type II errors with various non-normal distributions, such as a Chi-squared(1) and Weibull(1,.5). For sample sizes of at least 25, the t-test adequately controlled the Type I error rate at or below the nominal alpha level. However, power was highest with either a Kruskal-Wallis test or with a Rank-based Inverse Normal transformation (Blom scores) applied prior to the t-test. Beasley and colleagues generally argued against the normalizing approach, but it should be noted that the the normalizing approach controlled the Type I error rate for n>=25, and its power sometimes slightly exceeded that of the Kruskal-Wallis test. That is, the normalizing approach seems promising for your situation. See tables 1 and 4 in their article for details. References: Khan, A., & Rayner, G. D. (2003). Robustness to non-normality of common tests for the many-sample location problem. Journal of Applied Mathematics and Decision Sciences, 7, 187-206. Beasley, T. M., Erickson, S., & Allison, D. B. (2009). Rank-based inverse normal transformations are increasingly used, but are they merited? Behavioral Genetics, 39, 580-595. •$\text{(excess) kurtosis} \geq \text{skew}^2 -2$is true for a population; is it also true for estimates from a sample? Jan 8, 2015 at 2:19 • That seems like a question worthy of it's own thread. Perhaps your concern is that excess kurtosis will be downwardly biased in small samples? Of course, that was also the case in the simulation studies above, and kurtosis still caused low power in the t-test in those situations. Your question points to a more general limitation of most Monte Carlo studies: conclusions are often based on population characteristics, characteristics which the applied researcher cannot observe. It would be more useful to be able to predict relative power based on sample skew, kurtosis, etc. Jan 8, 2015 at 17:07 • I've posted a separate question about this issue: stats.stackexchange.com/questions/133247/… Jan 13, 2015 at 16:59 First of all, if you assume that the distribution of the two samples is different, make sure you are using Welch's version of the t-test which assumes unequal variances between the groups. This will at least attempt to account for some of the differences that occur because of the distribution. If we look at the formula for the Welch's t-test: $$t = {\overline{X}_1 - \overline{X}_2 \over s_{\overline{X}_1 - \overline{X}_2}}$$ where$s_{\overline{X}_1 - \overline{X}_2}\$ is

$$s_{\overline{X}_1 - \overline{X}_2} = \sqrt{{s_1^2 \over n_1} + {s_2^2 \over n_2}}$$

we can see that everytime there is an s we know the variance is being taken into account. Let's imagine that the two variances are in fact the same, but one is skewed, leading to a different variance estimate. If this estimate of the variance is not actually representative of your data because of skew, then the actually biasing effect will essentially be the square-root of that bias divided by the number of data points used to calculate it. Thus the effect of bad estimators of variance is muffled a bit by the square-root and a higher n, and that is probably why the consensus is that it remains a robust test.

The other issue of skewed distributions is that mean calculation will also be affected, and this is probably where the real problems of test assumption violations are since the means are relatively sensitive to skew. And the robustness of the test can be determined roughly by calculating the difference in means, compared to the difference in medians (as an idea). Perhaps you could even try replacing the difference in means by the difference in medians in the t-test as a more robust measure (I'm sure someone has discussed this but I couldn't find something on google quickly enough to link to).

I would also suggest running a permutation test if all you are doing is a t-test. The permutation test is an exact test, independent of distribution assumptions. Most importantly, the permutation tests and t-test will lead to identical results if the assumptions of the parametric test are met. Therefore, the robustness measure you seek can be 1 - the difference between the permutation and t-test p-values, where a score of 1 implies perfect robustness and 0 implies not robust at all.