How robust is the independent samples t-test when the distributions of the samples are non-normal?

Question

I've read that the t-test is "reasonably robust" when the distributions of the samples depart from normality. Of course, it's the sampling distribution of the differences that are important. I have data for two groups. One of the groups is highly skewed on the dependent variable. The sample size is quite small for both groups (n=33 in one and 45 in the other). Should I assume that, under these conditions, my t-test will be robust to violations of the normality assumption?

Answer 1

Questions about robustness are very hard to answer well - because the assumptions may be violated in so many ways, and in each way to different degrees. Simulation work can only sample a very small portion of the possible violations.

Given the state of computing, I think it is often worth the time to run both a parametric and a non-parametric test, if both are available. You can then compare results.

If you are really ambitious, you could even do a permutation test.

What if Alan Turing had done his work before Ronald Fisher did his? :-).

Answer 2

@PeterFlom hit the nail dead on with his first sentence.

I'll try to give a rough summary of what studies I have seen (if you want links it could be a while):

Overall, the two sample t-test is reasonably power-robust to symmetric non-normality (the true type-I-error-rate is affected somewhat by kurtosis, the power is impacted more by that).

When the two samples are mildly skew in the same direction, the one-tailed t-test is no longer unbiased. The t-statistic is skewed oppositely to the distribution, and has much more power if the test is in one direction than if it's in the other. If they're skew in opposite directions, the type I error rate can be heavily affected.

Heavy skewness can have bigger impacts, but generally speaking, moderate skewness with a two-tailed test isn't too bad if you don't mind your test in essence allocating more of its power to one direction that the other.

In short - the two-tailed, two-sample t-test is reasonably robust to those kinds of things if you can tolerate some impact on the significance level (typically downward, lowering power) and in some situations potentially some mild bias.

There are many, many, ways for distributions to be non-normal, though, which aren't covered by those comments.

Answer 3

@PeterFlom has already mentioned that simulation studies can never cover all scenarios and possibilities and therefore cannot lead to a definite answer. However, I still find it useful to actually explore an issue like this by conducting some simulations (this also happens to be exactly the type of exercise that I like to use when introducing the idea of Monte Carlo simulation studies to students). So, let's actually try this out. I'll use R for this.

The Code

n1 <- 33
n2 <- 45
mu1 <- 0
mu2 <- 0
sd1 <- 1
sd2 <- 1

iters <- 100000
p1 <- p2 <- p3 <- p4 <- p5 <- rep(NA, iters)

for (i in 1:iters) {

   ### normal distributions
   x1 <- rnorm(n1, mu1, sd1)
   x2 <- rnorm(n2, mu2, sd2)
   p1[i] <- t.test(x1, x2)$p.value

   ### both variables skewed to the right
   x1 <- (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1
   x2 <- (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2
   p2[i] <- t.test(x1, x2)$p.value

   ### both variables skewed to the left
   x1 <- -1 * (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1
   x2 <- -1 * (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2
   p3[i] <- t.test(x1, x2)$p.value

   ### first skewed to the left, second skewed to the right
   x1 <- -1 * (rchisq(n1, df=1) - 1)/sqrt(2) * sd1 + mu1
   x2 <- (rchisq(n2, df=1) - 1)/sqrt(2)      * sd2 + mu2
   p4[i] <- t.test(x1, x2)$p.value

   ### first skewed to the right, second skewed to the left
   x1 <- (rchisq(n1, df=1) - 1)/sqrt(2)      * sd1 + mu1
   x2 <- -1 * (rchisq(n2, df=1) - 1)/sqrt(2) * sd2 + mu2
   p5[i] <- t.test(x1, x2)$p.value

}

print(round((apply(cbind(p1, p2, p3, p4, p5), 2, function(p) mean(p <= .05))), 3))

Explanation

1. First we set the group size (n1 and n2), the true group means (mu1 and mu2), and the true standard deviations (sd1 and sd2).

2. Then we define the number of iterations to run and set up vectors to store the p-values in.

3. Then I simulate data under 5 scenarios:

   1. Both distributions are normal.
   2. Both distributions are skewed to the right.
   3. Both distributions are skewed to the left.
   4. The first distribution is skewed to the left, the second to the right.
   5. The first distribution is skewed to the right, the second to the left.

   Note that I am using chi-squared distributions for generating the skewed distributions. With one degree of freedom, those are heavily skewed distributions. Since the true mean and variance of a chi-squared distribution with one degree of freedom is equal to 1 and 2, respectively (see wikipedia), I rescale those distributions to first have mean 0 and standard deviation 1 and then rescale them to have the desired true mean and standard deviation (this could be done in one step, but doing it this way may be clearer).

4. In each case, I apply the t-test (Welch's version -- one could of course also consider Student's version that does assume equal variances in the two groups) and save the p-value to the vectors set up earlier.

5. Finally, once all iterations are complete, I compute for each vector how often the p-value is equal to or below .05 (i.e., the test is "significant"). This is the empirical rejection rate.

Some Results

1. Simulating exactly as described above yields:

      p1    p2    p3    p4    p5 
   0.049 0.048 0.047 0.070 0.070
   

   So, when the skewness is in the same direction in both groups, the Type I error rate appears to be quite close to being well controlled (i.e., it is quite close to the nominal $\alpha = .05$). When the skewness is in opposite directions, there is some slight inflation in the Type I error rate.

2. If we change the code to mu1 <- .5, then we get:

      p1    p2    p3    p4    p5 
   0.574 0.610 0.606 0.592 0.602
   

   So, compared to the case where both distributions are normal (as assumed by the test), power actually appears to be slightly higher when the skewness is in the same direction! If you are surprised by this, you may want to rerun this a few times (of course, each time getting slightly different results), but the pattern will remain.

   Note that we have to be careful with interpreting the empirical power values under the two scenarios where the skewness is in opposite directions, since the Type I error rate is not quite nominal (as an extreme case, suppose I always reject regardless of what the data show; then I will always have a test with maximal power, but of course the test also has a rather inflated Type I error rate).

One could start exploring a range of values for mu1 (and mu2 -- but what really matters is the difference between the two) and, more importantly, start changing the true standard deviations of the two groups (i.e., sd1 and sd2) and especially making them unequal. I also stuck to the sample sizes mentioned by the OP, but of course that could be adjusted as well. And skewness could of course take many other forms than what we see in a chi-squared distribution with one degree of freedom. I still think approaching things this way is useful, despite the fact that it cannot yield a definite answer.

Answer 4

In your situation, the t-test will likely be robust in terms of Type I error rate, but not Type II error rate. You would probably achieve more power through either a) a Kruskal-Wallis test, or b) a normalizing transformation prior to a t-test.

I'm basing this conclusion on two Monte Carlo studies. In the first (Khan & Rayner, 2003), skew and kurtosis were indirectly manipulated via the parameters of the g-and-k distribution family, and the resulting power was examined. Importantly, the Kruskal- Wallis test's power was less damaged by non-normality, particularly for n>=15.

A few caveats/qualifications about this study: Power was often hurt by high kurtosis, but it was less affected by skew. At first glance, this pattern might seem less relevant to your situation given that you noted a problem with skew, not kurtosis. However, I'm betting that excess kurtosis is also extreme in your case. Keep in mind that excess kurtosis will be at least as high as skew^2 - 2. (Let excess kurtosis equal the 4th standardized moment minus 3, so that excess kurtosis=0 for a normal distribution.) Note also that Khan and Rayner (2003) examined ANOVAs with 3 groups, but their results are likely to generalize to a two-sample t-test.

A second relevant study (Beasley, Erikson, & Allison, 2009) examined both Type I and Type II errors with various non-normal distributions, such as a Chi-squared(1) and Weibull(1,.5). For sample sizes of at least 25, the t-test adequately controlled the Type I error rate at or below the nominal alpha level. However, power was highest with either a Kruskal-Wallis test or with a Rank-based Inverse Normal transformation (Blom scores) applied prior to the t-test. Beasley and colleagues generally argued against the normalizing approach, but it should be noted that the the normalizing approach controlled the Type I error rate for n>=25, and its power sometimes slightly exceeded that of the Kruskal-Wallis test. That is, the normalizing approach seems promising for your situation. See tables 1 and 4 in their article for details.

References:

Khan, A., & Rayner, G. D. (2003). Robustness to non-normality of common tests for the many-sample location problem. Journal of Applied Mathematics and Decision Sciences, 7, 187-206.

Beasley, T. M., Erickson, S., & Allison, D. B. (2009). Rank-based inverse normal transformations are increasingly used, but are they merited? Behavioral Genetics, 39, 580-595.

Answer 5

First of all, if you assume that the distribution of the two samples is different, make sure you are using Welch's version of the t-test which assumes unequal variances between the groups. This will at least attempt to account for some of the differences that occur because of the distribution.

If we look at the formula for the Welch's t-test:

$$ t = {\overline{X}_1 - \overline{X}_2 \over s_{\overline{X}_1 - \overline{X}_2}} $$

where $s_{\overline{X}_1 - \overline{X}_2}$ is

$$ s_{\overline{X}_1 - \overline{X}_2} = \sqrt{{s_1^2 \over n_1} + {s_2^2 \over n_2}} $$

we can see that everytime there is an s we know the variance is being taken into account. Let's imagine that the two variances are in fact the same, but one is skewed, leading to a different variance estimate. If this estimate of the variance is not actually representative of your data because of skew, then the actually biasing effect will essentially be the square-root of that bias divided by the number of data points used to calculate it. Thus the effect of bad estimators of variance is muffled a bit by the square-root and a higher n, and that is probably why the consensus is that it remains a robust test.

The other issue of skewed distributions is that mean calculation will also be affected, and this is probably where the real problems of test assumption violations are since the means are relatively sensitive to skew. And the robustness of the test can be determined roughly by calculating the difference in means, compared to the difference in medians (as an idea). Perhaps you could even try replacing the difference in means by the difference in medians in the t-test as a more robust measure (I'm sure someone has discussed this but I couldn't find something on google quickly enough to link to).

I would also suggest running a permutation test if all you are doing is a t-test. The permutation test is an exact test, independent of distribution assumptions. Most importantly, the permutation tests and t-test will lead to identical results if the assumptions of the parametric test are met. Therefore, the robustness measure you seek can be 1 - the difference between the permutation and t-test p-values, where a score of 1 implies perfect robustness and 0 implies not robust at all.