Long ago I learnt that normal distribution was necessary to use a two sample T-test. Today a colleague told me that she learnt that for N>50 normal distribution was not necessary. Is that true?

If true is that because of the central limit theorem?


Normality assumption of a t-test

Consider a large population from which you could take many different samples of a particular size. (In a particular study, you generally collect just one of these samples.)

The t-test assumes that the means of the different samples are normally distributed; it does not assume that the population is normally distributed.

By the central limit theorem, means of samples from a population with finite variance approach a normal distribution regardless of the distribution of the population. Rules of thumb say that the sample means are basically normally distributed as long as the sample size is at least 20 or 30. For a t-test to be valid on a sample of smaller size, the population distribution would have to be approximately normal.

The t-test is invalid for small samples from non-normal distributions, but it is valid for large samples from non-normal distributions.

Small samples from non-normal distributions

As Michael notes below, sample size needed for the distribution of means to approximate normality depends on the degree of non-normality of the population. For approximately normal distributions, you won't need as large sample as a very non-normal distribution.

Here are some simulations you can run in R to get a feel for this. First, here are a couple of population distributions.

curve(dnorm,xlim=c(-4,4)) #Normal
curve(dchisq(x,df=1),xlim=c(0,30)) #Chi-square with 1 degree of freedom

Next are some simulations of samples from the population distributions. In each of these lines, "10" is the sample size, "100" is the number of samples and the function after that specifies the population distribution. They produce histograms of the sample means.

hist(colMeans(sapply(rep(10,100),rnorm)),xlab='Sample mean',main='')
hist(colMeans(sapply(rep(10,100),rchisq,df=1)),xlab='Sample mean',main='')

For a t-test to be valid, these histograms should be normal.

qqp(colMeans(sapply(rep(10,100),rnorm)),xlab='Sample mean',main='')
qqp(colMeans(sapply(rep(10,100),rchisq,df=1)),xlab='Sample mean',main='')

Utility of a t-test

I have to note that all of the knowledge I just imparted is somewhat obsolete; now that we have computers, we can do better than t-tests. As Frank notes, you probably want to use Wilcoxon tests anywhere you were taught to run a t-test.

| cite | improve this answer | |
  • 8
    $\begingroup$ Good explanation (+1). I would add, however, that the sample size needed for the distribution of means to approximate normality depnds on the degree of non-normalness of the population. For large samples there is no reason to prefer a t-test over a permutations test that makes no assumptions about the distributions. $\endgroup$ – Michael Lew Apr 15 '11 at 2:01
  • 2
    $\begingroup$ +1 although, as far as I know, t-test is fairly resistent to moderate deviations from normality. Also, an interesting related discussion: stats.stackexchange.com/questions/2492/… $\endgroup$ – nico Apr 15 '11 at 7:02
  • 4
    $\begingroup$ good answer, although there is one small detail that you missed: the distribution of the data must have finite variance. T-test is hopeless for comparing difference in location of two Cauchy distributions (or student with 2 degrees of freedom), not because it is "non-robust", but because for these distributions there is additional relevant information in the sample beyond the means and standard deviations which the t-test throws away. $\endgroup$ – probabilityislogic Apr 24 '11 at 0:55
  • 2
    $\begingroup$ In addition to this, t-test also naturally yield confidence intervals for the parameter being investigated. (still upvote because of the two first paragraphs which adress the question directly, I just disagree strongly with the third) $\endgroup$ – Erik Mar 7 '14 at 10:58
  • 9
    $\begingroup$ t-test DOES require normality of the population. That's an assumption needed for the t statistic to have a t-Student distribution. If you don't have a normal population, you can't express the t statistic as a standard normal variable divided by the root of a Chi-squared variable divided by its degrees of freedom. Maybe what you are trying to say is that if some conditions are true, like not too much skewness, or a big sample, the test can still be valid even when the population is not normal. $\endgroup$ – toneloy Nov 24 '16 at 15:22

The central limit theorem is less useful than one might think in this context. First, as someone pointed out already, one does not know if the current sample size is "large enough". Secondly, the CLT is more about achieving the desired type I error than about type II error. In other words, the t-test can be uncompetitive power-wise. That's why the Wilcoxon test is so popular. If normality holds, it is 95% as efficient as the t-test. If normality does not hold it can be arbitrarily more efficient than the t-test.

| cite | improve this answer | |
  • 7
    $\begingroup$ (+1) Welcome to the site, which I'm glad you've found. I look forward to your participation here. $\endgroup$ – cardinal Apr 20 '11 at 14:52
  • 5
    $\begingroup$ (+1) Good point about the Wilcoxon. $\endgroup$ – whuber Apr 20 '11 at 14:56

See my previous answer to a question on the robustness of the t-test.

In particular, I recommend playing around with the onlinestatsbook applet.

The image below is based on the following scenario:

  • null hypothesis is true
  • fairly severe skewness
  • same distribution in both groups
  • same variance in both groups
  • sample size per group 5 (i.e., much less than 50 as per your question)
  • I pressed the 10,000 simulations button about 100 times to get up to over one million simulations.

The simulation obtained suggests that instead of getting a 5% Type I errors, I was only getting 4.5% Type I errors.

Whether you consider this robust depends on your perspective.

enter image description here

| cite | improve this answer | |
  • 4
    $\begingroup$ +1 Good points. The power of the t-test with skewed alternatives, though, can degrade severely (to the point where it is essentially zero even for huge effect sizes). $\endgroup$ – whuber Apr 15 '11 at 14:20

In my experience with just the one-sample t-test, I have found that the skew of the distributions is more important than the kurtosis, say. For non-skewed but fat-tailed distributions (a t with 5 degrees of freedom, a Tukey h-distribution with $h=0.24999$, etc), I have found that 40 samples has always been sufficient to get an empirical type I rate near the nominal. When the distribution is very skewed, however, you may need many many more samples.

For example, suppose you were playing the lottery. With probability $p = 10^{-4}$ you will win 100 thousand dollars, and with probability $1-p$ you will lose one dollar. If you perform a t-test for the null that the mean return is zero based on a sample of one thousand draws of this process, I don't think you are going to achieve the nominal type I rate.

edit: duh, per @whuber's catch in the comment, the example I gave did not have mean zero, so testing for mean zero has nothing to do with the type I rate.

Because the lottery example often has a sample standard deviation of zero, the t-test chokes. So instead, I give a code example using Goerg's Lambert W x Gaussian distribution. The distribution I use here has a skew of around 1355.

#hey look! I'm learning R!

Gauss_input = create_LambertW_input("normal", beta=c(0,1))
params = list(delta = c(0), gamma = c(2), alpha = 1)
LW.Gauss = create_LambertW_output(input = Gauss_input, theta = params)
#get the moments of this distribution
moms <- mLambertW(beta=c(0,1),distname=c("normal"),delta = 0,gamma = 2, alpha = 1)

test_ttest <- function(sampsize) {
    samp <- LW.Gauss$rY(params)(n=sampsize)
    tval <- t.test(samp, mu = moms$mean)

#to replicate randomness

pvals <- replicate(1024,test_ttest(50))
#how many rejects at the 0.05 level?
print(sum(pvals < 0.05) / length(pvals))

pvals <- replicate(1024,test_ttest(250))
#how many rejects at the 0.05 level?
print(sum(pvals < 0.05) / length(pvals))

p    vals <- replicate(1024,test_ttest(1000))
#how many rejects at the 0.05 level?
print(sum(pvals < 0.05) / length(pvals))

pvals <- replicate(1024,test_ttest(2000))
#how many rejects at the 0.05 level?
print(sum(pvals < 0.05) / length(pvals))

This code gives the empirical reject rate at the nominal 0.05 level for different sample sizes. For sample of size 50, the empirical rate is 0.40 (!); for sample size 250, 0.29; for sample size 1000, 0.21; for sample size 2000, 0.18. Clearly the one-sample t-test suffers from skew.

| cite | improve this answer | |
  • $\begingroup$ In the example you're discussing the power of the test, not its size. The null, by the way, appears to be $p=0$, for which the distribution is degenerate (an atom at a single point): that's about as far from normality as one can be! $\endgroup$ – whuber Apr 15 '11 at 14:22

The central limit theorem establishes (under the required conditions) that the numerator of the t-statistic is asymptotically normal. The t-statistic also has a denominator. To have a t-distribution you'd need the denominator to be independent and square-root-of-a-chi-square-on-its-df.

And we know it won't be independent (that characterizes the normal!)

Slutsky's theorem combined with the CLT would give you that the t-statistic is asymptotically normal (but not necessarily at a very useful rate).

What theorem would establish that the t-statistic is approximately t-distributed when there's non-normality, and how fast it comes in? (Of course, eventually the t- will go close to the normal as well, but we're assuming that the approximation to another approximation will be better than just using the normal approximation ... )

However, more important than the purported level-robustness of the $t$ (at larger sample sizes) is the effect on its power. Note that the asymptotic relative efficiency of the t-test relative to the Wilcoxon-Mann-Whitney (for example) may be 0 (i.e. as sample sizes become large while effect sizes you want to pick up shrink, you may require vastly larger samples for the t to have the same power as an obvious alternative).

So while the t-test may end up having a nice normal-looking null distribution in many cases if $n$ is large enough, its performance under the null isn't really what people care most about -- it's performance under the alternative -- and there it may not be so great, if you care about rejecting the null in the cases where the effect is not so easy to pick up.

| cite | improve this answer | |
  • 3
    $\begingroup$ While we may not know if the sample mean and variance are independent, we always know that they are uncorrelated. This is because the sample mean is a function of bivariate sums $x_{i}+x_{j}$, and sample variance is a function of bivariate differences $x_{i}-x_{j}$ (these are called "U statistics") and we have $cov(x_{i}+x_{j},x_{i}-x_{j})=var(x_{i})-var(x_{j})+cov(x_{i},x_{j})-cov(x_{j},x_{i})=0$ as long as the distribution is "homogenous" $var(x_{i})=var(x_{j})$, which is part of the problem statement. $\endgroup$ – probabilityislogic Apr 24 '11 at 1:08
  • 1
    $\begingroup$ Unfortunately, the distinction between uncorrelated and independent is relevant if we are to be ending up with a t-distribution. $\endgroup$ – Glen_b May 9 '14 at 22:21

Yes, the Central Limit Theorem tells us this is true. So long as you avoid extremely heavy-tailed traits, non-Normality presents no problems in moderate-to-large samples.

Here's a helpful review paper;


The Wilcoxon test (mentioned by others) can have terrible power when the alternative is not a location shift of the original distribution. Furthermore, the way it measures differences between distributions is not transitive.

| cite | improve this answer | |
  • $\begingroup$ Interesting points about the Wilcoxon. However, the t-test has similar difficulties: it's especially bad at detecting shifts that are accompanied by increased variance. The bit about transitivity appears to be mainly a curiosity in the present context; it's difficult to see how it's relevant to the original hypothesis test or its interpretation. (But maybe intransitivity could become important in an ANOVA or multiple comparisons setting.) $\endgroup$ – whuber Nov 22 '11 at 20:43
  • $\begingroup$ The unequal variance t-test (which is the default in some software) doesn't have the problem with heteroskedasticity. $\endgroup$ – guest Nov 22 '11 at 22:02
  • $\begingroup$ Regarding transitivity; reporting the sample means, or differences in means (which is natural using a t-test approach) gives the reader something they can consider when sampling from other populations. The non-transitivity of the Wilcoxon test means that this approach has no such analog; using ranks of data is a very limited approach. $\endgroup$ – guest Nov 22 '11 at 22:13
  • 1
    $\begingroup$ (1) The Satterthwaite-Welch (unequal variance) test doesn't overcome the power loss I referred to (although it can help a little). (2) I think you're being extreme in characterizing using ranks as "limited." In his reply, @Frank Harrell was referring to studies showing how the Wilcoxon test maintains high efficiency in many settings: this demonstrates how using the ranks is both effective and more flexible, not more limited, compared to t tests. $\endgroup$ – whuber Nov 22 '11 at 22:36
  • $\begingroup$ (1) No, but it gives the right Type I error rate, in moderate-to-large samples (2) Thanks, but I respectfully disagree. Using t-tests over Wilcoxon makes it much easier to bridge the gap between testing and using confidence intervals. If one only wants to do testing, and never looks beyond the two groups in a study, Wilcoxon of course has situations where it works well. But often we don't want to do just testing, and want to help users generalize the results to other situations; the Wilcoxon test is then not helpful. $\endgroup$ – guest Nov 22 '11 at 23:39

About the use of Wilcoxon-Mann-Whitney test as an alternative I recommend the paper The Wilcoxon-Man-Whitney test under scrutiny

As a test of means or medians, the Wilcoxon–Mann–Whitney (WMW) test can be severely nonrobust for deviations from the pure shift model.

These are the recommendation of the authors of the paper:

The rank transformation can alter means, standard deviations, and skewnesses of the two samples differently. The only situation in which the rank transformation is guaranteed to achieve a beneficial effect is when distributions are identical and sample sizes are equal. For deviations from these rather strict assumptions, the effects of the rank transformation on sample moments are unpredictable. In the simulation study of the paper, the WMW test was compared with the Fligner–Policello test (FP), the Brunner–Munzel test (BM), the two-sample T test (T ), the Welch U test (U), and the Welch U test on ranks (RU). The four rank-based tests (WMW, FP, BM, and RU) performed similarly, although the BM test was frequently a little better than the others. When the sample sizes were equal, the parametric tests (T and U) were superior to the rank-based tests under the null hypothesis of equal means, but not under the null hypothesis of equal medians. When the sample sizes were unequal, the BM, RU, and U tests performed best. For several settings, small changes in population properties led to large alterations in the performance of the tests. In summary, the large sample approximate WMW test can be a poor method for comparing the means or medians of two populations, unless the two distributions have equal shapes and equal scales. This problem also seems to apply in various degrees to the exact WMW test, the FP test, the BM test, and the Welch U test on ranks. When using the WMW test, the authors recommend that the properties of the ranked samples are thoroughly investigated for signs of skewness and variance heterogeneity. If such effects are present, the Welch U test is preferable in many situations.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.