# T-test for non normal when N>50?

Long ago I learnt that normal distribution was necessary to use a two sample T-test. Today a colleague told me that she learnt that for N>50 normal distribution was not necessary. Is that true?

If true is that because of the central limit theorem?

Normality assumption of a t-test

Consider a large population from which you could take many different samples of a particular size. (In a particular study, you generally collect just one of these samples.)

The t-test assumes that the means of the different samples are normally distributed; it does not assume that the population is normally distributed.

By the central limit theorem, means of samples from a population with finite variance approach a normal distribution regardless of the distribution of the population. Rules of thumb say that the sample means are basically normally distributed as long as the sample size is at least 20 or 30. For a t-test to be valid on a sample of smaller size, the population distribution would have to be approximately normal.

The t-test is invalid for small samples from non-normal distributions, but it is valid for large samples from non-normal distributions.

Small samples from non-normal distributions

As Michael notes below, sample size needed for the distribution of means to approximate normality depends on the degree of non-normality of the population. For approximately normal distributions, you won't need as large sample as a very non-normal distribution.

Here are some simulations you can run in R to get a feel for this. First, here are a couple of population distributions.

curve(dnorm,xlim=c(-4,4)) #Normal
curve(dchisq(x,df=1),xlim=c(0,30)) #Chi-square with 1 degree of freedom


Next are some simulations of samples from the population distributions. In each of these lines, "10" is the sample size, "100" is the number of samples and the function after that specifies the population distribution. They produce histograms of the sample means.

hist(colMeans(sapply(rep(10,100),rnorm)),xlab='Sample mean',main='')
hist(colMeans(sapply(rep(10,100),rchisq,df=1)),xlab='Sample mean',main='')


For a t-test to be valid, these histograms should be normal.

require(car)
qqp(colMeans(sapply(rep(10,100),rnorm)),xlab='Sample mean',main='')
qqp(colMeans(sapply(rep(10,100),rchisq,df=1)),xlab='Sample mean',main='')


Utility of a t-test

I have to note that all of the knowledge I just imparted is somewhat obsolete; now that we have computers, we can do better than t-tests. As Frank notes, you probably want to use Wilcoxon tests anywhere you were taught to run a t-test.

• Good explanation (+1). I would add, however, that the sample size needed for the distribution of means to approximate normality depnds on the degree of non-normalness of the population. For large samples there is no reason to prefer a t-test over a permutations test that makes no assumptions about the distributions. Apr 15, 2011 at 2:01
• +1 although, as far as I know, t-test is fairly resistent to moderate deviations from normality. Also, an interesting related discussion: stats.stackexchange.com/questions/2492/…
– nico
Apr 15, 2011 at 7:02
• good answer, although there is one small detail that you missed: the distribution of the data must have finite variance. T-test is hopeless for comparing difference in location of two Cauchy distributions (or student with 2 degrees of freedom), not because it is "non-robust", but because for these distributions there is additional relevant information in the sample beyond the means and standard deviations which the t-test throws away. Apr 24, 2011 at 0:55
• In addition to this, t-test also naturally yield confidence intervals for the parameter being investigated. (still upvote because of the two first paragraphs which adress the question directly, I just disagree strongly with the third)
– Erik
Mar 7, 2014 at 10:58
• t-test DOES require normality of the population. That's an assumption needed for the t statistic to have a t-Student distribution. If you don't have a normal population, you can't express the t statistic as a standard normal variable divided by the root of a Chi-squared variable divided by its degrees of freedom. Maybe what you are trying to say is that if some conditions are true, like not too much skewness, or a big sample, the test can still be valid even when the population is not normal. Nov 24, 2016 at 15:22

The central limit theorem is less useful than one might think in this context. First, as someone pointed out already, one does not know if the current sample size is "large enough". Secondly, the CLT is more about achieving the desired type I error than about type II error. In other words, the t-test can be uncompetitive power-wise. That's why the Wilcoxon test is so popular. If normality holds, it is 95% as efficient as the t-test. If normality does not hold it can be arbitrarily more efficient than the t-test.

• (+1) Welcome to the site, which I'm glad you've found. I look forward to your participation here. Apr 20, 2011 at 14:52
• (+1) Good point about the Wilcoxon.
– whuber
Apr 20, 2011 at 14:56
• Interesting! Empirically speaking, what do you estimate is a sample size that is 'large-enough' for t-test to be better? Jan 4, 2022 at 17:20
• The power deficiency of the t-test will not be made up for by increasing the sample size, as the Wilcoxon test power will also continue to increase. Jan 4, 2022 at 17:38

See my previous answer to a question on the robustness of the t-test.

In particular, I recommend playing around with the onlinestatsbook applet.

The image below is based on the following scenario:

• null hypothesis is true
• fairly severe skewness
• same distribution in both groups
• same variance in both groups
• sample size per group 5 (i.e., much less than 50 as per your question)
• I pressed the 10,000 simulations button about 100 times to get up to over one million simulations.

The simulation obtained suggests that instead of getting a 5% Type I errors, I was only getting 4.5% Type I errors.

Whether you consider this robust depends on your perspective.

• +1 Good points. The power of the t-test with skewed alternatives, though, can degrade severely (to the point where it is essentially zero even for huge effect sizes).
– whuber
Apr 15, 2011 at 14:20

In my experience with just the one-sample t-test, I have found that the skew of the distributions is more important than the kurtosis, say. For non-skewed but fat-tailed distributions (a t with 5 degrees of freedom, a Tukey h-distribution with $h=0.24999$, etc), I have found that 40 samples has always been sufficient to get an empirical type I rate near the nominal. When the distribution is very skewed, however, you may need many many more samples.

For example, suppose you were playing the lottery. With probability $p = 10^{-4}$ you will win 100 thousand dollars, and with probability $1-p$ you will lose one dollar. If you perform a t-test for the null that the mean return is zero based on a sample of one thousand draws of this process, I don't think you are going to achieve the nominal type I rate.

edit: duh, per @whuber's catch in the comment, the example I gave did not have mean zero, so testing for mean zero has nothing to do with the type I rate.

Because the lottery example often has a sample standard deviation of zero, the t-test chokes. So instead, I give a code example using Goerg's Lambert W x Gaussian distribution. The distribution I use here has a skew of around 1355.

#hey look! I'm learning R!
library(LambertW)

Gauss_input = create_LambertW_input("normal", beta=c(0,1))
params = list(delta = c(0), gamma = c(2), alpha = 1)
LW.Gauss = create_LambertW_output(input = Gauss_input, theta = params)
#get the moments of this distribution
moms <- mLambertW(beta=c(0,1),distname=c("normal"),delta = 0,gamma = 2, alpha = 1)

test_ttest <- function(sampsize) {
samp <- LW.Gauss$rY(params)(n=sampsize) tval <- t.test(samp, mu = moms$mean)
return(tval$p.value) } #to replicate randomness set.seed(1) pvals <- replicate(1024,test_ttest(50)) #how many rejects at the 0.05 level? print(sum(pvals < 0.05) / length(pvals)) pvals <- replicate(1024,test_ttest(250)) #how many rejects at the 0.05 level? print(sum(pvals < 0.05) / length(pvals)) p vals <- replicate(1024,test_ttest(1000)) #how many rejects at the 0.05 level? print(sum(pvals < 0.05) / length(pvals)) pvals <- replicate(1024,test_ttest(2000)) #how many rejects at the 0.05 level? print(sum(pvals < 0.05) / length(pvals))  This code gives the empirical reject rate at the nominal 0.05 level for different sample sizes. For sample of size 50, the empirical rate is 0.40 (!); for sample size 250, 0.29; for sample size 1000, 0.21; for sample size 2000, 0.18. Clearly the one-sample t-test suffers from skew. • In the example you're discussing the power of the test, not its size. The null, by the way, appears to be$p=0$, for which the distribution is degenerate (an atom at a single point): that's about as far from normality as one can be! – whuber Apr 15, 2011 at 14:22 The central limit theorem establishes (under the required conditions) that the numerator of the t-statistic is asymptotically normal. The t-statistic also has a denominator. To have a t-distribution you'd need the denominator to be independent and square-root-of-a-chi-square-on-its-df. And we know it won't be independent (that characterizes the normal!) Slutsky's theorem combined with the CLT would give you that the t-statistic is asymptotically normal (but not necessarily at a very useful rate). We would want a theorem that would establish that the t-statistic is approximately t-distributed when there's non-normality, and how fast it comes in. (Of course, eventually the t- will go close to the normal as well, and in that sense we will often tend to get about the right significance level, at least in sufficiently large samples for all these effects to "work" sufficiently for our individual requirements.) However, more important* than the purported level-robustness of the $$t$$ (at larger sample sizes) is the effect on its power. Note that the asymptotic relative efficiency of the t-test relative to the Wilcoxon-Mann-Whitney (for example) may be 0 (i.e. as sample sizes become large while effect sizes you want to pick up shrink, you may require vastly larger samples for the t to have the same power as an obvious alternative). So while the t-test may end up having a nice normal-looking null distribution in many cases if $$n$$ is large enough, its performance under the null isn't really what people care most about -- it is the performance under the alternative -- and there it may not be so great, if you care about rejecting the null in cases where the effect is not so easy to pick up. * for one thing, there are other ways to control the significance level, such as via a permutation test • While we may not know if the sample mean and variance are independent, we always know that they are uncorrelated. This is because the sample mean is a function of bivariate sums$x_{i}+x_{j}$, and sample variance is a function of bivariate differences$x_{i}-x_{j}$(these are called "U statistics") and we have$cov(x_{i}+x_{j},x_{i}-x_{j})=var(x_{i})-var(x_{j})+cov(x_{i},x_{j})-cov(x_{j},x_{i})=0$as long as the distribution is "homogenous"$var(x_{i})=var(x_{j})\$, which is part of the problem statement. Apr 24, 2011 at 1:08
• Unfortunately, the distinction between uncorrelated and independent is relevant if we are to be ending up with a t-distribution. May 9, 2014 at 22:21

Yes, the Central Limit Theorem tells us this is true. So long as you avoid extremely heavy-tailed traits, non-Normality presents no problems in moderate-to-large samples.

http://www.annualreviews.org/doi/pdf/10.1146/annurev.publhealth.23.100901.140546

The Wilcoxon test (mentioned by others) can have terrible power when the alternative is not a location shift of the original distribution. Furthermore, the way it measures differences between distributions is not transitive.

• Interesting points about the Wilcoxon. However, the t-test has similar difficulties: it's especially bad at detecting shifts that are accompanied by increased variance. The bit about transitivity appears to be mainly a curiosity in the present context; it's difficult to see how it's relevant to the original hypothesis test or its interpretation. (But maybe intransitivity could become important in an ANOVA or multiple comparisons setting.)
– whuber
Nov 22, 2011 at 20:43
• The unequal variance t-test (which is the default in some software) doesn't have the problem with heteroskedasticity. Nov 22, 2011 at 22:02
• Regarding transitivity; reporting the sample means, or differences in means (which is natural using a t-test approach) gives the reader something they can consider when sampling from other populations. The non-transitivity of the Wilcoxon test means that this approach has no such analog; using ranks of data is a very limited approach. Nov 22, 2011 at 22:13
• (1) The Satterthwaite-Welch (unequal variance) test doesn't overcome the power loss I referred to (although it can help a little). (2) I think you're being extreme in characterizing using ranks as "limited." In his reply, @Frank Harrell was referring to studies showing how the Wilcoxon test maintains high efficiency in many settings: this demonstrates how using the ranks is both effective and more flexible, not more limited, compared to t tests.
– whuber
Nov 22, 2011 at 22:36
• (1) No, but it gives the right Type I error rate, in moderate-to-large samples (2) Thanks, but I respectfully disagree. Using t-tests over Wilcoxon makes it much easier to bridge the gap between testing and using confidence intervals. If one only wants to do testing, and never looks beyond the two groups in a study, Wilcoxon of course has situations where it works well. But often we don't want to do just testing, and want to help users generalize the results to other situations; the Wilcoxon test is then not helpful. Nov 22, 2011 at 23:39

About the use of Wilcoxon-Mann-Whitney test as an alternative I recommend the paper The Wilcoxon-Man-Whitney test under scrutiny

As a test of means or medians, the Wilcoxon–Mann–Whitney (WMW) test can be severely nonrobust for deviations from the pure shift model.

These are the recommendation of the authors of the paper:

The rank transformation can alter means, standard deviations, and skewnesses of the two samples differently. The only situation in which the rank transformation is guaranteed to achieve a beneficial effect is when distributions are identical and sample sizes are equal. For deviations from these rather strict assumptions, the effects of the rank transformation on sample moments are unpredictable. In the simulation study of the paper, the WMW test was compared with the Fligner–Policello test (FP), the Brunner–Munzel test (BM), the two-sample T test (T ), the Welch U test (U), and the Welch U test on ranks (RU). The four rank-based tests (WMW, FP, BM, and RU) performed similarly, although the BM test was frequently a little better than the others. When the sample sizes were equal, the parametric tests (T and U) were superior to the rank-based tests under the null hypothesis of equal means, but not under the null hypothesis of equal medians. When the sample sizes were unequal, the BM, RU, and U tests performed best. For several settings, small changes in population properties led to large alterations in the performance of the tests. In summary, the large sample approximate WMW test can be a poor method for comparing the means or medians of two populations, unless the two distributions have equal shapes and equal scales. This problem also seems to apply in various degrees to the exact WMW test, the FP test, the BM test, and the Welch U test on ranks. When using the WMW test, the authors recommend that the properties of the ranked samples are thoroughly investigated for signs of skewness and variance heterogeneity. If such effects are present, the Welch U test is preferable in many situations.