Single sample t-test variance assumptions for non-normal data

Question

So I understand that a key assumption behind the single sample t-test is that the sample mean is normally distributed. This is true when the data themselves are normally distributed, and it is approximately true when you have a large sample size of n>30 by the CLT.

However, another important assumption is that $s^2(n − 1)/\sigma^2$ follows a chi-square distribution, which is met when the data are drawn from a normal distribution. However, I don't see how this assumption is even approximately met when the data are not normally distributed, even with a large sample size.

Can someone explain why we are still comfortable performing a single sample t-test on non-normal data when we have a large sample size, even though this second condition is (I think) not met? Wikipedia says something about Slutskey's theorem implying that this assumption being broken has little effect on the distribution of the test statistic. Can someone walk me through this?

Answer 1

Some simulations following from my Comment:

With a sample of size $n = 10$ from $\mathsf{Norm}(0,1)$ a t test rejects $H_0: \mu = 0$ vs. $H_a: \mu \ne 0$ at the 5% level about 5% of the time, as it should. Furthermore, the power for testing $H_0: \mu = 1$ vs. $H_a: \mu \ne 1$ is about 80%.

set.seed(117)
pv = replicate(10^5, t.test(rnorm(10,0,1))$p.val)
mean(pv <= .05)
[1] 0.05

pv = replicate(10^5, t.test(rnorm(10,0,1), mu=1)$p.val)
mean(pv <= .05)
[1] 0.80169

The uniform distribution $\mathsf{Unif}(-\sqrt{3},\sqrt{3})$ has $\mu=0,\sigma=1).$ Let's look at the performance of t test for similar hypotheses and alternatives from a population with this uniform distribution. The actual significance level (of a test intended to have significance level 5%) is about 5.5% and power about 80%. Not quite the same as for normal data, but the t test shows tolerable robustness even for $n = 10.$

set.seed(118)
pv = replicate(10^5, t.test(runif(10,-sqrt(3),sqrt(3)))$p.val)
mean(pv <= .05)
[1] 0.05517

pv = replicate(10^5, t.test(runif(10,-sqrt(3),sqrt(3)), mu=1)$p.val)
mean(pv <= .05)
[1] 0.80508

A similar simulation for data from $\mathsf{Exp}(1),$ with $\mu = \sigma=1:$ We test $H_0: \mu = 1$ vs. $\ne$ and $H_0: \mu = 2$ vs. $\ne$ for samples of size $n = 10.$ The true significance level is nearly 10% (too many false rejections), which makes it difficult to interpret the alleged 'power' of about 76%. Not satisfactory performance.

set.seed(119)
pv = replicate(10^5, t.test(rexp(10), mu=1)$p.val)
mean(pv <= .05)                  # using P-values as above
[1] 0.09999

set.seed(119)
t.stat = replicate(10^5, t.test(rexp(10), mu=1)$stat)
mean(abs(t.stat) >= qt(.975,9))  # using t statistics
[1] 0.09999

The t statistic under $H_0$ has far from the distribution $\mathsf{T}(\nu = 9),$ as shown in the histogram below:

hist(t.stat, prob=T, br = 50, col="skyblue2")
curve(dt(x,9), add=T, col="red", lwd=2)
abline(v = qt(c(.025,.975), 9), lty="dotted")

pv = replicate(10^5, t.test(rexp(10), mu=2)$p.val)
mean(pv <= .05)
[1] 0.75679

Exponential data again, but with $n=40.$ The true significance level is nearer to 5% and the power is quite good. This is not really an accurate test, but some people might find it 'good enough'.

set.seed(119)
pv = replicate(10^5, t.test(rexp(40), mu=1)$p.val)
mean(pv <= .05)
[1] 0.06712

pv = replicate(10^5, t.test(rexp(40), mu=2)$p.val)
mean(pv <= .05)
[1] 0.99787

However, for an accurate test on centers of exponential (or other markedly skewed) samples of size $n = 40,$ one might want to explore alternative tests. In particular, if data are known to be exponential then an exact parametric test using gamma distribution is available.

Answer 2

The issue you mention here not only applies to one sample $t$-tests, but also to two-samples $t$-tests. In some elementary statistics textbooks, the general advice in these situations is that, whenever you have a not-normal population or an unknown underlying distribution, if the sample size is larger than $30$, then the $t$-statistic will be approximately normal, independently of the underlying population. This statement is then justified by the use of the CLT, but the problem is whenever you have to estimate the population standard deviation with the sample estimator, which you do in a single sample $t$-test, you can only really use the CLT if the sample standard deviation works as a measure of dispersion on the underlying distribution. This is generally the case if the population is symmetrical and has thinner tails than that of a normal distribution, for example, but in a lot of cases this may not be the case or it is simply not known if it is.

Thus, in most cases of non-normality, we cannot rely on the CLT to justify that $\bar{X}$ approaches normality. Another problem that arises with non-normality is that $(n-1)S^{2}/\sigma^{2}$ may have a distribution that is significantly different than a scaled chi-squared distribution, which would imply that the variable

\begin{equation*} \frac{\left(\bar{X}-\mu\right)\sqrt{n}}{s} \end{equation*}

would not be a $t$-Student variable. On a side note, Slutsky's theorem implies that the $t$-statistic will eventually approach normality, making the approximation to the $t$-distribution appropriate, but the problem with this is that we do not know how large the sample has to be in order to make this happen and this certainly depends in how different the underlying density is from normality. In a related question, it was shown that a sample size of $20000$ observations, drawn for a log-normal distribution, still was not large enough to valid the approximation to the $t$-distribution.

In practical cases, what really matters is how close the underlying density is from normality and not necessarily how large is the sample size. As for the guide of having more than $30$ observations, even if we could completely rely on the CLT, there is not a universal sample size that could justify the use of the CLT, as the size depends on how different the population distribution is from normality.