# Do you ever test the pre-asymptotic behaviour of a variable before performing a comparison of means t-test?

I understand that it is not the normality of a random variable that matters in a t-test, but rather the fact that the distribution of the mean follows a normal distribution for large samples. However, is it sometimes useful to test how large a sample must be before the assumption of a normal distribution is warranted? For example, if the number of samples required were very large (much larger than n=30), would assuming a normal distribution be inappropriate? If so, how would you go about checking this?

The inspiration for this question comes from reading about Nassim Nicholas Taleb's Kappa metric.

• A t-statistic consists of a numerator and a denominator. You can't just focus on the numerator. Asymptotic normality for the statistic as a whole requires invoking additional results ... and that would give you a Z-test (in the limit) not a t-test at finite sample sizes. To argue for a t-test in finite samples we would need to establish that the approximation to the significance level was close enough for our purposes in some specific case (at some specific sample size). It's not hard to find cases where the CLT applies but at a given sample size the approximation is poor (try the binomial) Jan 31, 2022 at 0:22
• ... and that would just give us that $\alpha$ was "close enough", not that the test would be useful. We have said nothing about power yet. Jan 31, 2022 at 0:23

Of course, the main issue is for the t statistic to have a t distribution with the appropriate degrees of freedom.

For normal data $$\bar X$$ and $$S$$ are stochastically independent. (One might argue that they are not functionally independent because $$S^2$$ can be defined in terms of $$\bar X.)$$ By contrast, for exponential data $$\bar X$$ and $$S$$ are not independent.

So, technically, it not enough for $$n$$ to be large enough that $$\bar X$$ is nearly normal. In order for the distribution of a t statistic, e.g, $$T = \frac{\bar X-\mu_0}{S/\sqrt{n}},$$ to have the appropriate t distribution, its numerator and denominator should be independent.

Using goodness-of-fit tests to check data for normality before doing a t test has been deprecated. But it is a good idea to look at plots of the data to see if they are nearly symmetrical and free of far outliers in either direction before using the data for a t test.

For example, consider a sample of size $$n = 100$$ from a Pareto distribution with minimum value $$1$$ and shape parameter $$\alpha = 10,$$ and mean $$\mu = 10/9.$$ [Observations can be sampled as $$e^Y,$$ where $$Y\sim\mathsf{Exp}(\mathrm{rate}=10).]$$ Each row of the matrix MAT below contains such a sample.

set.seed(2022)
m = 10^4;  n = 100
MAT = matrix(exp(rexp(m*n, 10)), nrow=m)


A boxplot of the first sample shows marked right-skewness and outliers.

boxplot(MAT[1,], horizontal=T, col="skyblue2")


a = rowMeans(MAT)
s = apply(MAT,1,sd)
cor(a, s)
[1] 0.7201542


Sample means and SDs are clearly correlated. But one might consider that the means $$\bar X$$ (a in the code) are "close enough" to normal.

hist(a, prob=T, br=30, col="skyblue2")


However, $$T$$ statistics are not distributed as $$\mathsf{T}(\nu=99)$$---especially not in the tails, where values are used to decide whether to reject the null hypothesis. So one should not trust results of a t test on such Pareto data.

t = (a - 10/9)/(s/sqrt(n))
hist(t, prob=T, br=30, col="skyblue2")


By contrast, samples of size $$n = 50$$ from a standard uniform distribution behave well.

set.seed(129)
m = 10^4;  n = 50
MAT = matrix(runif(m*n), nrow=m)
a = rowMeans(MAT);  s = apply(MAT,1,sd)
cor(a, s)
[1] -0.01452768   # nearly independent


$$T$$ statistics are nearly distributed as $$\mathsf{T}(\nu=49).$$ So one could trust results of a t test on such uniform data.

t = (a - 1/2)/(s/sqrt(50))
hist(t, prob=T, br=20, col="skyblue2")


When you are using a statistical test that relies on the central limit theorem (as the T-test does), the test is only valid if the underlying distribution has a finite variance. If the underlying distribution of the data has sufficiently heavy tails that it does not have a finite variance then the central limit theorem does not apply and the test will not work properly. Thus, the most important preliminary issue for applying a T-test is whether or not the data come from a distribution where the rate of decay in the tails is sufficiently fast to give finite variance; anything faster than cubic decay is sufficient to give a finite variance.

To examine the rate of decay in the tails of a distribution, using a sample from that distribution, the usual diagnostic tool is a tail plot. You can find a related post here discussing the construction of a tail plot and the examination of the plot to see if the distribution has a finite variance. An example of tail plots for the two tails of a dataset is shown below.

I think the normality on the population does matter. It is a necessity for the t-distribution to hold. See my own question body for why.

So I think it is a good idea to check the normality of the population.

Note: when normality is not there, t-test has certain robustness against it, but that is another topic.

• Few will disagree that taking a look at the data distribution is a good idea. But if people actually followed your advice, it would rule out all applications of t tests (and Z tests) to Binomial data! The point is that because some extremely Non-normal populations are perfectly fine for applying the t test, it's not Normality of the population one needs to check. What matters is the sampling distribution of the statistic. An example of this is studied in detail at stats.stackexchange.com/questions/69898.
– whuber
Feb 1, 2022 at 16:54
• @whuber, how do we check the sampling distribution? In real scenario, all we have is one single $\bar{X}$ from the sample we collect Feb 1, 2022 at 19:31
• My answer in the thread I linked to shows how one can check the sampling distribution with a bootstrap. It includes working R code.
– whuber
Feb 1, 2022 at 19:51