Why my simulation shows T test does not need to have normal distribution assumption on population?

Question

I am trying to learn T statistics and T test using simulation. Why my simulation shows the population distribution does not matter? (In cal_t_stats1, sample was coming from normal distribution and cal_t_stats2 sample was coming from uniform distribution. In both case, it satisfy T distribution with degree of freedom N-1)

    N=1e2
    rept=1e4

    cal_t_stats1 <- function(i){
      x=rnorm(N,mean=3,sd=2)
      t=(mean(x-3))/(sd(x)/sqrt(N))
      t
    }

    cal_t_stats2<-function(i){
      x=runif(N)
      t=(mean(x-0.5))/(sd(x)/sqrt(N))
      t
    }

    t1=sapply(1:rept,cal_t_stats1)
    t2=sapply(1:rept, cal_t_stats2)

    par(mfrow=c(1,2))
    hist(t1,100,freq = F)
    curve(dt(x,N-1),-4,4, add=T, col='red', lwd=2)
    hist(t2,100,freq = F)
    curve(dt(x,N-1),-4,4, add=T, col='red', lwd=2)

Answer 1

The student t test is based on a few assumptions (hereafter I take for granted that you use the one-sample t-test).

It's formula is the difference between means divided by the standard deviation and times the square root of the sample size. The last term is a constant.

The difference between means, according to the Central Limit Theorem (CLT), should conform to a normal distribution if

1. The sample is taken from a population who has a definite (knowable in principle) mean;
2. the sample is taken from a populations who has a definite (knowable in principle) standard deviation;
3. the sample size is large.

So, as @Taylor pointed out, if your $N$ is very small, the t-test won't be accurate anymore.

The sample standard deviation, if the population is indeed normal, should conform to a Chi-square distribution. Hence, if you data set is not taken from a normal distribution, the denominator of your t-test will be wrong and so the t-test won't be accurate anymore. Try a very skewed population, and you will see that clearly.

Both a normally distributed numerator (owing to the CLT) and a chi-square denominator were used to create the t distribution. Hence, when both are satisfied, the t-test is accurate.

It turns out that the t-test is robust, i.e. it can tolerate some violation of the normality assumption. It will remain fairly accurate if sample size is not too small (a general consensus is that $N$ should at least be 30), and when the true population, while not being normal, is symmetrical. As you used the uniform distribution (a symmetrical distribution), it explains why you did not notice any sizable inaccuracies in the test result.

Answer 2

Since I left a comment here, I've not had the time to come back and continue with an answer. I took a sick day today, so I've finally had the time to provide some additional comments to this question. Albeit Denis' answer is correct, I feel it is lacking in clear corrections/critique of the simulation the OP began his question with. I believe the objective for OP was to illustrate how sampling from a non-Normal distributed sample affects the t-statistic, and presumably Type I errors and power of the test. Thus, the purpose for this second answer is simply to add to the first to provide a more thorough answer.

So first, some pointed out stating that the reason OP did not see a difference between histograms was because of the sample size. As stated in my comment, I disagree. If for 1000 iterations you calculate the mean of a sample of 5 from a $U(0,1)$, you will still see a normal-ish histogram.

N=5
rept=1e3
cal_t_stats2<-function(i){
    x=runif(N) ## U(0, 1)
    t=(mean(x) -0.5)/(sd(x)/sqrt(N)) ## I made a correction here.
    t ## scaled mean
}

t2=sapply(1:rept,cal_t_stats2)

hist(x = t2, breaks = "Scott", freq = F, xlim = c(-5, 5))  ## Changed the breaks to Scott
curve(dnorm(x, mean = 0, sd = 1),-4,4,add=T,col='red',lwd=2) ## dnorm instead of dt; CLT

So I would disagree that the issue is sample size. What the example shows is that the distribution of the t-test centers around zero.

Next, below is a modified example taken from Veranzi (https://cran.r-project.org/doc/contrib/Verzani-SimpleR.pdf, page 51).

The example works to illustrate how $\frac{\bar{X} - \mu}{\sigma/ \sqrt{n}} \sim N(0, 1)$ for skewed data. In the example, $\sigma$ is estimated from the data, as opposed to being known apriori.

Example: CLT with exponential data

Let’s do one more example. Suppose we start with a skewed distribution, the central limit theorem says that the average will eventually look normal. That is, it is approximately normal for large n. What does “eventually” mean? What does “large” mean? We can get an idea through simulation.

f = function(n=100,mu=10) (mean(rexp(n,1/mu))-mu)/(mu/sqrt(n))

iters = c(5, 10, 20, 100)  ## sample sizes
par(mfrow = c(2,2), mar = c(2,2,2,2))
for(i in iters){
  s <- replicate(n = 1000, expr = f(n = i), simplify = TRUE)
  hist(x = s, breaks = "Scott", probability = TRUE, 
       main = paste("Number of samples: ", i), xlab = "mean", ylab = "probability",
       xlim = c(-5, 5), ylim = c(0, 0.45))
  curve(dnorm(x, mean = 0, sd = 1) ,from =  -4, to = 4 , add=T, col='blue',lwd=2)
}

library(car)
dev.off()
s <- replicate(n = 1000, expr = f(n = 10^3), simplify = TRUE)
hist(x = s, breaks = "Scott", probability = TRUE, 
     main = "Number of samples: 1000", xlab = "mean", ylab = "probability",
     xlim = c(-5, 5), ylim = c(0, 0.45))
curve(dnorm(x, mean = 0, sd = 1) ,from =  -4, to = 4 , add=T, col='blue',lwd=2)
car::qq.plot(s, main = "Normal QQ Plot")

First, data is being generated from a an exponential distribution with rate 0.1, at samples of 5, 10, 20, 100 and 1000. The first four graphs show how the scaled means progress to looking more normal-ish as sample size is increased for each scaled mean (or t-stat) calculation.

Finally, at 1000 samples, the histogram appears settled.

Beyond this, I think the next issue is power of a t-test if the sampled data is non-normal.

1. A More Realistic Look at the Robustness and Type II Error Properties of the $t$ test to Departures From Population Normality

TL;DR The authors say that data that come from a nice looking normal-ish distribution is pretty rare (unicorn), and is quite evident in education research data. So the authors run some Monte Carlo simulations on some 8 "real world" data sets whose distributions look funky and not very bell-curve-shaped. Their results show that the two sample t-test was fairly robust (usually, but not always) against Type I errors. Also, the t-test did fine (usually, but not always) against Type II errors. The authors comment on when the two sample t-test does well (read the abstract).

2. Robustness?

I've not finished reading this paper, but this brings some useful commentary when discussing the topic of "robustness" of a statistic. It was also cited in the first article.

That said, I hope this provides some useful comments to the simulation, or at least a good reference or two.