2
$\begingroup$

I am trying to learn T statistics and T test using simulation. Why my simulation shows the population distribution does not matter? (In cal_t_stats1, sample was coming from normal distribution and cal_t_stats2 sample was coming from uniform distribution. In both case, it satisfy T distribution with degree of freedom N-1)

enter image description here

N=1e2
rept=1e4

cal_t_stats1<-function(i){
  x=rnorm(N,mean=3,sd=2)
  t=(mean(x-3))/(sd(x)/sqrt(N))
  t
}

cal_t_stats2<-function(i){
  x=runif(N)
  t=(mean(x-0.5))/(sd(x)/sqrt(N))
  t
}

t1=sapply(1:rept,cal_t_stats1)
t2=sapply(1:rept,cal_t_stats2)

par(mfrow=c(1,2))
hist(t1,100,freq = F)
curve(dt(x,N-1),-4,4,add=T,col='red',lwd=2)
hist(t2,100,freq = F)
curve(dt(x,N-1),-4,4,add=T,col='red',lwd=2)
$\endgroup$
  • 1
    $\begingroup$ Hint: Central limit theorem. $\endgroup$ – amoeba says Reinstate Monica Jan 26 '17 at 21:55
  • $\begingroup$ The t-test by definition assumes the average of your samples is normally distributed, which is corroborated by the CLT. $\endgroup$ – Alex R. Jan 26 '17 at 21:56
  • 1
    $\begingroup$ make N smaller and it'll break down $\endgroup$ – Taylor Jan 26 '17 at 22:38
  • 1
    $\begingroup$ @amoeba 1. a t-statistic is a ratio of a numerator and a denominator. It's not sufficient to assert a fact about the numerator. 2. The CLT is a result in the limit as n goes to infinity; to establish that the mean is well approximated by a normal distribution at some finite sample size does not itself follow from the CLT. 3. (admittedly a side point) Showing the numerator is well approximated by a normal doesn't mean that the ratio is better approximated by a t (it might be worse, in which case you should just use the normal approximation for preference). $\endgroup$ – Glen_b -Reinstate Monica Jan 27 '17 at 0:35
  • 1
    $\begingroup$ @jon The distributions of the max and min will surprise you, then. The CLT does not apply to every statistic! $\endgroup$ – whuber Jan 27 '17 at 6:12
4
$\begingroup$

The student t test is based on a few assumptions (hereafter I take for granted that you use the one-sample t-test).

It's formula is the difference between means divided by the standard deviation and times the square root of the sample size. The last term is a constant.

The difference between means, according to the Central Limit Theorem (CLT), should conform to a normal distribution if

  1. The sample is taken from a population who has a definite (knowable in principle) mean;
  2. the sample is taken from a populations who has a definite (knowable in principle) standard deviation;
  3. the sample size is large.

So, as @Taylor pointed out, if your $N$ is very small, the t-test won't be accurate anymore.

The sample standard deviation, if the population is indeed normal, should conform to a Chi-square distribution. Hence, if you data set is not taken from a normal distribution, the denominator of your t-test will be wrong and so the t-test won't be accurate anymore. Try a very skewed population, and you will see that clearly.

Both a normally distributed numerator (owing to the CLT) and a chi-square denominator were used to create the t distribution. Hence, when both are satisfied, the t-test is accurate.

It turns out that the t-test is robust, i.e. it can tolerate some violation of the normality assumption. It will remain fairly accurate if sample size is not too small (a general consensus is that $N$ should at least be 30), and when the true population, while not being normal, is symmetrical. As you used the uniform distribution (a symmetrical distribution), it explains why you did not notice any sizable inaccuracies in the test result.

$\endgroup$
  • $\begingroup$ To be more precise, one would need to mention Slutsky's theorem, as e.g. explained here stats.stackexchange.com/questions/44262. $\endgroup$ – amoeba says Reinstate Monica Jan 26 '17 at 23:29
  • $\begingroup$ This is a good account of the major issues (+1). Note that to deduce that the t-statistic approximates a Student t distribution, one also needs to establish the (near) independence of the mean and the standard deviation. The $N\ge 30$ rule is definitely not a "general consensus": see stats.stackexchange.com/questions/6989, inter alia, for a discussion. $\endgroup$ – whuber Jan 26 '17 at 23:36
  • $\begingroup$ +1 for detailed answer. I think this is why I am get confused. Does that mean T test does not need to have normal assumption? $\endgroup$ – Haitao Du Jan 27 '17 at 3:01
  • $\begingroup$ @hxd1011: for very large samples, no. As the sample gets smaller, the distribution should get to a normal distribution (for example, with a sample of about 100, make sure it is symmetrical; for a sample below 60, make sure it is symmetrical and unimodal, etc.). $\endgroup$ – Denis Cousineau Jan 28 '17 at 0:17
1
$\begingroup$

Since I left a comment here, I've not had the time to come back and continue with an answer. I took a sick day today, so I've finally had the time to provide some additional comments to this question. Albeit Denis' answer is correct, I feel it is lacking in clear corrections/critique of the simulation the OP began his question with. I believe the objective for OP was to illustrate how sampling from a non-Normal distributed sample affects the t-statistic, and presumably Type I errors and power of the test. Thus, the purpose for this second answer is simply to add to the first to provide a more thorough answer.

So first, some pointed out stating that the reason OP did not see a difference between histograms was because of the sample size. As stated in my comment, I disagree. If for 1000 iterations you calculate the mean of a sample of 5 from a $U(0,1)$, you will still see a normal-ish histogram.

N=5
rept=1e3
cal_t_stats2<-function(i){
    x=runif(N) ## U(0, 1)
    t=(mean(x) -0.5)/(sd(x)/sqrt(N)) ## I made a correction here.
    t ## scaled mean
}

t2=sapply(1:rept,cal_t_stats2)

hist(x = t2, breaks = "Scott", freq = F, xlim = c(-5, 5))  ## Changed the breaks to Scott
curve(dnorm(x, mean = 0, sd = 1),-4,4,add=T,col='red',lwd=2) ## dnorm instead of dt; CLT

enter image description here

So I would disagree that the issue is sample size. What the example shows is that the distribution of the t-test centers around zero.

Next, below is a modified example taken from Veranzi (https://cran.r-project.org/doc/contrib/Verzani-SimpleR.pdf, page 51).

The example works to illustrate how $\frac{\bar{X} - \mu}{\sigma/ \sqrt{n}} \sim N(0, 1)$ for skewed data. In the example, $\sigma$ is estimated from the data, as opposed to being known apriori.

Example: CLT with exponential data

Let’s do one more example. Suppose we start with a skewed distribution, the central limit theorem says that the average will eventually look normal. That is, it is approximately normal for large n. What does “eventually” mean? What does “large” mean? We can get an idea through simulation.

f = function(n=100,mu=10) (mean(rexp(n,1/mu))-mu)/(mu/sqrt(n))

iters = c(5, 10, 20, 100)  ## sample sizes
par(mfrow = c(2,2), mar = c(2,2,2,2))
for(i in iters){
  s <- replicate(n = 1000, expr = f(n = i), simplify = TRUE)
  hist(x = s, breaks = "Scott", probability = TRUE, 
       main = paste("Number of samples: ", i), xlab = "mean", ylab = "probability",
       xlim = c(-5, 5), ylim = c(0, 0.45))
  curve(dnorm(x, mean = 0, sd = 1) ,from =  -4, to = 4 , add=T, col='blue',lwd=2)
}

library(car)
dev.off()
s <- replicate(n = 1000, expr = f(n = 10^3), simplify = TRUE)
hist(x = s, breaks = "Scott", probability = TRUE, 
     main = "Number of samples: 1000", xlab = "mean", ylab = "probability",
     xlim = c(-5, 5), ylim = c(0, 0.45))
curve(dnorm(x, mean = 0, sd = 1) ,from =  -4, to = 4 , add=T, col='blue',lwd=2)
car::qq.plot(s, main = "Normal QQ Plot")

First, data is being generated from a an exponential distribution with rate 0.1, at samples of 5, 10, 20, 100 and 1000. The first four graphs show how the scaled means progress to looking more normal-ish as sample size is increased for each scaled mean (or t-stat) calculation. enter image description here

Finally, at 1000 samples, the histogram appears settled.

enter image description here

Beyond this, I think the next issue is power of a t-test if the sampled data is non-normal.

  1. A More Realistic Look at the Robustness and Type II Error Properties of the $t$ test to Departures From Population Normality

TL;DR The authors say that data that come from a nice looking normal-ish distribution is pretty rare (unicorn), and is quite evident in education research data. So the authors run some Monte Carlo simulations on some 8 "real world" data sets whose distributions look funky and not very bell-curve-shaped. Their results show that the two sample t-test was fairly robust (usually, but not always) against Type I errors. Also, the t-test did fine (usually, but not always) against Type II errors. The authors comment on when the two sample t-test does well (read the abstract).

  1. Robustness?

I've not finished reading this paper, but this brings some useful commentary when discussing the topic of "robustness" of a statistic. It was also cited in the first article.

That said, I hope this provides some useful comments to the simulation, or at least a good reference or two.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.