Okay, so simulation time (thank you).
Lets start by looking at a case, where the central limit theorem holds. Here I compare the distribution of the sample mean from three populations each distributed as: $N (1,1), \text{exp}(\lambda = 1), \text{Weibull}(\lambda = 1, k = 1)$. I use thes specifications, since they should more or less yield the same mean, which makes a neat plot.
In R we could do something like:
# Set up sample function:
sample_mean <- function(simulations = 10000, sample_size = 1000, population){
data <- rep(0, simulations)
for (i in 1:simulations){
srs <- sample(population, sample_size, replace = TRUE) # Simple random sample
data[i] <- mean(srs) # Save the mean
}
return(data)
}
# Create empty data storage:
set.seed(42)
data <- data.frame(matrix(, nrow = 10000, ncol = 3))
colnames(data) <- c("norm_dist", "exp_dist", "wei_dist")
# Use sample function:
data$norm_dist <- sample_mean(population = rnorm(10^6, mean = 1))
data$exp_dist <- sample_mean(population = rexp(10^6))
data$wei_dist <- sample_mean(population = rweibull(10^6, shape = 1))
# Plot it, and compare:
require(reshape)
data <- melt(data)
colnames(data)[2] <- "Sample_mean"
ggplot(data = data, aes(x = Sample_mean, colour = variable)) + geom_density()
So the code will: Create a population of 1.000.000 numbers, from each of the above distributions. Then the function will draw a sample of 1000 (with replacement). Then rerun 10.000 times, finally it plots the distribution. This leaves us with something like this:
So far so good, we get something that looks like a normal dsitribution of the sample mean - no matter the type of distribution. Now suppose that we keep a sample of 1000 in each simulation, but scale down the number of simulations to 100 then we get:
Ugh! This does not look nearly as good as before. The tails are fatter, and the distribution is much less "normal"-like. This should tell you one important thing: the central limit theorem (CLT) is only an asymptotic result.
Therefore to apply the CLT, for a sample of 10 or 20, would require some strong assumptions - typically that the data is normal. If you just have some "not even close to normal" data, then you can get very far off, leading to wrong conclusions.
Also, the convergence in my code is quite fast. If you scale down the number of observations in each sample to say 20, then you still get pretty results. If you use "real world" data, you might need much more, to get something similar.
So your question was about testing. If the distribution of the population is normal, then you you should use a t distribution since:
$$
t = \frac{\mu - \bar{x}}{s.e.(\bar{x})} \sim t_{n-1}
$$
If the data is not normal, then we have:
$$
z = \frac{\mu - \bar{x}}{s.e.(\bar{x})} \sim ^{A} N(0,1)
$$
But now it is only an asymptotic result, some say that this is good at 100 observations. But if the data is skewed, or is not unimodal or have many outliers, Then it is hard to justify anything for such a small sample.
Futher the distinction bwtween normal or t does not matter, once when you get over 1000 observations (and 1000 is not even close to asymptotic). The p-values are essentially the same.
Note that the CLT (in the usual $\bar{x}$ form), requires that we have a simple random sample from a population with finite mean and varinace. This is something that many textbooks, at least in the social sciences, overlooks. This means that there actually cases where the CLT does not apply, for instance if you sample from the the Cauchy distribution.
