Edit: As requested by the OP, I have changed the code to also perform the same experiments for the z-test, so that the performance of both tests can be compared.
set.seed(1)
try.one <- function(gen, n, N=1000000, alpha=0.05) {
crit.t <- qt(1 - alpha/2, n-1)
reject.t <- 0
crit.z <- qnorm(1 - alpha/2)
reject.z <- 0
for (j in 1:N) {
X <- gen(n)
Z <- sqrt(n) * mean(X) / sd(X)
if (abs(Z) > crit.t) {
reject.t <- reject.t + 1
}
if (abs(Z) > crit.z) {
reject.z <- reject.z + 1
}
}
p.t <- reject.t/N
p.z <- reject.z/N
list(prob=pprob.t=p.t, sd=sqrtsd.t=sqrt(p*p.t*(1-p.t)/N), prob.z=p.z, sd.z=sqrt(p.z*(1-p.z)/N))
}
distributions <- c("normal", "uniform", "double exponential", "exponential",
"discrete", "asym. discrete")
try.all <- function() {
dist.name <- character(0)
nn <- numeric(0)
fp.rate.t <- numeric(0)
sd std.err.t <- numeric(0)
fp.rate.z <- numeric(0)
std.err.z <- numeric(0)
for (dist in distributions) {
if (dist == "normal") {
gen <- rnorm
} else if (dist == "uniform") {
gen <- function(n) runif(n, -1, 1)
} else if (dist == "double exponential") {
gen <- function(n) rexp(n) * sample(c(-1,1), n, replace=TRUE)
} else if (dist == "exponential") {
gen <- function(n) rexp(n) - 1
} else if (dist == "discrete") {
gen <- function(n) sample(c(-1,1), n, replace=TRUE)
} else if (dist == "asym. discrete") {
gen <- function(n) sample(c(-1, 9), n, replace=TRUE, prob=c(0.9,0.1))
}
for (n in c(10, 30, 100)) {
row <- try.one(gen, n)
dist.name <- c(dist.name, dist)
nn <- c(nn, n)
fp.rate.t <- c(fp.rate.t, row$prob)
sd <- c(sd, row$$prob.t)
std.err.t <- c(std.err.t, row$sd.t)
fp.rate.z <- c(fp.rate.z, row$prob.z)
std.err.z <- c(std.err.z, row$sd.z)
}
}
data.frame(dist.name, n=nn, fp.rate.t, std.err=sderr.t, fp.rate.z, std.err.z)
}
print(try.all(), row.names=FALSE)
dist.name n fp.rate.t std.err.t fp.rate.z std.err.z
normal 10 0.050029 0.0002180048 0.081694 0.0002738980
normal 30 0.050059 0.0002180667 0.059824 0.0002371605
normal 100 0.049930 0.0002178004 0.052726 0.0002234859
uniform 10 0.054490 0.0002269820 0.084445 0.0002780540
uniform 30 0.050906 0.0002198058 0.060263 0.0002379735
uniform 100 0.050116 0.0002181843 0.053001 0.0002240355
double exponential 10 0.042272 0.0002012090 0.074645 0.0002628177
double exponential 30 0.047506 0.0002127185 0.057410 0.0002326244
double exponential 100 0.049646 0.0002172125 0.052526 0.0002230852
exponential 10 0.099738 0.0002996503 0.130045 0.0003363529
exponential 30 0.072758 0.0002597389 0.082090 0.0002745018
exponential 100 0.058040 0.0002338191 0.060755 0.0002388804
discrete 10 0.021386 0.0001446673 0.109666 0.0003124730
discrete 30 0.042853 0.0002025256 0.042853 0.0002025256
discrete 100 0.056972 0.0002317891 0.056972 0.0002317891
asym. discrete 10 0.350463 0.0004771150 0.350463 0.0004771150
asym. discrete 30 0.044408 0.0002059998 0.191153 0.0003932093
asym. discrete 100 0.067916 0.0002516017 0.067916 0.0002516017
The rate of type I errors is listed in the column
fp.rate. As expected, for the normal distribution this is very close to $5\%$.In nearly all cases, the rate of type I errors gets closer to $5\%$ as $n$ increases, sometimes from below and sometimes from above. The only exception is the asymmetric discrete distribution.
The weight of the tails seems not to have too much effect: the test performs reasonably well for both uniform and double exponential distributions.
For small sample size ($n=10$) there are notable deviation of the type I error rate from $5\%$, both for the discrete distributions and for the asymmetric distributions.
The worst case is the discrete, asymmetric distribution where the t-test at $5\%$-level shows type I errors in $35\%$ of the cases. Given this huge discrepancy, I would argue that care is required when attempting to use the $t$-test for distributions which are far from normal.
Edit: Using the updated code, we can also compare the performance of the t-test to the performance of a z-test (still using the sample variance):
As expected, for normally distributed data the z-test performs worse that the t-test (because we didn't use the exact variance). The effect is quite noticeable for $n=10$ and nearly disappears for $n=100$. For $n=10$, the t-test seems superior to the $z$-test (using estimated variances) for all examples tested.
The worst case (assymetric+discrete, $n=10$) is equally bad for both tests.
For $n=100$ the results of both tests are very similar, but in some cases the t-test seems to perform slightly better.