Here is an attempt at answering the question using numerical experiments: using Monte Carlo estimation it is easy to determine the rate of type I errors for the test with a given distribution of input data. Here I try data from the following distributions:
Normally distributed data: here the t-test is guaranteed to work.
Samples from the uniform distribution on $[-1,1]$: this is a prototype for a distribution with light tail (or rather, the extreme case of no tails).
The double-exponential distribution: this is a distribution with heavier tails than the normal distribution has.
A shifted exponential distribution, $\mathrm{Exp}(1) - 1$: this is a very asymmetric distribution, with a tail only on one side.
The discrete uniform distribution on the set $\{-1,+1\}$: this could be seen as an extreme case of a bi-modal distribution.
The discrete distribution with $P(X=-1) = 0.9$ and $P(X=9)=0.1$: this is very far from a normal distribution because it is both discrete and very asymmetric.
Since we expect the test to get more accurate as $n$ increases, I try only small and moderate values of $n$, namely $n \in \{10, 30, 100\}$. For the significance level I choose the commonly used value $\alpha = 5\%$.
My experiment is performed using the following R script: the script simulates $N=1,000,000$ dataset of size $n$, applies the t-test and counts how often $H_0\colon \mu=0$ is (wrongly) rejected. If the t-test still works, this should be the case in $5\%$ of the cases, any deviation from $5\%$ indicates that for the given distribution and $n$ the t-test did not perform optimally.
set.seed(1)
try.one <- function(gen, n, N=1000000, alpha=0.05) {
crit <- qt(1 - alpha/2, n-1)
reject <- 0
for (j in 1:N) {
X <- gen(n)
Z <- sqrt(n) * mean(X) / sd(X)
if (abs(Z) > crit) {
reject <- reject + 1
}
}
p <- reject/N
list(prob=p, sd=sqrt(p*(1-p)/N))
}
distributions <- c("normal", "uniform", "double exponential", "exponential",
"discrete", "asym. discrete")
try.all <- function() {
dist.name <- character(0)
nn <- numeric(0)
fp.rate <- numeric(0)
sd <- numeric(0)
for (dist in distributions) {
if (dist == "normal") {
gen <- rnorm
} else if (dist == "uniform") {
gen <- function(n) runif(n, -1, 1)
} else if (dist == "double exponential") {
gen <- function(n) rexp(n) * sample(c(-1,1), n, replace=TRUE)
} else if (dist == "exponential") {
gen <- function(n) rexp(n) - 1
} else if (dist == "discrete") {
gen <- function(n) sample(c(-1,1), n, replace=TRUE)
} else if (dist == "asym. discrete") {
gen <- function(n) sample(c(-1, 9), n, replace=TRUE, prob=c(0.9,0.1))
}
for (n in c(10, 30, 100)) {
row <- try.one(gen, n)
dist.name <- c(dist.name, dist)
nn <- c(nn, n)
fp.rate <- c(fp.rate, row$prob)
sd <- c(sd, row$sd)
}
}
data.frame(dist.name, n=nn, fp.rate, std.err=sd)
}
print(try.all(), row.names=FALSE)
The output, after some minutes, is
dist.name n fp.rate std.err
normal 10 0.050029 0.0002180048
normal 30 0.050059 0.0002180667
normal 100 0.049930 0.0002178004
uniform 10 0.054490 0.0002269820
uniform 30 0.050906 0.0002198058
uniform 100 0.050116 0.0002181843
double exponential 10 0.042272 0.0002012090
double exponential 30 0.047506 0.0002127185
double exponential 100 0.049646 0.0002172125
exponential 10 0.099738 0.0002996503
exponential 30 0.072758 0.0002597389
exponential 100 0.058040 0.0002338191
discrete 10 0.021386 0.0001446673
discrete 30 0.042853 0.0002025256
discrete 100 0.056972 0.0002317891
asym. discrete 10 0.350463 0.0004771150
asym. discrete 30 0.044408 0.0002059998
asym. discrete 100 0.067916 0.0002516017
Some observations about these results:
The rate of type I errors is listed in the column
fp.rate. As expected, for the normal distribution this is very close to $5\%$.In nearly all cases, the rate of type I errors gets closer to $5\%$ as $n$ increases, sometimes from below and sometimes from above. The only exception is the asymmetric discrete distribution.
The weight of the tails seems not to have too much effect: the test performs reasonably well for both uniform and double exponential distributions.
For small sample size ($n=10$) there are notable deviation of the type I error rate from $5\%$, both for the discrete distributions and for the asymmetric distributions.
The worst case is the discrete, asymmetric distribution where the t-test at $5\%$-level shows type I errors in $35\%$ of the cases. Given this huge discrepancy, I would argue that care is required when attempting to use the $t$-test for distributions which are far from normal.
This experiment only considers the type I error, but experiments along similar lines could be used to compare type II errors between distributions.