<!-- language-all: lang-r -->
### About the t-test
With a t-test you are considering whether the means of two samples are significantly different or not.
For sufficiently large samples the value $d = \bar{x_1} - \bar{x_2}$ is approximately Gaussian distributed with some mean $\mu_d$ and $\sigma_d$ and the normalized/standardized value $z= (\bar{x_1} - \bar{x_2})/\sigma$ can be used to test the hypothesis that $\mu_d=0$. (and if the samples are drawn from a normal distribution, then $d$ will be exactly normal distributed)
The value $\sigma_d$ is often unknown and estimated based on the data giving an *estimate* $\hat\sigma_d$ of the standard deviation of the difference in sample means . So we do not really compute $z=(\bar{x_1} - \bar{x_2})/\sigma$ but instead $t=(\bar{x_1} - \bar{x_2})/\hat\sigma$. The distribution of $t$ does not follow a normal distribution like $z$. If the samples are normally distributed with equal standard deviation then $t$ will be following a t-distribution.
But if the samples are not normally distributed then this will not be the case.
However, if the sample size is large then the distribution will approximate a normal distribution a lot (because the distribution in $\hat\sigma$ is getting more narrow and the distribution in $d$ is getting more like a normal distributed variable). How large the sample needs to be depends on the way how the population distribution deviates from the normal distribution. If the population has large outliers then $d$ will not approach a normal distribution quickly. But if the population is more like a truncated distribution, then the sample will approach a normal distribution quickly.
### Example/demonstration
See the below demonstration for the difference in three different population distributions: the normal distribution, a uniform distribution (representing low kurtosis), and a t-distribution (representing high kurtosis). When you have samples of size 50 then there is hardly and difference.
In the image below you see the joint distribution of 1: the estimate of the pooled standard deviation 2: the difference in the means. The t-test draws two boundaries, in this case excluding 5% of the points on the left and right. These boundaries are diagonal. The values $-1.984 \hat\sigma < \bar{x}_1-\bar{x}_2 < 1.984 \hat\sigma$ are considered not significant (with 95% level).
[![comparing three different population distributions][1]][1]
In the above image you see that there is still some reasonable difference in the distribution of $\hat\sigma$, but in the distribution of the values $t=(\bar{x}_1-\bar{x}_2)/\hat\sigma$ there is hardly any difference as you can see below.
[![comparing three different population distributions][2]][2]
-------
So you need not worry much about using a t-test when the samples/population are not exactly normal distributed. The t-test is not very sensitive to deviations like these because with large samples the distribution of the sample mean is gonna approximate a normal distribution no matter what the underlying distribution is. And the distribution of the sample mean scaled by the sample standard deviation is gonna approximate a t-distribution/normal distribution as well because the error in the estimate of the standard deviation is gonna reduce for larger samples.
The exception is: 1 distribution with infinite variance or very high kurtosis such that they do not approach a normal distribution quickly. 2 when the sample size is small.
You can use simulations based on empirical distribution (or maybe you have theoretical consideration that allow you to make assumptions about the population distribution) to verify whether the approximation with a t-distribution make sense.
set.seed(1)
nt <- 10^4 # we do 10^4 trials
ns<- 50 # with samples of size 50
tm <- c()
for (j in 1:3) {
# generate samples in a matrix
if (j ==1) {
x <- matrix(rnorm(nt*ns,0,10),nt)
y <- matrix(rnorm(nt*ns,0,10),nt)
}
if (j == 2) {
x <- matrix(runif(nt*ns,-sqrt(300),sqrt(300)),nt)
y <- matrix(runif(nt*ns,-sqrt(300),sqrt(300)),nt)
}
if (j == 3) {
x <- matrix(rt(nt*ns,3),nt)*sqrt(100/3)
y <- matrix(rt(nt*ns,3),nt)*sqrt(100/3)
}
# compute statistics/estimates (means and pooled standard deviation)
xm <- rowMeans(x)
ym <- rowMeans(y)
s_est <- sapply(1:nt, FUN = function(i) sqrt(sd(x[i,])^2 + sd(y[i,])^2)/sqrt(ns))
# perform t-test
t_est <- sapply(1:nt, FUN = function(i) t.test(x[i,],y[i,])$p.value)
colours <- hsv(0,(t_est<=0.05),(t_est<=0.05),0.3)
# display in plot
plot(xm-ym,s_est, ylim = c(0,3), xlim = c(-10,10),
xlab = "sample mean difference",
ylab = "sample mean difference \n estimated standard deviation",
col = colours, bg = colours, pch = 21, cex =0.3)
sum((t_est<=0.05))/nt
# the limits used by the t-test
lines(c(0,10),c(0,10)/qt(0.975,ns*2-2))
lines(-c(0,10),c(0,10)/qt(0.975,ns*2-2))
tm <- rbind(tm,(xm-ym)/s_est)
title(c("samples from normal distribution",
"samples from uniform distribution",
"samples from t-distribution"
)[j],cex.main = 1)
}
h1 <- hist(tm[1,],breaks = seq(-10,10,0.2))
h2 <- hist(tm[2,],breaks = seq(-10,10,0.2))
h3 <- hist(tm[3,],breaks = seq(-10,10,0.2))
plot(h1$mids,h1$density, type = "l", xlim = c(-5,5), log = "y",
ylab = "log-density", xlab = "t - value",
yaxt = "n")
axis(2,at = c(0.001,0.01,0.1,1))
lines(h2$mids,h2$density)
lines(h3$mids,h3$density)
lines(h1$mids,dt(h1$mids,98), col =2)
title("comparison of \n emperical distributions(black)\n with t-distribution(red)", cex.main = 1)
### About the difference with the Mann Whitney U test
In your case the Mann-Whitney U test is significant and the t-test is not, even though the Mann-Whitney U test is generally considered to have a lower power (and hence will be less likely to turn significant if there is a difference).
The reason for this might be that you are dealing with a distribution that has a few large outliers that make the estimate of the variance very large and the t statistic very low.
It is a bit difficult to go change things afterwards (because it is a bit cherry picking to go look for whichever test gives the result that you desire/expect). But for the next time that you perform a test and you expect that these outliers are gonna make the t-test inaccurate, then you may either use the Mann-Whitney U test or find some way to deal with outliers (maybe it makes sense to use a different scale, or the average of the samples with outliers removed). Also, maybe you should not be comparing the means of the samples, but a different statistic makes more sense (e.g. the medium or some other values). The groups may differ on more aspects than just the mean.
Keep in mind that a Mann-Whitney U test is answering a *different* question than the question whether two populations have the same means or not.
[1]: https://i.sstatic.net/bEYSI.png
[2]: https://i.sstatic.net/imHML.png