In reviewing the meaning of the Central Limit Theorem. While doing so, realize that the two-sample t-test’s test statistic has sample averages in it. If the representative samples of the patients being treated with each drug are very large, does this change any of the t-test assumptions?
$\begingroup$
$\endgroup$
3
-
5$\begingroup$ Please note that the CLT says nothing about "very large" samples: it is purely a statement about a limit, with no implications about samples of any size. $\endgroup$– whuber ♦Nov 21, 2021 at 18:04
-
2$\begingroup$ Even if sample sizes are large enough that the sample means are approx. normally distributed, the t-statistic has a t dist'n only if numerator [containing sample mean(s)] is normal and independent of denominator [containing sample variance(s)]. Stochastic independence of $\bar X$ and $S^2$ holds only for normal data. // In particular, if data are exponential, $n$ needs to be quite large for $\bar X$ not to have a noticeably skewed dist'n, and $\bar X$ and $S$ both estimate the mean ; hence they are correlated. // In gen'l, you mustn't assume t statistic is t-dist'd just because $n$'s $> 30.$ $\endgroup$– BruceETNov 21, 2021 at 20:04
-
1$\begingroup$ Even if you're in a situation where the sample size is large and you're satisfied that the significance level was not too far off, you should still worry about power. If your sample is large because you're trying to pick up a small effect size (assuming you have a reasonable distributional model in mind for your variable), getting the actual significance-level close to your nominal $\alpha$ using a t-test isn't necessarily much use if you're throwing most of your power away (requiring a fairly large effect to be confident of seeing it). Better to use a good test for the situation you're in. $\endgroup$– Glen_bNov 22, 2021 at 0:00
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Illustration of distinctly "non-t" distribution of one-sample "t statistics," resulting from samples of size $n = 100$ from an exponential distribution with mean $5.$ Simulation of 100,000 such statistics, using R.
set.seed(2021)
n = 100; m = 10^5
MAT = matrix(rexp(m*n, 1/5), nrow=m)
a = rowMeans(MAT)
s = apply(MAT, 1, sd)
cor(a,s)
[1] 0.717151 # not consistent with indep
t = (a - 5)/(s/10)
hist(t, prob=T, col="skyblue2")
curve(dt(x,88), add=T, col="orange", lwd=2)
LU = quantile(t, c(.025,.975))
abline(v = LU, col="blue", lwd=2, lty="dotted")
Quantiles $0.025$ and $0.975$ of the resulting distribution (blue histogram bars) are far from $\pm 1.984.$ Results from one-tailed tests would be especially problematic.
qt(c(.025,.975), 99)
[1] -1.984217 1.984217
With data from $\mathsf{Exp}(\mathrm{rate}=1/5),$ a one sided test of $H_0: \mu = 5$ nominally at the 5% level would have actual significance level either about $3.3\%$ or about $7.5\%,$ depending on sidedness.
set.seed(1121)
pv = replicate(10^5, t.test(rexp(100,1/5), mu=5, alt="g")$p.val)
mean(pv <= 0.05)
[1] 0.03202
pv = replicate(10^5, t.test(rexp(100,1/5), mu=5, alt="less")$p.val)
mean(pv <= 0.05)
[1] 0.07481
In the same circumstances, a two-sided test at intended $5\%$ level would a have a true level of about $6\%.$
pv = replicate(10^5, t.test(rexp(100,1/5), mu=5)$p.val)
mean(pv <= 0.05)
[1] 0.05936
-
1$\begingroup$ The exponential is fairly mild, too -- it's easy to find distributions that are more extreme (e.g. a gamma with small shape parameter, say one below 0.1, a lognormal with a shape parameter much above 1, a Pareto with shape parameter around 4 or smaller). I've seen (on more than a few occasions) real data that look somewhat like those. $\endgroup$– Glen_bNov 21, 2021 at 23:57