2
$\begingroup$

In reviewing the meaning of the Central Limit Theorem. While doing so, realize that the two-sample t-test’s test statistic has sample averages in it. If the representative samples of the patients being treated with each drug are very large, does this change any of the t-test assumptions?

$\endgroup$
3
  • 5
    $\begingroup$ Please note that the CLT says nothing about "very large" samples: it is purely a statement about a limit, with no implications about samples of any size. $\endgroup$
    – whuber
    Commented Nov 21, 2021 at 18:04
  • 2
    $\begingroup$ Even if sample sizes are large enough that the sample means are approx. normally distributed, the t-statistic has a t dist'n only if numerator [containing sample mean(s)] is normal and independent of denominator [containing sample variance(s)]. Stochastic independence of $\bar X$ and $S^2$ holds only for normal data. // In particular, if data are exponential, $n$ needs to be quite large for $\bar X$ not to have a noticeably skewed dist'n, and $\bar X$ and $S$ both estimate the mean ; hence they are correlated. // In gen'l, you mustn't assume t statistic is t-dist'd just because $n$'s $> 30.$ $\endgroup$
    – BruceET
    Commented Nov 21, 2021 at 20:04
  • 1
    $\begingroup$ Even if you're in a situation where the sample size is large and you're satisfied that the significance level was not too far off, you should still worry about power. If your sample is large because you're trying to pick up a small effect size (assuming you have a reasonable distributional model in mind for your variable), getting the actual significance-level close to your nominal $\alpha$ using a t-test isn't necessarily much use if you're throwing most of your power away (requiring a fairly large effect to be confident of seeing it). Better to use a good test for the situation you're in. $\endgroup$
    – Glen_b
    Commented Nov 22, 2021 at 0:00

1 Answer 1

1
$\begingroup$

Illustration of distinctly "non-t" distribution of one-sample "t statistics," resulting from samples of size $n = 100$ from an exponential distribution with mean $5.$ Simulation of 100,000 such statistics, using R.

set.seed(2021)
n = 100;  m = 10^5
MAT = matrix(rexp(m*n, 1/5), nrow=m)
a = rowMeans(MAT)
s = apply(MAT, 1, sd)
cor(a,s)
[1] 0.717151   # not consistent with indep

t = (a - 5)/(s/10)
hist(t, prob=T, col="skyblue2")
 curve(dt(x,88), add=T, col="orange", lwd=2)
 LU = quantile(t, c(.025,.975))
 abline(v = LU, col="blue", lwd=2, lty="dotted")

enter image description here

Quantiles $0.025$ and $0.975$ of the resulting distribution (blue histogram bars) are far from $\pm 1.984.$ Results from one-tailed tests would be especially problematic.

qt(c(.025,.975), 99)
[1] -1.984217  1.984217

With data from $\mathsf{Exp}(\mathrm{rate}=1/5),$ a one sided test of $H_0: \mu = 5$ nominally at the 5% level would have actual significance level either about $3.3\%$ or about $7.5\%,$ depending on sidedness.

set.seed(1121)
pv = replicate(10^5, t.test(rexp(100,1/5), mu=5, alt="g")$p.val)
mean(pv <= 0.05)
[1] 0.03202
pv = replicate(10^5, t.test(rexp(100,1/5), mu=5, alt="less")$p.val)
mean(pv <= 0.05)
[1] 0.07481

In the same circumstances, a two-sided test at intended $5\%$ level would a have a true level of about $6\%.$

pv = replicate(10^5, t.test(rexp(100,1/5), mu=5)$p.val)
mean(pv <= 0.05)
[1] 0.05936
$\endgroup$
1
  • 1
    $\begingroup$ The exponential is fairly mild, too -- it's easy to find distributions that are more extreme (e.g. a gamma with small shape parameter, say one below 0.1, a lognormal with a shape parameter much above 1, a Pareto with shape parameter around 4 or smaller). I've seen (on more than a few occasions) real data that look somewhat like those. $\endgroup$
    – Glen_b
    Commented Nov 21, 2021 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.