1
$\begingroup$

Suppose you want to determine whether the means of two populations can reasonably be considered equal. You draw one sample from each population, and those samples have different bias-adjusted variance. Then you're computing the following test statistic: $$ T=\frac{\bar{X_1}-\bar{X_2}-(\mu_1-\mu_2)}{\sqrt{\frac{\sigma^2_1}{n_1}+\frac{\sigma^2_2}{n_2}}} $$ Where $\bar{X_1}$ and $\bar{X_2}$ are observed sample means; $\mu_1$and $\mu_2$ are unknown population means; $\sigma^2_1$ and $\sigma^2_2$ are Bessel-corrected sample variances; and $n_1$ and $n_2$ are sample sizes.

Why is it common practice to use $(min(n_1,n_2)-1)$ degrees of freedom for the test statistic? This is in particular what's taught to students in introductory statistics classes.

As far as I know, you should be running a Welch's t-test with a more complex approximation of the degrees of freedom.

I get the intuition that the amount of information you get is bottlenecked by the smaller sample, and I see that the resulting expression mirrors the degrees of freedom for computing a confidence interval on a single population mean, but I failed to find out where this $(min(n_1,n_2)-1)$ actually comes from.

Thanks in advance for your explanations.

ANSWER

Page 3 of the following document. $(min(n_1,n_2)-1)$ is a lower bound of the Welch–Satterthwaite approximation for the degrees of freedom. Thus, using it leads to sometimes underestimating the degrees of freedom but never overestimating them, so at least you're not going to underestimate the fatness of tails and attribute false power to your test.

$\endgroup$
4
  • 1
    $\begingroup$ Could you provide a source for this "common practice"? I have never seen it. $\endgroup$
    – whuber
    Feb 9 at 14:47
  • $\begingroup$ I have TA'd for two university professors who used it in undergrad courses. I asked one of them where this came from and he was honest about not actually knowing. So I assumed it was one of those approximations people use though they don't know why. Might have something to do with a given textbook... I'll have a look. $\endgroup$
    – Magean
    Feb 9 at 16:32
  • 1
    $\begingroup$ Thank you for the explanation It's a decent approximation to the Welch test--I can see why it might be taught to undergraduates. That helps me understand what "common practice" might mean in this context. $\endgroup$
    – whuber
    Feb 9 at 16:35
  • 2
    $\begingroup$ Thanks, I've eventually found a source in that direction. I'm going to answer myself. $\endgroup$
    – Magean
    Feb 9 at 17:10
2
$\begingroup$

The Welch 2-sample t test uses (approximate) degrees of freedom $\nu^\prime$ based on the two sample sizes $n_1, n_2$ and sample variances $S_1^2, S_2^2,$ respectively. Degrees of freedom in a pooled t test are simply $\nu = n_1 + n_2 - 2,$ but the formula for Welch's $\nu^\prime$ is a little messier.

One can show that $$\min{(n_1-1, n_2 -1)} \le \nu^\prime \le n_1 + n_2 - 2,$$

As a simplification, David Moore has observed in some of his elementary books (authored and co-authored) that if $n_1 > 30$ and $n_2 > 30,$ then $\nu^\prime \ge 30.$ [See one reference below.]

Thus, if both samples are of size greater than 30 and the absolute value of the Welch t statistic exceeds $c = 2.04,$ one can reject $H_0$ against a two-sided alternative at the 5% level.

qt(.975, 30)
[1] 2.042272

I have never seen use of $\nu^\prime = \min(n_1-1, n_2-1),$ in all cases, recommended as a 'general practice'. If you are using statistical software, procedures for the Welch 2-sample t test routinely base the P-value on the exact value of $\nu^\prime.$

In R, the following example uses $\nu^\prime = 14.492.$ Some other statistical software programs round $\nu^\prime$ down to the nearest integer.

set.seed(2021)
x1 = rnorm(10, 50, 7)
x2 = rnorm(10, 65, 9)
t.test(x1, x2)  # Welch t is default

        Welch Two Sample t-test

data:  x1 and x2
t = -4.1494, df = 14.492, p-value = 0.0009169
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -26.120658  -8.356365
sample estimates:
 mean of x mean of y 
 52.12437  69.36288 

Directly in R, using the (slightly rounded) t statistic, we have about the same P-value reported in the output above for the Welch t procedure.

2*pt(-4.1492, 14.492)
[1] 0.0009172653

Ref: DS Moore & GP McCabe, Basic Practice of Statistics, 3e (1999), Freeman, p541.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.