25
$\begingroup$

When the null hypothesis is true, the p-value of a test should have the standard uniform distribution. Here is what I get with t.test(...) in R using two Gaussian samples of size 5.

set.seed(123)
p.val <- replicate(n=100000, t.test(rnorm(n=5), rnorm(n=5))$p.value)
hist(p.val, breaks=50)

You can see that there is a deficit of low p-values. Below is what I get with somewhat bigger samples of size 10.

set.seed(123)
p.val <- replicate(n=100000, t.test(rnorm(n=10), rnorm(n=10))$p.value)
hist(p.val, breaks=50)

The deficit of low p-values is gone. So what happens in the first example? Is there something wrong with t.test(...) in R for small-ish sample sizes?


> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.2.1
$\endgroup$
3
  • 2
    $\begingroup$ Cool question and a +1 from me! How does it in other software? Does Python/SciPy or SAS give a similar result? $\endgroup$
    – Dave
    Commented Oct 21, 2022 at 23:53
  • $\begingroup$ Here is a Python gist of this simulation. $\endgroup$
    – Galen
    Commented Oct 22, 2022 at 2:16
  • $\begingroup$ What do you consider "small" if 10 is only "small-ish"? $\endgroup$ Commented Oct 23, 2022 at 15:11

2 Answers 2

42
$\begingroup$

t.test performs Welch's t-test if the argument var.equal is not explicitly set to TRUE. The distribution of the test statistic (under the null hypothesis) in Welch's t-test is only approximated by a t-distribution and this approximation gets better with increasing sample sizes. Therefore, the result of your simulation is not particularly surprising.


Addendum
The test statistics in Welch's t-test and Student's t-test coincide if the two sample sizes $n_1$ and $n_2$ (of group $1$ and $2$, respectively) are equal. Hence, the discrepancy in the (simulated) p-value distributions of the two tests (note that the one of Student's t-test is uniform on $\left[0,1\right]$) under the null hypothesis is due to the discrepancy between the estimated degrees of freedom $\nu$ in Welch's t-test and the degrees of freedom $\tilde\nu=2\left(n-1\right)$ in Student's t-test, where $n=n_1=n_2$.

It is easy to see that, if $n_1=n_2$, the estimated degrees of freedom are given by $$ \nu = \frac{\left(n-1\right)\left(s_1^2 + s_2^2\right)^2}{s_1^4+s_2^4} = \frac{\left(n-1\right)\left(s_1^4 + s_2^4 + 2s_1^2s_2^2 \right)}{s_1^4+s_2^4}, $$ where $s_1$ and $s_2$ are the Bessel-corrected sample standard deviations.

By the AM–GM inequality and the non-negativity of sample standard deviations, $2s_1^2s_2^2 \leq s_1^4 + s_2^4$ (with equality only if $s_1 = s_2$) and $2s_1^2s_2^2 \geq 0$, therefore $n-1 \leq \nu \leq 2\left(n-1\right)=\tilde\nu$. This shows that the estimated degrees of freedom can only underestimate (or$-$but almost never$-$coincide with) the true degrees of freedom in the given situation, which leads to the conservative p-values seen in your simulation.
This behavior is nicely illustrated in Thomas Lumley's answer.

Since $s_1$ will tend to be closer to $s_2$ with increasing $n$, we can also see that $\nu$ will tend to be closer to $\tilde\nu$ as $n$ increases. Additionally, for a fixed difference $\nu - \tilde\nu$ in degrees of freedom of two t-distributions, their PDFs become increasingly similar with increasing $\nu$, and $\nu$ increases with $n$ in our case. This explains the improvement of the approximation and hence the p-value distribution with increasing group/total sample size.

$\endgroup$
5
  • $\begingroup$ Amazing, thanks for the clarification! I was wondering if you could point to a didactic source explaining the rationale of the df correction. Wikipedia is very laconic. That would be very useful for everyone. $\endgroup$
    – gui11aume
    Commented Oct 22, 2022 at 19:14
  • 2
    $\begingroup$ This link: statisticaloddsandends.wordpress.com/2020/07/03/… describes how Satterthwaite derived it, to match the variance of $\hat\sigma^2$. Welch derived it differently: jstor.org/stable/2332510 $\endgroup$ Commented Oct 22, 2022 at 21:28
  • $\begingroup$ @gui11aume One way to derive $\nu$ is to assume that the distribution of $\nu\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)/\left(\frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2}\right)$ can be approximated by a $\chi^2\left(\nu\right)$ distribution. Equate the two variances, solve for $\nu$, and replace population by sample variances. This goes back to Satterthwaite (1941, 1946) and is, e.g, detailed in this paper in the section "Derivation of the degrees of freedom formula". $\endgroup$
    – statmerkur
    Commented Oct 24, 2022 at 7:33
  • 1
    $\begingroup$ Thanks! I see the logic now. Quite intuitive actually. $\endgroup$
    – gui11aume
    Commented Oct 25, 2022 at 0:56
  • $\begingroup$ @gui11aume I'm glad I could help! $\endgroup$
    – statmerkur
    Commented Oct 26, 2022 at 13:26
23
$\begingroup$

Following up on @statmerkur's correct answer: first, here's what you get with var.equal=TRUE enter image description here

which is well calibrated. Second, here's the distribution of estimated degrees of freedom for the Welch t-test

enter image description here

As you can see, the estimated degrees of freedom are typically near 8, but occasionally quite a bit smaller. When the df is smaller, the p-value will be conservative.

Finally, here's the p-value distribution separately for estimated df>7 and $\leq 7$

enter image description here

$\endgroup$
2
  • 6
    $\begingroup$ Nice illustration (+1). I've added a theoretical justification for that to my answer. $\endgroup$
    – statmerkur
    Commented Oct 22, 2022 at 10:17
  • $\begingroup$ Here is a Python gist producing similar histograms. $\endgroup$
    – Galen
    Commented Oct 22, 2022 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.