When the null hypothesis is true, the p-value of a test should have the standard uniform distribution. Here is what I get with
t.test(...) in R using two Gaussian samples of size 5.
set.seed(123) p.val <- replicate(n=100000, t.test(rnorm(n=5), rnorm(n=5))$p.value) hist(p.val, breaks=50)
You can see that there is a deficit of low p-values. Below is what I get with somewhat bigger samples of size 10.
set.seed(123) p.val <- replicate(n=100000, t.test(rnorm(n=10), rnorm(n=10))$p.value) hist(p.val, breaks=50)
The deficit of low p-values is gone. So what happens in the first example? Is there something wrong with
t.test(...) in R for small-ish sample sizes?
> sessionInfo() R version 4.2.1 (2022-06-23) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.7 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib locale:  en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8 attached base packages:  stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached):  compiler_4.2.1