Explain non-uniform p-values for small sample t-tests in R

Question

This R script produces a deficit of p-values < 0.1 for small sample sizes of n < 10. Can anyone explain why this is the case? If this is an R bug it could have some impact. Note: One-sample z-tests do have uniform p-values for small sample sizes.

m <- 10000
n <- 3
p.value <- rep(m, 0)

for (i in 1:m) {
  x <- rnorm(n, mean= 0, sd= 1)    
  y <- rnorm(n, mean= 0, sd= 1)    

  p.value[i] <- t.test(x, y, alt= "two.sided")$p.value 
} # for

par(mfrow= c(1,1))
hist(p.value, breaks= 20, xaxs="i", yaxs="i", col="skyblue", prob= F, las=T)   # appears uniform

abline(h= m/20, col= "red",  lty=2)
abline(v= 0.05, col= "cornflowerblue")

Answer 1

This isn't a bug in R.

Welch-Satterthwaite type t-tests (the default two sample t-test in R) don't actually have a t-distribution.

The t-with-fractional-d.f. you get is an approximation to the null distribution.

The Welch-Satterthwaite tests work well in a variety of situations, but even when all the assumptions hold the null distribution of p-values will be somewhat non-uniform (this will impact significance levels; you won't have quite the significance level you were aiming for).

There are effectively 3 parameters that control the null distribution -- the ratio of population variances, and the two sample sizes. The test uses an approximation to make it just a function of a single parameter (the Welch-Satterthwaite d.f.).

For some choices of variance ratio and sample-size ratio the distribution of p-values will tend to be somewhat skewed to lower values and for other choices it will tend to be skewed a bit to higher values.

This will tend to be more noticeable at small sample sizes, but occurs quite generally.

It's possible to use simulation at your specific n's and variance ratio rather than the t-approximation to get better control of significance levels and so more accurate p-values, if that's necessary. However, if your sample sizes are equal (as it looks like they are in your simulation), an equal-variance t-test has little problem with control of significance level even when the variances are unequal, so that may actually be a reasonable default choice when you have equal sample sizes.