Why is the p-value distribution not uniform?

Question

While learning on the q-value

http://www.totallab.com/products/samespots/support/faq/pq-values.aspx

I saw that, under the null-hypothesis, the distribution of the p-value is expected to be uniform.

I made up a simple t-test between two Gaussian samples, and do not understand the results:

If one sample is fixed and the other is random, then the p-value distribution is not uniform:

R
s0 = rnorm( n = 100, mean=0, sd=1 )
vec = c()
for( i in 1:10000 ){
s1 = rnorm( n = 100, mean=0, sd=1 )
t = t.test( x=s0, y=s1 )
p = t$p.value
vec = c( vec, p )
}
hist(vec, breaks=seq(0,1,0.01))

But if the two samples are random, then the distribution is uniform:

R
vec = c()
for( i in 1:10000 ){
s0 = rnorm( n = 100, mean=0, sd=1 )
s1 = rnorm( n = 100, mean=0, sd=1 )
t = t.test( x=s0, y=s1 )
p = t$p.value
vec = c( vec, p )
}
hist(vec, breaks=seq(0,1,0.01))

I don't catch here the link with the "null-hypothesis", any enlightening explanation is welcome!

Answer 1

When one sample is fixed (as in the first set of code), the Null hypothesis is not true. This happens because, in the first set of code, s0 will have some mean that does not exactly equal 0. Because the normal distribution is continuous, the probability of drawing 100 i.i.d. normals with a mean of exactly 0 is 0. Therefore, the loops in the first set of code all involve t-tests where the Null hypothesis is false because they involve comparing $\mu_1$ to $\mu_2$ where $\mu_1\neq0$ and $\mu_2=0$. Because the Null is false, the distribution of p-values won't be exactly uniform.

Answer 2

The difference is attributable to sample size. Examining your first R code block, s0 is assigned only 100 draws in a single run whereas s1 is assigned 100 values over 10000 runs. With a large sample of normally distributed values, any small sample will naturally be skewed. You can verify this by gradually increasing n. I did this and at about 30000 the results smoothed out, becoming essentially indistinguishable from the second R code block.