If the null hypothesis is correct, the distribution of the p-value should be uniform, by definition. I have come across two cases in my career where the results looked "too consistent" with the null...out of 50+ tests, we have most p-values $\geq 0.9$!
In cases such as this, I considered two possible explanations:
- Samples not truly independent
- Incorrect null distribution (if parametric)
Since p-values test for "extremeness" they are not very good at individually testing for an abundance of "closeness"....more than expected by chance.
However, first you need to see how often you'd get $0.02*n$ rejections if the base rejection rate were $0.05$. This is a binomial test.
It could also be bad samples (inconsistent samples) where they do not follow any consistent distribution...this will have unpredictable effects.
In the particular example, the sample sizes were actually quite moderate ($5 \leq n \leq 20)$ an the number of Monte Carlo runs were decent at $10,000$. However, the tests under consideration all involved large-sample approximations of one form or another, so my bets are on my second bullet here...they are really deriving approximate p-vaules.