# What does it mean if test was rejected less than in 5% cases?

I was reading a paper and authors performed multiple tests for Poisson distribution on single vectors. Large dataset of $n$ quite long vectors was used. The test was rejected only $0.02 \cdot n$ times under 0.05 level of significance.

I can imagine non-uniform distribution of p-values and I understand what means 10% of rejections under 5% significance (test's assumptions are not satisfied), but this case is different. Does it mean that suggested test statistic is asymptotically less than theoretical and test is not correct?

(I found test that was used)

If the null hypothesis is correct, the distribution of the p-value should be uniform, by definition. I have come across two cases in my career where the results looked "too consistent" with the null...out of 50+ tests, we have most p-values $\geq 0.9$!

In cases such as this, I considered two possible explanations:

• Samples not truly independent
• Incorrect null distribution (if parametric)

Since p-values test for "extremeness" they are not very good at individually testing for an abundance of "closeness"....more than expected by chance.

However, first you need to see how often you'd get $0.02*n$ rejections if the base rejection rate were $0.05$. This is a binomial test.

It could also be bad samples (inconsistent samples) where they do not follow any consistent distribution...this will have unpredictable effects.

In the particular example, the sample sizes were actually quite moderate ($5 \leq n \leq 20)$ an the number of Monte Carlo runs were decent at $10,000$. However, the tests under consideration all involved large-sample approximations of one form or another, so my bets are on my second bullet here...they are really deriving approximate p-vaules.

• Thank you! But incorrect null can only make p values worse...not better, cant they? I can not check result with binomial (actually story starts with nar.oxfordjournals.org/content/40/9/e69.full.pdf+html), but number of regions should be really big. – German Demidov Apr 11 '16 at 19:30
• @GermanDemidov when I say incorrect null....I mean incorrect null distribution of the sampling statistic....in the case of fitting poisson, your null may be asymptotically correct, but if the actual distribution of the null is not near asymptotic, then you are comparing your sample statistic to the incorrect reference distribution. – user75138 Apr 11 '16 at 19:33