# Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here).

I ran the test on 1000 simulated datasets (simulated under the null hypothesis of no effect). The smallest p value I got was 0.45, giving my procedure a type 1 error rate of <0.001.

Is this a contradiction? Shouldn't I find p<0.05 in about 5% of the datasets?

• Please show the null distribution of the test statistic and give your sample value. It would be useful if you can explain something about the statistic itself so that it's possible to work out if your results make sense or if they might point to an error. Mar 30, 2015 at 6:06
• You can also show us the qq plot of your p-values (they should be uniformly distributed between 0 and 1). Mar 30, 2015 at 8:47
• Which software package/random number generator are you using? Some, like the old rand() in C are terrible. Mar 30, 2015 at 10:42
• @Elvis that is often true, but not always. For example, in a one-sided test the $p$-value is only uniformly distributed when the true value is on the boundary of the range covered by $H_0$. All other true values that still satisfy $H_0$ will lead to a distribution of $p$-values that deviate from the uniform distribution. Mar 30, 2015 at 11:56
• @MaartenBuis If $H_0$ is a composite hypothesis, yes, you’re right. But I would rather consider that in this case, the distribution of the test statistics is not completely specified, and neither is the $p$-value distribution... matter of point of view. Mar 30, 2015 at 12:13

We can be more precise: if your test is performing as expected then the probability of finding a 1,000 samples from a population where the null hypothesis is true with a $p$-value more than .05 is $.95^{1000} \approx 5.3 \cdot 10^{-23}$. That chance is not zero, but pretty small (that is an understatement...).

If this were happening in a project of mine, my first step would be to assume that I made an error in implementing this simulation.

• Right. Moreover Hoeffding's inequality says there's prob. at most $\exp(-2 (50 - 10)^2/1000) = 0.041$ to see less than 10 significant results.. Mar 30, 2015 at 10:57
• @TrynnaDoStat It is a simulation study, so these tests are the same test performed in a 1,000 different samples. Mar 30, 2015 at 11:48
• @MaartenBuis I know, I just wanted to add that little bit of information. I'll clarify my comment. Mar 30, 2015 at 11:49
• Since you are simulating different samples, your p-values are independent and this answer is correct. Be careful though, when performing hypothesis test on the same data your p-values are likely not independent and this does not generalize. Mar 30, 2015 at 11:51
• @TrynnaDoStat what do you mean? bootstrap simulating different samples from the same data? Mar 30, 2015 at 16:16

Yes, you should find $p < 0.05$ in about 5% of the datasets. Two things to consider:

1. The P-value is a random variable, so it's entirely possible (if improbable) that you didn't get any significant P's just by chance.

2. More likely, you have a bug in your code.

• +1 If the distribution of the test statistic is highly discrete (i.e. most samples under the null produce one of only a few distinct values), it might happen, but it's usually obvious if this is happening. Mar 30, 2015 at 6:08
• Along the lines of @Glen_b's comment, I have a small quibble with "you should find p<0.05 in about 5%". A test is defined as being an $\alpha$ test if its size $\text{sup}_{\theta \in \Theta_0} P_\theta(X \in \{x:T(x)>c\})=\alpha$, so any test with a size of less than $\alpha$ is a valid level $\alpha$ test. So, say, you never reject the null, the test is still valid at, say, $0.05$.
– Momo
Mar 30, 2015 at 11:05