Suppose I want to see whether the p-values generated by a certain parametric method are close to the “true” p-values. To avoid relying on any parametric assumptions, the latter are produced via bootstrap.

The lowest possible p-value that can be obtained from bootstrap is about $1/B$ where $B$ is the number of bootstrap samples. As a result, the bootstrap p-values have a lower bound, say $10^{-6}$, because using $B$ greater than $10^{6}$ would take too much time. On the other hand, the parametric method can easily produce much smaller p-values, $10^{-10} – 10^{-20}$. As a result, it's not very clear whether it is ok to use bootstrap as the source of “true” p-values in that range.

If you came across this problem before, please provide some references.

  • $\begingroup$ Your reference to "true" p-values suggests you might be thinking of assessing goodness of fit of a model, but that's purely a guess. Otherwise I cannot fathom what you might mean by such a thing. After all, if you think the best procedure to use is a parametric one, then use it and accept its p-value; if you think the best is a bootstrap, then use it and accept its p-value. The one thing you should not do is to run both procedures and subsequently pick the p-value according to some rule or preference: that guarantees the p-value you use is incorrect! $\endgroup$ – whuber Jan 20 '15 at 16:10
  • $\begingroup$ I think the problem I am trying to solve is a lot more technical and narrow: bootstrap imposes a low bound on the p-values and it gets in my way. Even if I were to use bootstrap only, I would still have to decide what to do if, say, 1/2 of all cases generate p-values at the lower bound. $\endgroup$ – James Jan 20 '15 at 17:36
  • $\begingroup$ What exactly is your question then? Would it be about the amount of variation exhibited in small p-values estimated via a simulation approximation to the bootstrap p-value? (Remember, the bootstrap p-value is definite: it is defined in terms of all possible bootstrap samples. Simulation is used only as a practical way to estimate the bootstrap p-value; its output not identical to the bootstrap p-value. For smallish datasets of size $n$, however, it still remains the case that the smallest possible nonzero p-value of $n^{-n}$ is finite and may seem pretty large.) $\endgroup$ – whuber Jan 20 '15 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.