2
$\begingroup$

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with the exact value being unknown (this is the case for 11 out of 30 of my hypothesis tests). Now, my question: is the FDR correction (specifically the Benjamini Hochberg linear step up procedure) valid in this situation where the exact values aren't available for some p-values? In other words, if I have to pretend that my p-values are 0 for those that are < 0.0001, does this invalidate the correction?

$\endgroup$
2
$\begingroup$

Yes, saying your $p$-value is zero is a problem. But you can say it's $1/(B+1)$ (eg, 1/10001).

For some resampling-based tests that's always the right thing to do anyway, and it's not going to do any harm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.