3
$\begingroup$

I have a list of 300 $p$ values selected from all $p$ values obtained in multiple testing such that $p<.05$. I do not know the values of $p>.05$ or their number. The $p$ values are not corrected for multiple testing, and I would like to do so for the 300 raw $p$ values that I have.

Is it OK to apply the Benjamini-Hochberg correction method, for instance with p.adjust() in R, on my list of $p$ values even though they have been pre-selected for being less than $.05$? If not, is there another method that can be used?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

No, that is not ok. The FDR looks at the entire distribution of all your p-values to make the correction. If it does not see the right side of the distribution it will give much too optimistic estimates.

Options to still use Benjamini-Hochberg:

  1. Replace all the missing p-values with 1. This would be conservative.
  2. Use multiple imputation where you repeatedly impute the missing p-values with values between 0.05 and 1. This would also be acceptable, since conditional on the p-value being larger than 0.05 and the null hypothesis being true for all of them (conservative assumption), this would be the actual distribution.

An alternative is a less powerful method like Bonferroni or Bonferroni-Holm. In both cases as in the method described above you will still need to know the number of missing p-values - or at least an upper bound for that. Note that the $n$ in the correction formulas would be the total number of p-values (your 300 + the number of p-values that are missing).

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Just a comment to emphasize that, for any of these methods, you need to know the total number of p values and the author said he/she does not know that $\endgroup$
    – Peter Flom
    Commented Jan 9, 2014 at 11:55
  • $\begingroup$ True - I mentioned that he needs them but managed to overread the part where he wrote that he doed not have that. Thanks for the helpful comment. Still, some effort might be made to attain an upper bound. $\endgroup$
    – Erik
    Commented Jan 9, 2014 at 12:13
1
$\begingroup$

All reasonable multiple testing procedures necessarily need the number of tests. Multiplicity correction always depends on the number of (possibly true) Null hypotheses tested: The more hypotheses, the lower your local significance level must be to counter type I error inflation and in order to meet your global FDR or FWER.

Therefore there is no possible solution for your task, unless you find some upper bound for the number of tests that have possibly done. Then you can proceed as Erik answered.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.