Suppose we have a population of patients. For each patient, we measure 1000 features and observe if they suffer from any of 10 diseases. We wish to determine which features, if any, are significant in predicting each disease.

Because of the multiple hypotheses (the 1000 features), we need to correct our $p$-values. For example, we might focus on the false discover rate and apply the Benjamini-Hochberg procedure

My question is: do we need to treat this as a single multiple hypothesis problem (1000 features $\times$ 10 diseases = 10,000 hypotheses), or can we treat it as 10 individual problems (each with 1000 hypotheses)? I'd certainly prefer the latter, since the $p$-value correction will suppress fewer terms.

It's pretty clear that to compute the family-wise error rate, we'd need to combine all the hypotheses. I was hoping that the FDR might behave differently, and allow me to analyze each disease independently.

More broadly, I would appreciate any pointers towards different statistical tool in case this general approach is off-base.


I'm treating Michael Lew's thoughtful answer below as correct, but I subsequently stumbled across a more (statistically) powerful tool for handling FDR. It seems to be very relevant for my problem. Anyone interested in this problem may find these papers helpful:


1 Answer 1


You are not testing hypotheses, but fishing for interesting findings. There is nothing wrong with that. Do not do things that are often demanded by those who cannot tell the difference between a preliminary scientific investigation and bad statistics. See my commentary on the ASA's statement on P-values for more detail. (It's in the supplementary material and takes a ridiculous amount of clicking to reach, so here is a direct link to a preprint of it I found online.)

'Correction' of P-value for multiplicity burns the power to detect real effects. Never do it unless you have no alternative, where no alternative comes from an inability to do any sort of follow-up and the absence of any corroboration from other data or theory. Do not dichotomise the results into 'significant' and 'not significant', but show all of the observed effect sizes. (I suspect this response will be voted down, but it is not wrong.)

Treat this as a preliminary investigation. Do not adjust the P-values, but rank the statistical 'interestingness' of the features in order of smallness of the P-values. Then follow up with a study designed to investigate just those features that are statistically and/or scientifically interesting. In this case I would say that even if no follow-up study is possible you should publish the raw P-values so that other investigators can use your data as corroboration for their findings.

  • $\begingroup$ Thank you for that link (which eventually lead me to this and several other interesting papers of yours). Your specific suggestion of sharing the raw P-values is definitely a good one. Your broader point about recognizing exploratory statistics is excellent. This question was motivated by a reviewer's request for corrected p-values; instead of providing that, I think I will try to echo some version of your points above. Thank you! (I am left a little mystified as to when one uses the FDR; the "discovery" part lead me to expect this case) $\endgroup$ Commented Nov 6, 2017 at 1:55
  • $\begingroup$ PS: I updated my question; if you're not already aware of them, you may find the linked papers interesting. $\endgroup$ Commented Nov 22, 2017 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.