2
$\begingroup$

My initial feeling was never. But I notice that whenever people write up a larger study (or a meta-study) they do not adjust the significance thresholds of the individual measurements to account for multiple comparison.

I am assuming the formal justification for this is that whenever you are trying to "tell a story" or "set up a theory" all individual hypotheses are never fully independent, but different degrees of related.

On the example of a psychotropic drug test, I am curious where one would draw the line (if at all) beyond which multiple comparison correction is no longer needed:

  • Multiple measures for the same psychotropic effect - which in literature have been shown to reliably predict each other
  • Multiple measures for different psychotropic effects - that are, however, likely to be underpinned by the same neuronal structure.
  • Multiple measures for different psychotropic effects - likely to be underpinned by separate neuronal sturctures.
  • Multiple measures of physiological parameters likely to be influenced by the drug.
  • Multiple measures of physiological parameters unlikely to be influenced by the drug.
Improve this question
$\endgroup$
1
  • $\begingroup$ maybe when there are random effects? Although arguably these are already mean and variance corrected, so leaving the means alone and correcting the variances as in multiple comparisons correction is superfluous, and/or inefficient. $\endgroup$
    – conjugateprior
    Commented Apr 12, 2016 at 3:09

1 Answer 1

Reset to default
1
$\begingroup$

In theory, this is an easy question. In practice, it is really, really hard.

In theory, we should only use multiple comparisons methods when we are worried about making too many Type I errors. We should not use multiple comparison methods if are worried about making too many Type II errors.

For example, suppose we are screening to see if any genes are associated with a particular disease. Well, since there's about 20,000 genes, if you we don't use multiple comparison methods, we would expect to get about 1,000 genes showing up as significant, even if no genes were associated with the disease. As such, it would be a big waste of everyone's time to say "Hey, look, I've found that these 987 genes are significantly related to the disease!" (ie the very real cost of Type I errors with these studies is that they waste everyone's time trying to reproduce your results).

On the other hand, suppose Type II errors were of more concern. For example, suppose you're evaluating potential terrorist threats. Of course, there is a very real Type I error cost (false accusation), but there's also a very significant Type II error cost (not catching someone in time). Now this becomes a difficult problem, but you can see right away that just saying "I'm doing multiple comparisons, therefor I must use multiple comparison adjustments" is a really bad idea.

In the cases you have presented, I would say that none of them are really clear cut. In particular, you appear to doing a screening, rather than a necessarily confirmatory analysis. As such, I would somewhat discourage multiple comparisons: you should see it a little more as an exploratory analysis. As such, in a perfect world I would recommend unadjusted confidence intervals, with a reminder to the reader that these are unadjusted.

Very unfortunately, reviewers are likely to require p-values for publication. And sadly, which types of studies they will require you to use multiple comparison methods is very dependent on which reviewer you get.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.