My study has me running several regression analyses to investigate my hypothesis, and I am struggling with the multiple comparisons problem. I have three separate metabolites as predictor variables, and four other covariates in each model. (Running each metabolite in separate models is typical for these metabolites.) I am running both logistic regression analyses and linear regression for each metabolite. The outcome variable is the same for the respective sets of regression analyses (it was dichotomized for logistic regression and left continuous for the linear regression).

I've been through every note I've ever taken, all my books, and a lot of the internet, and I'm still not sure about the Bonferroni correction. I know it's probably a simple problem, but I'll be grateful for any and all help with it. Thanks!

  • $\begingroup$ In my understanding if the underlying dataset is the same, you should apply Bonferroni correction. $\endgroup$ Mar 8 '16 at 22:49
  • $\begingroup$ It's all the same dataset, yes. Thank you for being so clear and to the point. Almost everything I've read so far has been clear as mud. $\endgroup$
    – Sparks
    Mar 9 '16 at 5:13

You may be disappointed or relieved to hear that the answer is it depends. If your study is a preliminary exploratory study intended to determine which of the metabolites are worthy of following up, then there is no need to perform any correction for multiple comparisons. To do so robs you of power to find a real effect. On the other hand, if your experiment is designed to give a decisive result that you will be declaring without confirmatory studies, then a correction for multiple comparisons will reduce the risk of you being embarrassed by a false positive finding.

  • $\begingroup$ I'm relieved to see someone say that, because that was exactly what I was getting from all my notes and reading. I'm glad it wasn't just my lack of understanding. I will do the correction. Thanks so much! $\endgroup$
    – Sparks
    Mar 9 '16 at 5:20
  • $\begingroup$ @michael lew "...will reduce the risk of you being embarrassed by a false positive finding". Yes. At the cost of an increased probability of being embarrassed by a false negative finding! $\endgroup$
    – colin
    Mar 10 '16 at 22:24
  • 1
    $\begingroup$ @colin Absolutely true! Confirmatory experiments are by far a better way to gain confidence about the results than correcting for multiple comparisons. (At the cost of more effort and expense.) $\endgroup$ Mar 11 '16 at 4:00
  • $\begingroup$ @MichaelLew I'm inclined to agree with you both; however, a reviewer has specifically asked for the multiple comparisons correction so... Rock-Sparks-Hard Place $\endgroup$
    – Sparks
    Mar 11 '16 at 5:58
  • 2
    $\begingroup$ @Sparks understood. However, alternatively you could refer to this really nice paper by Andrew Gelman about why multiple comparison tests are usually unnecessary. arxiv.org/pdf/0907.2478.pdf $\endgroup$
    – colin
    Mar 11 '16 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.