3
$\begingroup$

If I have a study in which I run four regression models, each containing 5 independent variables, this would result in 5*4 = 20 p-values.

However, let's say the first variable is what I predict has an impact on my outcome and the other variables are simply there for adjusting for, and it is never my intention to report on these (say I want to know if previous bleeding results in a new bleeding and I want to adjust for age, sex, smoking and alcohol). I only want to report on "previous bleeding" but I am adjusting for age, sex, smoking and alcohol.

Assume I want to adjust using Bonferroni. Should I still adjust for 20 variables or would it suffice to adjust for 5, seeing as those were the original 5 I was going to report on?

$\endgroup$

1 Answer 1

2
$\begingroup$

First, correction for multiple comparisons is only needed for the particular hypotheses you are testing together at once. If you aren't testing hypotheses on some predictors then you don't need to include those predictor coefficients or their associated p-values in the correction.

Second, the Bonferroni-type control (for family-wise error rate, FWER) is done to protect you from finding any false positives when (a) a set of independent null hypotheses is evaluated together and (b) all those null hypotheses are true. If you are performing a multiple regression, the first and most important null hypothesis you test is a single test for the whole model.

If the null hypothesis doesn't hold for the whole regression model, then (a) it's likely that the null hypothesis doesn't hold for at least one of the predictor coefficients, and (b) the hypotheses on the predictor coefficients typically aren't independent. So Bonferroni-type FWER control in that context isn't appropriate.

In a clinical research study with a reasonably small number of predictors, you typically won't be expected to do such a correction among the individual regression coefficients from a "statistically significant" regression model. For studies involving large numbers of predictors like gene-expression levels, you would try to control the false-discovery rate (FDR) rather than FWER.

Finally, the Bonferroni correction is needlessly conservative; the Holm modification provides the same control over FWER with more power.

$\endgroup$
1
  • $\begingroup$ Great discussion, thank you,. $\endgroup$
    – Paze
    Commented Nov 19, 2020 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.