# Is it good practice to adjust for multiple comparisons when performing different multivariable regression models?

I have five different outcomes and the same group of independent variables which have been used for the five different multivariable Cox regression models. The indipendent variables I have used in the multivariable models are part of a pretty standard choice of variables.

I used to think that multiple comparison adjustment (i.e. Bonferroni correction) was a standard procedure when performing univariate analyses but this could be skipped when performing multivariable models, especially when the choice of independent variables is pretty standard. However, by discussing this with a colleague, he pointed out that a multiple comparison adjustment should be performed anyway (I replied that a cross validation approach using two sub-sets would be a better way to prove the significance).

In my case I perform 5 different models (using five different outcomes) and 3 out of 5 are significant, but only one p value is <0.01 (the other two around 0.02). If applying the multiple comparison adjustment I should completely change my view on the results. What do you think?

• Could you please say more about the relations among the 5 different outcomes you are modeling? Are they mutually exclusive? Can any of them happen more than once? Significance tests with multiple outcome variables in regression models typically need to take the covariance structure of the outcomes and the regression coefficients associated with each outcome into account.
– EdM
Jul 8 '16 at 13:07
• The outcomes are quite related in a sense that they are different measures of the broader field Jul 8 '16 at 14:44
• I'm still a bit confused by what you mean by "different measures of the broader field." Is this some sort of meta-analysis (looking at 5 related studies, each with say overall survival as the outcome, but different populations), or is this the same set of subjects with their individual values of the multiple predictors associated in parallel with 5 different survival-type outcomes? Or is it something else? That might matter in terms of the best way to test significance, particularly if there is a single set of subjects with potentially related types of outcome variables.
– EdM
Jul 8 '16 at 20:44
• 5 health outcomes, e.g. hospital admission, hospital re-admission, clinical relapse.... Jul 9 '16 at 16:51

Firstly, when you perform multiple hypothesis tests (as you do by looking at whether p-values from multiple outcomes), there is in principle a multiplicity issue in the sense that comparing each p-value versus the level $\alpha$ will result in a familywise type I error rate $>\alpha$. I do not think this really goes away, if you do cross-validation. Whether you need to control the familywise type I error rate and across which analyses is a complicated issue. E.g. if you write two separate papers on the same dataset, do you get twice the familywise type I error rate, but not if you put the results in the same paper? This is really only (relatively) clear in a few settings such as confirmatory clinical trials for getting regulatory approval for a drug.

Secondly, the practical reason why many people are keen on adjustments is, because many people take a "many shots on target" approach, where they do lots of comparisons and then emphasize those with an unadjusted p-value <0.05. It is clear that when people study a lot of things including a huge number of things that really do not affect the outcomes being studied, that this will fill the scientific literature with many purported findings that are just random noise. This only gets worse when there are many small decisions left open until the data has been collected, which may lead to the potential for the choices in the analyses being data dependent. It may be debatable whether multiplicity adjustments help that much for such issues, but I guess I am not alone in trusting results with p<0.05 less when (a) the study was not pre-registered with outcomes and analyses pre-specified, (b) lots of outcomes were studied, (c) no adjustment was made for multiplicity and of course (d) the claimed effect is not a-priori plausible.

Thirdly, you should not completely change your view of results just because p=0.02 or p is just over 0.05. The former is not completely compelling evidence (and I would not get too excited about it) and the latter does not mean that the hypothesized effect is not there. Of course, this may affect what journal editors and reviewers will let you write (and whether they get excited about your paper) so in practical terms it may sadly be a major difference.

• My point was that my sample size was relatively large and I was already adjusting for all the covariates suggested by the literature. Moreover, the five outcomes are all different measures of the same macro-area. I felt safe with my approach (as rarely seen Bonferroni adjustment in multivariable regression models in this situation) but now, with your comments and also looking at some papers, I am starting to question it. Jul 8 '16 at 14:56

As I understand your situation, you have the same set of patients, each with a set of multiple predictor variables, for whom you are evaluating 5 different types of outcomes (hospital admission, hospital readmission, relapse, ...). In this case you probably do not really have independent outcomes so you need to be very cautious when you try to compare coefficients across regressions for each outcome separately. In a very important way, you do not really have 5 "different" Cox regressions, and the Bonferroni correction thus is inappropriate.

For example, a patient who relapses earlier is also likely to be readmitted earlier. When you analyze the problem as 5 separate regressions you are throwing away all of the information about the correlations among the different types of outcomes that are linked through individual patients. That might, in fact, make it harder to identify predictors that are truly related to outcome than if you considered some type of joint modeling of all the outcomes.

This type of modeling is beyond my expertise, but this paper is a recent and freely available paper that seems to be pretty closely related to the type of problem you face. It contains references to the earlier literature that might be helpful. For your project it might be wise to consult with a statistician who has some experience with this type of modeling.