Variable selection for Cox regression repeated for multiple covariates of interest

Question

I am doing a retrospective analysis of the effect of various measures of haemodynamics in sepsis on mortality. I will separately look at the effect of 5 independent variables: 1) shock index 2) blood pressure 3) heart rate 4) arterial BP 5) non invasive BP

I plan to do use Cox regression analysis to calculate the odds ratio of each of the five above variables with respect to mortality. I will repeat my method 5 times, once for each of the 5 above independent variables. Each time, I will take 1 of the above independent variables and several other covariates to be adjusted for (ie. age, sex, source of admission, mean glucose, presence of positive blood culture, lactate, presence of CKD, hepatic failure, cancer, heart failure and diabetes). All will be included in the Cox regression model. Then, I will optimise the model by stepwise removing the covariate with the highest p-value, until all p-values for each covariate are p<0.2. The number I'm interested in will be the odds ratio of that independent variable (shock index, blood pressure, heart rate, arterial BP or non invasive BP) with respect to mortality.

Now, my problem is that each of the five calculations of my odds ratio, a different set of covariates is included. (EDIT: different covariates are included because each of the five independent variables interact different with the other covariates changing their multivariate significance and hence which ones are removed in the stepwise feature selection process). I think this creates bias and makes it difficult to compare the effect size of each of my 5 independent variables of interest. For example, overfitting will effect each of my 5 effect sizes differently.

Is this something I can avoid? Would it be better to not optimise my model each time? Or would there be some other way to ensure that my list of covariates are the same on each of the 5 repetitions?

Answer 1

Looking separately at predictors and omitting predictors likely to be related to outcome are not generally good ideas in Cox survival modeling. Seriously consider a different approach.

Even in linear regression, omitting a predictor that is correlated both with the included predictors and with outcome leads to omitted-variable bias in the regression coefficients. In logistic or Cox proportional-hazards regressions, omitting a variable related to outcome can lead to bias even if it isn't correlated with the included predictors; see this page and this page, among others. So there is seldom anything to be gained by removing predictors expected to be related to outcome from a model based on their p values, particularly if prediction is the goal; see this page for some discussion.

In your case the 5 predictors of main interest are highly correlated with each other and with outcome: you have 3 measures of blood pressure, the heart rate (often inversely related to blood pressure), and a shock index that is evidently the ratio of heart rate to systolic blood pressure. Perhaps you wish to determine which of these is "best" in some sense, but if values of all are going to be available there is no reason to omit any of them from a predictive model.

Much depends on the size of your study, in terms of the numbers of deaths for mortality studies. The usual rule of thumb to avoid overfitting is about 15 deaths per predictor that is to be evaluated in a Cox model. If your study is large enough on that basis to include all of the covariates, then do so. Some "limited" backward selection might be OK "if parsimony is more important than accuracy," as Harrell's course notes put it in Chapter 4. Setting a cutoff at p = 0.2 isn't necessarily limited.

If the study isn't that large, consider an approach like ridge regression, which allows inclusion of all predictors by penalizing their regression coefficients to minimize overfitting. With one or more particular predictors in mind, you could consider penalizing all of the covariates except for the predictor(s) of interest, as in this paper.

If you really need to do 5 separate models, one for each of your main predictors of interest, then apply the above principles so that the covariates included are the same in all cases, penalized as necessary to avoid overfitting. Otherwise, the differences in included covariates are more likely to represent the vagaries of your particular data sample than anything that will generalize well to other data sets. And for that type of comparison you would be best off using bootstrapping to evaluate the quality of each model and the differences among them.