Dealing with a statistically insignificant variable while AIC decreases when introduced in the survival analysis model selection?

Question

I am new to survival analysis and currently very confused regarding a model selection process (forward stepwise) in cox proportional hazard model.

What happens is basically the following step by step:

1. I identified n number of statistically significant independent variables, and none of them were correlated with each other.
2. I wanted to put one last variable, which has the largest effect on AIC reduction out of all the variables.
3. However, this variable is also very insignificant (p-value of parameter is about 0.88).
4. I checked the correlation of this variable with other independent variables in the model and could not find anything.
5. Plotted against all to see if I observe any type of nonlinear dependence, still no pattern.
6. Tried to look at vif as suggested on the internet but all seem to be smaller than 2.

From here on I am not sure how to proceed. While my statistically highly significant variables are reducing my AIC by something between 4 and 10, this particular statistically insignificant variable (which was a variable I thought should be in the model definitely before building the model, and the parameter relationship is as expected intuitively) does reduce AIC by 250-300 (by far the most among all others), yet it does not come out significant.

Please guide me on what to do in this type of analysis. What statistical approaches should I take?

Not necessarily looking for a direct answer, so I'd appreciate if there is any article you can direct me as well.

Answer 1

Stepwise model selection, particularly forward stepwise, is not very reliable. This page provides much general discussion.

With Cox models the problem is even worse than for ordinary linear regression.

In ordinary linear regression there is no bias if you omit a predictor that is uncorrelated to the predictors you include. In other types of regressions like logistic regression or Cox proportional hazard regressions, however, if you omit a predictor that is associated with outcome your estimates of the coefficients for the included predictors are inherently biased. There is a wonderful analytic illustration for probit models on this page; the principle applies to Cox models too.

So both inter-correlations of predictor values and omission of predictors associated with outcome pose problems for Cox regressions. The coefficient for a predictor considered separately from all the other predictors can be very unreliable.

My guess is that your "statistically insignificant variable" is actually associated with outcome, as you suspected from the beginning. As a single predictor, though, its coefficient was biased to a low and "statistically insignificant" value because other predictors associated with outcome were omitted.

For unpenalized regressions (that is, not using LASSO, ridge, elastic net, etc.) a useful approach is to see how many predictors you can reasonably expect to handle without overfitting. For Cox models, that's on the order of 1 predictor for every 15 events. Within that limit, try to include all predictors that might reasonably be associated with outcome, based on your knowledge of the subject matter, and add in candidate predictors for any specific hypotheses you want to test. Then validate the model-building process with bootstrapping or cross-validation.

Frank Harrell's course notes provide a valuable resource for further study of regression modeling in general and Cox models in particular.