1
$\begingroup$

I am new to survival analysis and currently very confused regarding a model selection process (forward stepwise) in cox proportional hazard model.

What happens is basically the following step by step:

  1. I identified n number of statistically significant independent variables, and none of them were correlated with each other.
  2. I wanted to put one last variable, which has the largest effect on AIC reduction out of all the variables.
  3. However, this variable is also very insignificant (p-value of parameter is about 0.88).
  4. I checked the correlation of this variable with other independent variables in the model and could not find anything.
  5. Plotted against all to see if I observe any type of nonlinear dependence, still no pattern.
  6. Tried to look at vif as suggested on the internet but all seem to be smaller than 2.

From here on I am not sure how to proceed. While my statistically highly significant variables are reducing my AIC by something between 4 and 10, this particular statistically insignificant variable (which was a variable I thought should be in the model definitely before building the model, and the parameter relationship is as expected intuitively) does reduce AIC by 250-300 (by far the most among all others), yet it does not come out significant.

Please guide me on what to do in this type of analysis. What statistical approaches should I take?

Not necessarily looking for a direct answer, so I'd appreciate if there is any article you can direct me as well.

$\endgroup$
  • $\begingroup$ Please show the output from your statistical software. A decrease of overall AIC by 250-300 should be significant. $\endgroup$ – Todd D Nov 12 at 14:40
1
$\begingroup$

Stepwise model selection, particularly forward stepwise, is not very reliable. This page provides much general discussion.

With Cox models the problem is even worse than for ordinary linear regression.

In ordinary linear regression there is no bias if you omit a predictor that is uncorrelated to the predictors you include. In other types of regressions like logistic regression or Cox proportional hazard regressions, however, if you omit a predictor that is associated with outcome your estimates of the coefficients for the included predictors are inherently biased. There is a wonderful analytic illustration for probit models on this page; the principle applies to Cox models too.

So both inter-correlations of predictor values and omission of predictors associated with outcome pose problems for Cox regressions. The coefficient for a predictor considered separately from all the other predictors can be very unreliable.

My guess is that your "statistically insignificant variable" is actually associated with outcome, as you suspected from the beginning. As a single predictor, though, its coefficient was biased to a low and "statistically insignificant" value because other predictors associated with outcome were omitted.

For unpenalized regressions (that is, not using LASSO, ridge, elastic net, etc.) a useful approach is to see how many predictors you can reasonably expect to handle without overfitting. For Cox models, that's on the order of 1 predictor for every 15 events. Within that limit, try to include all predictors that might reasonably be associated with outcome, based on your knowledge of the subject matter, and add in candidate predictors for any specific hypotheses you want to test. Then validate the model-building process with bootstrapping or cross-validation.

Frank Harrell's course notes provide a valuable resource for further study of regression modeling in general and Cox models in particular.

$\endgroup$
  • $\begingroup$ Thank you for extensive information on this issue. For now, I was thinking of going with: 1) Try Lasso or some sort of penalized regression 2) If not working, then add 1 predictor per 15 events ( which is then about ~60 variables and use maybe leave-one-out cv. Would you recommend this approach? $\endgroup$ – kukushkin Nov 13 at 13:55
  • 1
    $\begingroup$ @kukushkin leave-one-out CV can be unstable; 5- or 10-fold CV provides a good balance between stability and usefulness. Your choice depends on how you want to use your model. If it's for prediction, then ridge regression might be best; you keep information from all the predictors but penalize coefficients to avoid overfitting. If you want to cut down on the number of predictors, LASSO is reasonable. In either case you use CV to choose the penalty. If you want to see how new predictors improve over what's already known, then include them with known predictors in an unpenalized regression. $\endgroup$ – EdM Nov 13 at 15:29
  • $\begingroup$ I read exactly the same regarding ridge and lasso to determine which one to use, and will be going with LASSO. Regarding leave one out method, this was suggested by many for survival analysis however I didn't read or find anything regarding its stability. Do you mind expanding on that or maybe share if there is any very basic level explanatory article or textbook chapter I can read on? $\endgroup$ – kukushkin Nov 13 at 15:39
  • 1
    $\begingroup$ @kukushkin ISLR in Chapter 5 discusses cross-validation in general; see section 5.1.4 in particular for how a bias-variance tradeoff tends to favor 5- or 10-fold CV over leave-one-out. $\endgroup$ – EdM Nov 13 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.