Comparing non-nested cox regression models

Question

I'm trying to see if a new variable adding in a prediction model is better than the old model. Therefore I've made 3 cox regression models, based on backwards stepwise regression.

So I got models:

1. Variables X,Y
2. Variables Q,Z
3. Variables X,Y,Q,Z

To compare the additive value of Q and Z on top of X and Y, I had a look at the C-statistic, AIC and BIC. All of these indicators show model 3 is the best. Now I'm trying to see if model 3 is a statistically significant better prediction model than 1 and 2. For this; I've read in the literature that I should use a deLong's test. However, I did not manage to do this, because I cannot get the AUC's out of the cox-regression model.

To continue; is the way I see the above correct? Nex; how can I compare non-nested cox regression models?

Answer 1

I'm trying to see if model 3 is a statistically significant better prediction model than 1 and 2.

It doesn't have to be "a statistically significant better prediction model" for you to use it in preference to the others. Frank Harrell's course notes and book emphasize the futility of depending on p-values to decide on what predictors to keep in a predictive model. Unless the model is over-fit, you can (and often, should) keep "non-significant" predictors in your model to maintain performance. See Harrell's brief but highly informative answer here.

In your case you already have information about the superiority of model 3, via the AIC and its basis in likelihood. Harrell considers that a "gold standard" for predictive value. You also could do direct likelihood-ratio tests of model 3 against model 1 and of model 3 against model 2, as those are both nested comparisons.

There is some dispute about the use of AIC for non-nested models; see this page and its links for discussion. You might want to think about why you want to compare non-nested models directly in the first place. If you have a useful predictive model but need more parsimony in your number of predictors, simple backward elimination will lead to a set of nested models. You should only get non-nested models if you allowed a combination of backward and forward predictor selection.

The C-index is closely related to an AUC. Neither provides a very sensitive basis for comparing models.