0
$\begingroup$

I have a model for 2 year mortality which was derived using binomial regression.

Now I want to create a time-to-event model with Cox and I want to compare the performance to the already available binomial model at 2 years.

Now I know there are time-dependent AUC methods for Cox (e.g. incident/dynamic AUC by Heagerty et al.). But if I want to compare just the AUC at 2 years, is it valid to simply use cox linear predictors, set all events within 2 years to 1, the remaining patients to 0 and calculate a regular AUC? If it's not ok for some reason, what would be the best way to compare the models?

Edit: And vice versa, would it be ok to use linear predictors from the binomial model to calculate Harrell's C to compare to the cox model?

$\endgroup$
2
  • $\begingroup$ In the binomial regression model, how did you deal with individuals who didn't die but weren't followed up for a full 2 years? $\endgroup$
    – EdM
    Jan 24, 2021 at 20:23
  • $\begingroup$ Set to 0 (not dead). That's why I think Cox should perform better here. $\endgroup$
    – StanW
    Jan 26, 2021 at 5:32

1 Answer 1

1
$\begingroup$

First, you'll be better off using a more sensitive measure than ROC-AUC to compare model types. See this page, for example. Harrell notes there (and elsewhere) that AUC (equivalent to the C-index and directly related to Somers' $D_{xy}$) is OK for describing a single model but isn't sensitive enough to distinguish quality between models. I appreciate (and have used) the ways to incorporate censored survival data into ROC curves that Heagerty and colleagues have developed, but those won't be any more useful for comparing between models.

It's most helpful to examine and compare detailed validation and calibration of the models. The R rms package provides general functions that can be applied to a wide variety of models including binomial/logistic and Cox proportional-hazards regressions at a specified survival time. These functions use resampling to estimate optimism in the fit and the quality of predictions both overall (validate()) and as a function of predicted values (calibrate()).

The resampling used by both validate() and calibrate() helps to evaluate model overfitting and thus how well the models might be expected to perform on new data samples. So if you want to use the $D_{xy}$/C-index to compare the models, at least use the optimism-corrected values returned by validate(). Other reported measures that could be more sensitive to differences between models are those related to the overall calibration in terms of expected versus observed probabilities (discrimination index, unreliability index, and their combination in an overall quality index).

The most sensitive comparison would tend to be the calibration curves provided by the calibrate() function, as they give not only overall estimates of reliability but also displays of model performance over the range of outcomes. Depending on how you intend to use the models, particular regions of those curves might be more important than an overall estimate of reliability.

Second, you have a potentially big problem with your binomial model that might not show up in comparison against the Cox model. You coded those who were still alive but followed up for less than 2 years the same as those who survived a full 2 years. So you're not really modeling 2-year survival; it's whether there was either 2-year survival or any survival less than 2 years without evidence of death. What if some of your predictors are associated specifically with those surviving at less than 2 years' follow-up? Then your model might predict well that combined class of survivors and look very good on your data as coded, but the model wouldn't properly be predicting 2-year survival. So a high AUC in that circumstance might be technically correct but clinically meaningless.

For binomial/logistic regressions, it's better to treat cases without complete follow up as having missing outcomes, and use multiple imputation to fill in the outcomes. Stef van Buuren's book goes through the rationale, including imputation of outcomes. If data are "missing at random" (as distinguished from "missing completely at random") in the following technical sense, multiple imputation is a well respected and valid approach:

If the probability of being missing is the same only within groups defined by the observed data, then the data are missing at random (MAR).

Note that MAR could describe a situation in which your predictors are associated specifically with those surviving at less than 2 years' follow-up. So multiple imputation could overcome that problem with your data, while providing a valid model of 2-year survival.

$\endgroup$
7
  • $\begingroup$ Thanks for pointing out the problem about 2 year mortality. I wouldn't call it a "big" problem though, as it depends on how many cases were there and other factors, such as competing risks vs. loss to follow up. Lets assume we have set those to missing and imputed. I think in this setting then, at least C-index seems more reliable than AUC. So is there a) an argument why the linear predictor from the binomial regression can't be used to calculate Harrell's C to compare to Cox, b) another readout for better comparison. Which readout from validate() would you use to compare the models? $\endgroup$
    – StanW
    Jan 27, 2021 at 12:07
  • $\begingroup$ For binomial/logistic regression, AUC and C-index are equivalent and represent a rank correlation. The validate() function for such models uses the linear predictor as you suggest, but reports Somers' Dxy (from which you can calculate the C-index). For Cox models, the C-index is the proportion of cases for which the predicted and actual order of events is correct. (Don't know how well that corresponds to the AUC for time-dependent ROC in survival models.) $\endgroup$
    – EdM
    Jan 27, 2021 at 17:33
  • $\begingroup$ @StanW I've edited the answer to provide more details about how to compare validate() and calibrate() results between models. That now comes just before my warning about the "potentially big" 2-year mortality issue. As you note, the details of a data set will determine how serious that might be, but it's something that can get one into trouble all too easily if it's not always on your mind. $\endgroup$
    – EdM
    Jan 27, 2021 at 18:06
  • $\begingroup$ thanks for the answers, though the issue about C-index and AUC for binomial/logistic regression is quite clear. Maybe I wasn't clear in my question on that. What I am wondering is, whether it is ok to input the linear predictor from binomial regression into the Harrell's C-index calculation (same calculation as for Cox). $\endgroup$
    – StanW
    Jan 27, 2021 at 19:37
  • $\begingroup$ All the formula needs is a predictor, time and status (it's different from the C-statistic normally used for binomial regression because the latter does not take time into account). This way you calculate the same formula for Harrell's C, once with the predictor from binomial once with cox and could potentially directly compare the models in this way? $\endgroup$
    – StanW
    Jan 27, 2021 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.