Frank Harrell describes the concordance (or Somer's D) as not being sensitive enough to compare multiple survival models for their diagnostic ability, and I've observed this in my own work with multiple candidates providing very similar concordance scores. In this answer he provides an approach for comparing models with varying numbers of predictors.

However, what is the most effective evaluation criteria to use when you want to compare quite different models? I.e. a standard Cox model vs a gradient boosted one, or one using the raw predictors vs a model having transformed them.

The survAUC R package provides a multitude of ways to compare models, with the AUC approaches providing the largest difference between models in my cursory look, however there are several implementations to choose from.

In Classification one can use accuracy or AUC to aid model selection or MSE in regression problems, what would be the most useful measure for survival analysis?

  • $\begingroup$ Note that AUC gives the same information as concordance. See this page for example. Could you say a bit more about the purpose of your modeling? What is "effective" for one purpose might not be so "effective" for another. $\endgroup$ – EdM Nov 13 '15 at 18:03
  • $\begingroup$ In survival models I get different scores from the concordance and the AUC, for instance the different implementations of AUC in the survAUC package linked in the question provide different answers to the concordance. In addition, Heagerty's AUC calculates the AUC across survival probabilities at a specific time threshold, while the concordance is typically calculated of the linear predictors (i.e. time isn't considered) $\endgroup$ – Stuart Lacy Nov 16 '15 at 9:09
  • $\begingroup$ And the purpose is to build a model with the most predictive ability $\endgroup$ – Stuart Lacy Nov 16 '15 at 14:24

I think it's useful to consider transformations of predictors separately from choices of models. For transformations of predictors, the task is to find transformations that give well behaved residuals. That needs to be done regardless of how you then choose to combine those predictors into a model.

In terms of choosing among models you want the "most predictive ability." That needs to be a bit better defined, in terms of how the model will be used for prediction in practice. There may be different costs to different types of errors. Is the practical cost of under-estimating survival really the same as that of over-estimating survival? That assumption hides in the use of AUC or concordance as a measure of model quality, yet it seldom is true in practice. That may be even more of a reason to avoid such measures than is lack of sensitivity for discriminating among models.

There also may be costs in acquiring information about some predictor variables that would need to be taken into account in practice. Develop a metric that takes all the costs (monetary and prediction-error) and benefits into account. If you don't consider those explicitly, your analysis will be based on hidden assumptions that might be contrary to your ultimate goals.

To get the "most predictive ability" as thus re-defined, you need a model that will generalize well to the underlying population rather than simply fit the sample you have. So once you have developed your net-cost metric, choose the modeling scheme that minimizes that cost as evaluated by cross-validation or bootstrapping. This evaluation should include all steps in the model building. Then apply the best scheme to your data.

This approach isn't as simple as evaluating a pre-defined metric like AUC or AIC, but it should be more aligned with your ultimate goal of developing a generally useful predictive model. Finally, in case you've missed it, there is a new edition of Harrell's Regression Modeling strategies, which is a rich source of information and practical advice.

| cite | improve this answer | |
  • $\begingroup$ I appreciate the advice and can see the benefits of this approach. In classification one can build a group of models from different learning algorithms with their hyper-parameters optimised by cross-validation, using accuracy (or AUC) as a guiding metric. Why is it generally less straight forward in survival analysis? $\endgroup$ – Stuart Lacy Nov 17 '15 at 11:03
  • $\begingroup$ @StuartLacy I don't think that the underlying issues are much different between classification and survival analysis. It's just that the life-or-death types of choices implicit in survival analysis highlight the issue that different types of classification errors have different consequences. Whether the application is classification or estimating survival, using AUC as a metric implicitly assumes that all mis-classifications have the same cost. Similarly, if there are costs to acquiring predictors, those also should be considered in prospective classification schemes. $\endgroup$ – EdM Nov 17 '15 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.