Concordance index in survival analysis (Gonen and Heller)

Question

I am working on a project to externally validate a clinical prediction model. The original model coefficients were estimated using a Cox model. The model uses the baseline hazard and coefficients to predict survival probability after cancer diagnosis.

I have come across the Gonen and Heller's Concordance Index for Cox models. It appears as if this only uses predictions from the model and does not consider the observed data (events and timings) at all. For example, see the GHCI() function in the R package survAUC:

https://search.r-project.org/CRAN/refmans/survAUC/html/GHCI.html

As one can see, this only has one argument, namely predictions from the model. I don’t understand how this can perform (external) validation if it does not consider the observed events or timings at all.

Answer 1

The Gönen-Heller modification of the c-index is based solely on the distribution of the (ordering of) predicted survival times. It thus assumes that you have a validated model. You can use that index in the process of external validation, as explained by Royston and Altman in "External validation of a Cox prognostic model: principles and methods," BMC Medical Research Methodology 2013, 13:33, but its use follows prior steps in validation. In the context of external validation, your interest is primarily in whether the discrimination is similar between the original data set and your validation data set.

As Royston and Altman say (p. 13):

the process of validation means assessing the performance of a predefined model in new data. It does not mean tinkering with the original model.

If you want to "tinker with the original model," then discrimination indices aren't the best way to compare your new model against the original model, as Frank Harrell notes.

This page and its links illustrate the Royston-Altman validation approach. If you have the Cox regression coefficients from the model you are evaluating, you start by seeing how well the linear predictor from that model works in your validation data set. Start by applying the model's linear predictor function to your data and use the resulting linear predictor as the sole predictor in a Cox model. Its coefficient should be very close to 1 if the model is well calibrated in your data.

You can also include the linear predictors from the model, applied to your data, as an offset (coefficient forced to be 1) in a Cox model that also includes the individual predictors of the model. The coefficients of the individual predictors then represent the differences from their values in the original model's linear predictor, and all should be close to 0 if the original model holds in your validation data. You also should re-evaluate the proportional hazards assumption.

It's only after those steps that Royston and Altman suggest to evaluate discrimination indices. Depending on the information available, you can further evaluate survival curves over time and their calibration at specific times of interest.

Response to comment on externally validating a competing risks model

You should be able to extend methods suggested by Royston and Altman to see how well a published Cox competing risks model works on an external data set. If you have values for all the covariates in the published model, calculate the linear predictor for all individuals for each of the competing risks. Royston and Altman call those "PI" values.

Build your own competing risks model with the external data, allowing for different baseline hazards and different regression coefficients for each event type. Then follow the Royston-Altman methods of "Regression on the PI," first using only the PI values for each of the risks as predictors. The regression coefficient on the PI should be close to 1 for each of the outcomes, and the baseline hazards should be close to what's reported in the published model (happily, they are given full functional forms in the model you want to validate).

If the slopes are different from 1, you can proceed to find out why with the methods described by Royston and Altman under "Check model misspecification/fit": use all the the predictors of the models in a new competing-risks Cox model while including the corresponding PI values as offsets forced to have slopes of 1. Do a "chunk test" combining all the predictor coefficients in the PI-offset model against the null hypothesis that all are 0. Predictors poorly modeled in the external data will have non-0 coefficients.

In this context, the Gönen-Heller index (which doesn't use the event times in the external data) is probably of secondary interest. As a discrimination measure, it essentially evaluates whether the distributions of outcome-associated predictor values are as wide in the external data as they were in the model you are validating. For applying C-indices to competing-risks models, see Wolbers et al., "Concordance for prognostic models with competing risks," Biostatistics 15: 526–539 (2014).

Answer 2

Keep in mind that most concordance indexes ($c$-statistic, etc.) use both X and Y and are measures of pure predictive discrimination. They are not sensitive enough for comparing models.

Once you know that a model is well calibrated (using a smooth calibration curve that accounts for overfitting) then predictive discrimination can come solely from the distribution of predicted values. A wide distribution corresponds to higher discrimination, and you can convert measures of dispersion to measures of explained variation. For example the numerator of ordinary $R^2$ is just the variance of predicted values. More information is available here.

Also, pseudo $R^2$ measures, since they come from the log-likelihood function, have a lot of advantages. See this.