Statistical test to compare models results for concordance index

Question

I'm working on a model for survival prediction and using the concordance index to evaluate the results (https://medium.com/analytics-vidhya/concordance-index-72298c11eac7) . I want to show that my model is better than a baseline model using my test set. My understanding is that for a typical test metric (ie prediction of housing prices for example ) one could use the Wilcoxson signed rank test to see if my model is statistically significantly better than a baseline model. However, in this case, the concordance metric doesn't have any meaning for a single sample-- it's a description of how well the model can discriminate ordering amongst a set of inputs. Therefore, is doing the following valid: divide test set into batches, get each model's prediction on each batch, and run the wilcoxon signed-rank test on the predictions on the batches? Perhaps also vary batch size and shuffle the samples and run the test again to verify the result still holds?

Clarifications: I should have clarified that I'm not use the Harrel C-index but the adjusted Antolini index. I know the Harrel C-index is not often used anymore. The baseline model is a machine learning model that provides decent values. My model is also the same type of machine learning model but trained differently . The two models have the exact same architecture, and therefore the same set of predictors (just different weights on the predictors since the two models were trained differently).

Answer 1

A few thoughts:

First, Frank Harrell (who developed the C-index) doesn't think that it is useful for comparing models. See this answer and its links, especially to a web post by Harrell. The C-index nicely summarizes how well a single model discriminates between individuals, but it says nothing directly about the calibration of the model: how well the model's predicted survival estimates correspond with what was observed. The link contains references to better ways to compare models.

In response to edited question: The Antolini index, as I understand it, is still just a measure of discrimination. It evidently extends the C-index to a situation with time dependence. It does not seem to provide any more information about calibration than the C-index.

If you are interested in the quality of a prediction "for a single sample," as you say, then what you need is a measure of calibration instead. That's typically done by estimating "observed" probabilities of survival at some time by a very flexible fit to the data set and comparing against model-predicted probabilities. Repeating the process on resampled data sets can give corrections for optimism in the fit. See this page for an outline.

The potential problem with a calibration measure is that your use of the Antolini index suggests that you have time-dependent covariates in your model. In that situation I don't know of a reliable way to estimate "observed" and "expected" probabilities of events at any given survival time, at least in a way that can extend reliably to new data samples.

The problem (at least for survival models with at most one event per individual) is that if you have a covariate value for an individual at some time, you already know that the individual is alive at that time. The lifelines package, for example, thus won't even allow for predictions from Cox models with time-dependent covariates. Harrell's calibrate() function in his R rms package won't handle them, either. There might be ways around that with a joint model of covariates and survival over time, but that's beyond my expertise.

Second, unless you have tens of thousands of cases, using separate training and test sets isn't a good idea. See this post by Frank Harrell.

In response to edited question: If you have a large enough sample to set aside a completely separate test set, then you already can compare your discrimination measure directly between the two models. The Wilcoxon-Mann-Whitney test is just a discrimination test, a rescaling of the C-index. See the section of this answer about that test.

Resampling via cross-validation or bootstrapping is close to what you suggest for getting estimates of variability; explaining what you did to others would be easiest if you directly invoked one of those approaches. They have the advantage that you could compare the whole modeling process between your two training methods. For example, you could repeat each of the training methods on multiple bootstrap samples of the training data and evaluate the discrimination either on the entire data set (for a small data set) or on the held-out test set (for very large data set). That would compare the performance of the two training methods directly.

Third, an answer would depend on what you mean by the "baseline model." Trivially, if the baseline model is a null model (no model at all, no predictors) and you really wanted to use the C-index, then the standard error reported for the C-index (concordance) would indicate whether it's statistically different from the value of 0.5 expected from a null model.

If the "baseline model" is a model only using a subset of the predictors in your complete model (that is, the baseline model is nested within the complete model), then a likelihood-ratio test between those models would be a better-accepted and more sensitive comparison. (Again trivially, the likelihood-ratio test reported just for the complete model also documents its superiority to a null model.)

In response to edited question: These don't seem to be nested models of the type that a likelihood-ratio test could evaluate.