Is it ok to restrict data to uncensored patient in order to compare models

Question

I have read this answer already "removing-censored-observation". I understand that removing censored data induce a bias in the analysis. But I have somehow a specific case.

I am just interested into compare discriminative power of two models w.r.t to independant (of training) test data in a survival analysis setting.

I have two possibilities for concordance index.

The first is to use Uno's concordance index that takes into account distribution of censoring on training data. I don't really understand how it works. I don't have access of one of the model's training data.

The other one is to accept that the bias exists but it's the same for both. A first tricky thing is that the behavior of the model is not independent to this bias. But it looks very unlikely. A second one would be that the population the models aims to screen on a, has a subset of patient that behave in certain way that is both linked to the outcome (Event), with the censoring process. Which also looks unlikely.

I am a beginner as a statistician. I hope you will forgive my unscientific way of thinking. But is there a way to prove that I'm right or wrong and if so. What should I do ?

details

The goal is to evaluate deep learning model for long term cancer risk diagnosis based on lung radiography

Answer 1

As I understand the situation, you have two "deep learning" survival models whose performances you want to compare on the same new test set. You should use all of the data in the test set, whether the observations are right-censored or not. The review by Leung et al. in Annu. Rev. Public Health 1997; 18:83–104 explains the problems with omitting censored observations in general.

It's possible that those problems are less important in a test set than in a training set, but there's no reason to omit the information provided by cases whose event times are right-censored. The C-index that you want to use to evaluate the models is the fraction of pairs of comparable cases in which the observed and predicted event times are in the same order. A case with a right-censored event time thus can be included in a pair with any case whose observed event time is less than or equal to that right-censoring time. Such cases may be "less informative" than those with an observed event time, but they nevertheless provide useful information up through their right-censoring times, both for training and for testing models. Why throw away such informative data?

As you are taking the two models as given, I understand that the Uno adjustment would weight the cases based on the pattern of censoring in the test set and thus would use the same case weights for both models.

A couple of additional warnings:

First, train/test evaluation of models is generally unreliable unless you have on many thousands of observations. See this page. In your situation it seems that at least one of the models is based on raw data that you don't have, so comparing the models against a single test set might make sense for you.

Second, Harrell finds that the C-index is not a very sensitive way to compare models, as it is only a test of discrimination rather than calibration. See this post, for example. Evaluating plots of observed versus predicted survival probability at a time point of interest is generally better. There are ways to do that with censored data, outlined on this page.