3
$\begingroup$

I have read this answer already "removing-censored-observation". I understand that removing censored data induce a bias in the analysis. But I have somehow a specific case.

I am just interested into compare discriminative power of two models w.r.t to independant (of training) test data in a survival analysis setting.

I have two possibilities for concordance index.

The first is to use Uno's concordance index that takes into account distribution of censoring on training data. I don't really understand how it works. I don't have access of one of the model's training data.

The other one is to accept that the bias exists but it's the same for both. A first tricky thing is that the behavior of the model is not independent to this bias. But it looks very unlikely. A second one would be that the population the models aims to screen on a, has a subset of patient that behave in certain way that is both linked to the outcome (Event), with the censoring process. Which also looks unlikely.

I am a beginner as a statistician. I hope you will forgive my unscientific way of thinking. But is there a way to prove that I'm right or wrong and if so. What should I do ?

details

The goal is to evaluate deep learning model for long term cancer risk diagnosis based on lung radiography

$\endgroup$
3
  • 2
    $\begingroup$ You don't need to remove censored observations to calculate the C-index. It's just the fraction of comparable case pairs for which predicted and observed event times are in the same order. A case with a right-censored event time can be compared against all cases with earlier observed event times, but not against other cases with right censoring or cases with later observed event times. Just do that with both models on the same test set. Even better, repeat that on multiple bootstrapped samples of the test data. $\endgroup$
    – EdM
    Mar 28 at 1:58
  • 1
    $\begingroup$ Frank Harrell, however, doesn't consider his C-index to be a very sensitive way to compare models. See this page, for example. If you could edit your question to provide more details about the types of survival models (e.g., Cox versus parametric) and the nature of the data, you might get suggestions for better ways to compare the models. $\endgroup$
    – EdM
    Mar 28 at 2:00
  • $\begingroup$ Hi @EdM, thank you for your answer. I actually provided some details about the use case. My concern is that if I have enough uncensored data, why would I use censored data that is less informative. If a model is prone to overestimate cancer risk. it will be less penalized on such data. And if the both model are impacted by the same bias, comparison is fair providing that, censorship process is CMAR, (censorship is at least independent of the cancer event which is often assumed). What are your thought about this ? $\endgroup$ Mar 28 at 16:28

1 Answer 1

3
$\begingroup$

As I understand the situation, you have two "deep learning" survival models whose performances you want to compare on the same new test set. You should use all of the data in the test set, whether the observations are right-censored or not. The review by Leung et al. in Annu. Rev. Public Health 1997; 18:83–104 explains the problems with omitting censored observations in general.

It's possible that those problems are less important in a test set than in a training set, but there's no reason to omit the information provided by cases whose event times are right-censored. The C-index that you want to use to evaluate the models is the fraction of pairs of comparable cases in which the observed and predicted event times are in the same order. A case with a right-censored event time thus can be included in a pair with any case whose observed event time is less than or equal to that right-censoring time. Such cases may be "less informative" than those with an observed event time, but they nevertheless provide useful information up through their right-censoring times, both for training and for testing models. Why throw away such informative data?

As you are taking the two models as given, I understand that the Uno adjustment would weight the cases based on the pattern of censoring in the test set and thus would use the same case weights for both models.

A couple of additional warnings:

First, train/test evaluation of models is generally unreliable unless you have on many thousands of observations. See this page. In your situation it seems that at least one of the models is based on raw data that you don't have, so comparing the models against a single test set might make sense for you.

Second, Harrell finds that the C-index is not a very sensitive way to compare models, as it is only a test of discrimination rather than calibration. See this post, for example. Evaluating plots of observed versus predicted survival probability at a time point of interest is generally better. There are ways to do that with censored data, outlined on this page.

$\endgroup$
2
  • $\begingroup$ I find it interesting (unsatisfactory actually) that the problem is posed as comparing two deep learning models -- presumably to determine which one is better which is not the same as determining if either one is good. Shouldn't this situation be treated similarly to clinical trials where a new treatment is meant to be compared to the current best treatment? What's the current best method for predicting cancer risk? $\endgroup$
    – dipetkov
    Mar 29 at 12:21
  • 1
    $\begingroup$ @dipetkov as I understand the clinical issues, "deep learning" from lung radiographs is an active area of research in its infancy, particularly in terms of use for predicting cancer risk. The lack of raw data for one of the models suggested to me that the OP has a new model to compare against a published model (the one for which the raw data aren't available). It certainly would be good to know whether either of these works better than (or adds to) the well known risks presented by smoking and exposure to radon. See this page. $\endgroup$
    – EdM
    Mar 29 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.