0
$\begingroup$

I'm doing a survival analysis for cancer patients. I have two different models for two different types of cancer (different datasets).

Some key differences between model 1 and 2 are:

  • Model 1 has only 152 covariates and model 2 has 650
  • Model 1 subjects are being clustered and model 2 not

Model 1 seems to be well fit when analyzing p-value for the covariates. A total of 29 covariates have p-values<0.005

My problem is Model 2 shows a c-index of 0.83, however not even one covariate has p-value<0.005. Why is this happening and shouldn't model 2 perform way worse?

Model 1:

 <lifelines.CoxPHFitter: fitted with 519 total observations, 333 right-censored observations>
                 duration col = 'duration_col'
                    event col = 'vital_status.Dead'
                  cluster col = 'paper_iCluster.Group'
                    penalizer = 0.1
                     l1 ratio = 0.0
              robust variance = True
          baseline estimation = breslow
       number of observations = 519
    number of events observed = 186
       partial log-likelihood = -919.93
    
    Concordance = 0.79
    Partial AIC = 2143.85
    log-likelihood ratio test = 161.27 on 152 df
    -log2(p) of ll-ratio test = 1.80

Model 2:

 <lifelines.CoxPHFitter: fitted with 495 total observations, 283 right-censored observations>
                 duration col = 'duration_col'
                    event col = 'vital_status.Dead'
                    penalizer = 0.1
                     l1 ratio = 0.0
          baseline estimation = breslow
       number of observations = 495
    number of events observed = 212
       partial log-likelihood = -974.79
    
    Concordance = 0.83
    Partial AIC = 3249.57
    log-likelihood ratio test = 307.35 on 650 df
    -log2(p) of ll-ratio test = -0.00
$\endgroup$
4
  • 1
    $\begingroup$ Your models' samples are not the same- the sample size and event count change in the output. You are not comparing the same model on the same subjects and should expect a different answer. Make sure Model1 is run on the same sample as Model2, which will allow comparison. Ultimately, the model with more covariates may be better, and you will need to take great care the larger model isn't pertinent only to the derivation data (e.g. overfit). $\endgroup$
    – Todd D
    Dec 21 '20 at 0:21
  • $\begingroup$ I understand your comment and I do agree. However, I was not comparing the models sorry If I misformulated but I intended to use Model 1 more as an example. I know it's not even accurate to compare models using c-index. But my question is still valid, why does model 2 has very good c-index (>0.8) when non of the covariates are significant? $\endgroup$
    – beerzy
    Dec 21 '20 at 1:13
  • 1
    $\begingroup$ p-values and predictive accuracy are different concepts, and don't need to be related at all, in fact. See an extreme example here: stats.stackexchange.com/questions/451417/… $\endgroup$ Dec 21 '20 at 1:40
  • $\begingroup$ Thanks, that was precisely what I was looking for! And cheers for the dedication to lifelines it's great material :) $\endgroup$
    – beerzy
    Dec 21 '20 at 3:06
0
$\begingroup$

First, as noted in a comment, you seem to have two different data sets with two different models, so comparisons between them are problematic at best.

Second, you seem to be using a particular fixed penalization on all model coefficients, a ridge regression. You don't seem, however, to have tried to optimize the penalty factor, just choosing a value of 0.1 for all predictors. It's standard practice to choose that penalty factor by cross validation, as a particular choice might otherwise lead to substantial over- or under-fitting. And if you did that, then any p-values you get would be questionable, as they wouldn't take into account your use of the data to build the model.

Third, as the lifelines author pointed out in a comment, those p-values don't really matter if your interest is in prediction. You could consider repeating your modeling process on multiple bootstrapped samples of your data to get some estimate of coefficient-estimate reproducibility, though.

Fourth, Harrell's c-index isn't very sensitive for comparisons between models, as he states in the related context of logistic regression. It only tells you how good the relative risk rankings are among cases, not how precise the event-time estimates are. A measure more directly related to model calibration and prediction accuracy would be better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.