Deviance vs c-statistic for fitting a L1 penalized Cox model with cross validation

Question

I'm trying to use penalized Cox regression to identify genes that are strongly associated with survival for further investigation via experimental studies. I have about 160 genes and 350 patients and I'm using glmnet package in R.

The glmnet package offers two different metrics for assessing model performance: deviance and Harrell's c-statistic. When I fit LASSO penalized model using deviance as the cross validation loss (CV)function, the model identifies several genes with non-zero $\beta$ values, but checking the best model's c-statistic on the overall dataset shows no predictive ability (c-stat = .5).

Although the overall goal of the analysis is feature selection for further study and not predictive performance, the lack of any predictive ability doesn't give me much confidence about the selected genes.

When I repeat the model fitting process but use the c-statistic as the CV loss function, a different set of genes are selected and the best model's c-statistic on the overall dataset shows some predictive ability (c-stat = .62).

Which metric should be used as the loss function for Cox cross validation? It's my understanding that the c-statistic should not generally be used for model selection because it is an improper scoring rule. Also using the c-statistic as a loss function and then accepting the results based on the overall c-statistic is double dipping because the c-statistic is being optimized. With these limitations in mind, are there any scenarios where using the c-statistic as the loss function is preferable? I assume there must be some scenarios or the authors wouldn't have included it in the package.

Answer 1

I wouldn't rely too much on the fact that glmnet includes the c-statistic as a built-in criterion. It also includes accuracy as a criterion for logistic regression, and you will find few on this site who think of that as useful. Frank Harrell himself doesn't think it is useful for discriminating among models, although it can be a very convenient and easily interpretable measure of overall model quality.

My suspicion is that you managed to find a particular combination of genes that provided a c-statistic above 0.5 in this particular data set, but that would not work well on a new data set. To test this, try a type of validation that Harrell recommends. He uses this type of approach in his rms package, but I don't think it's implemented there for LASSO'd Cox models.

Take a set of a few hundred bootstrap samples from your 350 patients. On each bootstrap sample, try repeating the entire model-building process, including cross-validation to find LASSO penalties. You thus get a few hundred models, each with potentially different selections of genes and coefficients. Then test the results of each model against the whole original data set, while keeping track of the performance of each model against its own bootstrap sample.

The idea is that taking bootstrap samples from the full data sample is like repeating the process of taking the full data sample from the underlying population. So the average difference between the performances of models on their corresponding bootstrap samples and those models' performances on the full data set gives an estimate of the "optimism" that your original model might have when you try to apply it to a new sample from the underlying population. My guess is that optimism will be pretty high for the modeling process that you optimized on the c-statistic.

You might improve your model if you include relevant clinical characteristics associated with outcome. Cox regression can have omitted-variable bias like logistic regression when predictors associated with outcome are left out. That bias can underestimate the actual magnitudes of the associations of included predictors with outcome. You might want to include some clinical characteristics as unpenalized and limit LASSO to your set of genes.