Why is there a difference in approach between cross-validation and using the test set in ISL's explanation of survival analysis?

Question

The following is from Introduction to Statistical Learning Python edition, page 485, from the survival analysis chapter.

They make a distinction between the cross-validation process (to choose a penalty lambda) and the use of the fitted model on the withheld test set. I don't really understand why there's a difference. K-fold cross-validation inevitably involves fitting a model on K-1 folds and then using the final holdout fold as a temporary validation or test set. It seems that any procedure or metric used within K-fold validation could be used mutatis mutandis on the final step where we bring in the test set. In this case, why can't we again use the partial likelihood deviance on the test set, rather than taking the strange approach below and manually stratifying the data into three discrete groups?

"We now apply the lasso-penalized Cox model to the Publication data, described in Section 11.5.4. We first randomly split the 244 trials into equally sized training and test sets. The cross-validation results from the training set are shown in Figure 11.7. The “partial likelihood deviance”, shown on the y-axis, is twice the cross-validated negative log partial likelihood; it plays the role of the cross-validation error.15 Note the “U-shape” of the partial likelihood deviance: just as we saw in previous chapters, the cross validation error is minimized for an intermediate level of model complexity. Specifically, this occurs when just two predictors, budget and impact, have non-zero estimated coefficients.

"Now, how do we apply this model to the test set? This brings up an important conceptual point: in essence, there is no simple way to compare predicted survival times and true survival times on the test set. The first problem is that some of the observations are censored, and so the true survival times for those observations are unobserved. The second issue arises from the fact that in the Cox model, rather than predicting a single survival time given a covariate vector x, we instead estimate an entire survival curve, S(t|x), as a function of t. Therefore, to assess the model fit, we must take a different approach, which involves stratifying the observations using the coefficient estimates."

After this subsection, the authors introduce the C-index. Is that a more standard metric to be used in cross-validation and testing of a survival analysis model?

Answer 1

Although An Introduction to Statistical Learning (ISL) has much to offer, you should not put too much emphasis on the details of its suggestions with respect to survival analysis.

First, train-test splits can be unreliable in practice unless you have tens of thousands of observations. See this post by Frank Harrell. In survival analysis, train/test splits are even more of a problem, as it's the number of events, not the number of cases, that provides power. That said, the concept of a train/test split is important as it does underlie cross-validation (as used for penalty selection in this situation), and it certainly is important in very large scale data analysis.

Second, checking a survival model requires some type of comparison between observed and predicted outcomes. That poses two problems in survival analysis. First, you don't have individual predictions of survival times, only of relative hazards (in Cox models; relative time acceleration in accelerated failure time models). Second, with censoring you don't have observed event times for the censored cases. That requires some alternate approach for evaluating the model.

The simple stratification by predicted hazard shown in ISL is just a very simple way to deal with those problems. You see if groups distinguished by predicted hazards actually differ in terms of their observed survival curves. That's a first-level evaluation of predicted versus observed outcomes.

There are much better ways to evaluate survival models, implemented in Frank Harrell's rms package in R. For example, with its calibrate() function you can compare smoothed estimates of observed outcomes against predicted risk at a particular time of interest, as outlined on this page. Its validate() function provides several measures of model quality, based on resampling. Although those include a $D_{xy}$ value from which you can calculate Harrell's C-index referred to in ISL, Harrell doesn't find that to be the best way to distinguish among models. If you want to pursue survival analysis, I'd recommend Chapters 17 to 21 of his Regression Modeling Stratgies.

Additional thoughts

First, the partial likelihood deviance is a good way to evaluate relative performance among Cox models on the same data, as in selecting the penalty value for LASSO. The value of the partial likelihood deviance, however, doesn't provide much information about the performance of a single model on a single set of data. That's why it isn't very useful for evaluating an already built model on a test data set.

Second, if you are going to use resampling (e.g., bootstrapping or repeated cross validation) to evaluate a LASSO model, you need to repeat all steps of the model-building process on each resample. That means repeating the choice of penalty, getting a new model, and evaluating the performance as outlined above for each resample.