Cox prediction models: Statistical inference versus cohort-split (derivation->test)

Question

First I want to clarify that I understand that all prediction models needs external validation, and this applies to both machine learning models and conventional regression models. My question is regarding the development of prediction model using Cox-regression and the need for internal validation.

Machine learning models, which iteratively adjusts the weighting of prediction-variables trough a loss function, are prone to overfitting, and it is easy to understand why both internal and external validation is necessary. Perhaps the one could use some alternatives to the Train-test-splitting with boot-strap-aggregating methods or multi-fold cross-validation, but this is another issue.

The question is: In regression methods where you fit the model to the data without iterative adapting the variable coefficients: Does this approach really need the same form of internal validation by splitting in a derivation cohort and a test cohort? If so, what does this really achieve?

I understand some situations where this intuitively can be useful: 1)If you have a temporal difference in sub-cohorts where you compare older cases with newer cases 2) Other identifiable differences in the cohort that you can categorize the data by(for example variables potentially representing a source of noise). However, it is not clear to me why one should expect different results when random splitting a sample into a derivation sample and a test sample, fitting a model on the derivation sample and testing on the test-sample, compared to fitting the model on the whole sample and evaluating the predictive potential by inference from the sample.

It is of course obligatory to perform external validations to evaluate if the model predictions are valid for external data, before even thinking about to actually use a model in real-life-settings.

I have tried doing random splits in several datasets, and I never get different results when evaluating the model by splitting and predicting on test set, compared to the inferential approach.

If anyone have some enlightening insights on the matter, it would be greatly appreciated.

Answer 1

In regression models (including Cox regressions and generalized linear models), you can easily overfit by trying to include more predictors than are warranted by the size of the data sample. So validation of some sort is always warranted.

Frank Harrell's course notes have a very useful outline of the major issues, in the chapter on "Describing, Resampling, Validating, and Simplifying the Model." Completely external validation is best, but not always possible. The choice between a completely separate train/test split and resampling from a single data set should be informed by data size. With fewer than a few tens of thousands of cases, separate train/test groups tend to lose precision in the modeling and power for the testing. For smaller data sets, resampling validation is best.

A particularly useful form of such validation, outlined in that chapter, is the optimism bootstrap. After a model is developed on the full data set, modeling is repeated on multiple bootstrap samples of the data and evaluated against the full data set. This gives an estimate both of modeling variability and of the bias in the original model introduced by potential over-fitting. That bias estimate can be used to correct the original model and provide a calibration curve. That approach to validation and calibration is implemented for example in Harrell's rms package in R for regression models including Cox models.

Note that the optimism-bootstrap approach can also be used in a more general machine-learning context when your sample size is limited. You just repeat all of the modeling steps on multiple bootstrap samples, evaluating against the full data set, to get an estimate of how well your modeling approach works on the underlying population.

If you "never get different results when evaluating the model by splitting and predicting on test set, compared to the inferential approach," then you probably aren't overfitting to start with. That's admirable. The temptation to overfit regression models can be so great, however, that it's always best to check.

Answer 2

A table from real life data, illustrating the issue, with a "positive" optimism estimate: