nested cross validaiton vs train-test split

Question

I am trying to understand the main benefits of conducting a nested cross-validation compared to a simpler train-test split. Let us say I would like to build a prediction model. I initially split my data so that 80% of it is used for training and the remaining 20% of it for testing. Then, I run CV on the 80% to tune the hyperparameters and finally run the model using the optimal hyperparameters on the test sample, in order to get an unbiased estimate of my model performance.

Now, my understanding is that nested-CV has two main benefits:

1. You get to use the entire data you have as part the training process (so the inner CV would essentially get to see all the data at some point).

2. The model performance estimate you get could be more stable (in the sense that it is not based on a single run using the test data, but on multiple runs.

Am I missing something? And from a practical standpoint, assuming a large-enough database, does one really gain much from adding the computational complexity of a nested-CV compared to a simpler train-test split?

Thanks a lot.

Answer 1

Just to make it clear, the procedure given in your first paragraph is train/test split followed by a cross-validation. Nested CV is obtained by looping through the train/test split multiple times. This is to get more stable performance estimates, and it has diminishing returns as the data gets bigger, and may not be worthy of the complexity. But, with small data sets, it can be of paramount importance.

1. You get to use the entire data you have as part the training process (so the inner CV would essentially get to see all the data at some point).

Yes, but the main duty of the outer process is to stabilize the performance measurement.

Note that, nested CV is sometimes used in hierarchical model selection, e.g. the outer CV loops through the algorithms (random forest, logistic regression etc.) and the inner CV loops through the hyper-parameters.