How does cross-validation fit in the training, validation and testing phases?

Question

I'm having trouble trying to understand how cross-validation fits in within the training, validation and testing concepts.

Is cross-validation supposed to be conducted during the training+validation phase? Or during the validation+testing phase?

My context: I get confused because I'm using cross-validation during the training+validation phase of a feature selection model (e.g. repeated k-fold cross-validating the parameters for lasso, elastic net, or repeated k-fold cross-validation for caret's recursive feature elimination). However, I would then presumably need to test on a new sample during the 'test phase'. This would imply that cross-validation is training+validation, but not the testing. Or am I wrong, and cross-validation can involve validation+testing too?

Answer 1

It often involves both, with one cross-validation (CV) loop nested inside another one. For instance, let's say you have data organized in 10 independent "chunks". In the outer loop, you set aside one chunk of this data for testing, and use the rest for training. You iterate over chunks until each has been used for testing once (k-fold cross-validation).

Within this outer loop however is another loop, over the 9 chunks of data that you defined as your training data. In each cycle of this loop, you set aside one of these 9 chunks for validation (e.g. for selecting the Lasso hyperparameter, i.e. the amount of shrinkage), while you use the other 8 to estimate your model parameters, and again you repeat this until each of the 9 chunks has been used for validation once. So you have a 9-fold CV-loop within a 10-fold CV-loop.

To give a concrete example: for Lasso, you'd use the 8 "estimation chunks" to estimate the regression coefficients, for each in a range of possible Lasso hyperparameters (e.g. $10^{-5}:10^5$). You'd then use the 9th "validation chunk" you had set aside, to calculate the cross-validated cost for each of these hyperparameter settings. Then repeat with a new validation chunk, etc., so that by the end of this loop, you have a cross-validation loss for each of your 9 chunks. You can then average over these 9 losses (per hyperparameter setting) to get a single validation loss for each level of shrinkage. Then select the shrinkage that produced the best results, run the regression one more time on all 9 training chunks with this optimal shrinkage, and finally, use the resulting regression model to make predictions for the 10th chunk that you originally withheld for testing. And then you pick a new chunk for testing and start all over again with the inner loop (and do that a total of 10 times, until you've got predictions for each of your 10 chunks).

This CV-within-CV approach is computationally expensive but it allows you to use as much of your data as possible for training, validation and testing (without information "leaking" between what should be independent sets of data). It's especially useful in situations where the amount of data is the limiting factor (rather than computational time or resources).