# How does cross-validation fit in the training, validation and testing phases?

I'm having trouble trying to understand how cross-validation fits in within the training, validation and testing concepts.

Is cross-validation supposed to be conducted during the training+validation phase? Or during the validation+testing phase?

My context: I get confused because I'm using cross-validation during the training+validation phase of a feature selection model (e.g. repeated k-fold cross-validating the parameters for lasso, elastic net, or repeated k-fold cross-validation for caret's recursive feature elimination). However, I would then presumably need to test on a new sample during the 'test phase'. This would imply that cross-validation is training+validation, but not the testing. Or am I wrong, and cross-validation can involve validation+testing too?

To give a concrete example: for Lasso, you'd use the 8 "estimation chunks" to estimate the regression coefficients, for each in a range of possible Lasso hyperparameters (e.g. $10^{-5}:10^5$). You'd then use the 9th "validation chunk" you had set aside, to calculate the cross-validated cost for each of these hyperparameter settings. Then repeat with a new validation chunk, etc., so that by the end of this loop, you have a cross-validation loss for each of your 9 chunks. You can then average over these 9 losses (per hyperparameter setting) to get a single validation loss for each level of shrinkage. Then select the shrinkage that produced the best results, run the regression one more time on all 9 training chunks with this optimal shrinkage, and finally, use the resulting regression model to make predictions for the 10th chunk that you originally withheld for testing. And then you pick a new chunk for testing and start all over again with the inner loop (and do that a total of 10 times, until you've got predictions for each of your 10 chunks).