How do I test the success of my cross validation fit across all samples?

Question

I performed 10-fold cross-validation (cv.glmnet) on training data (80/20 split), total data consisting of 1000 samples. Next, I took the lambda.1se value and then fit my model (glmnet) on all the training data. Finally, I test the model on the test data and record the MSE.

I'm being asked by my supervisor to "test the success of my cross validation fit across all samples" since that one holdout test set "meant nothing". My instructions are to 1) choose a quarter of the samples for testing 2) do it 1000 times 3) record MSE for each of those 4) create a histogram of the MSEs.

I don't quite understand how to approach this since they want me to use the same data to cross-validate and test? They said this is internal validation to choose the model by using test sets (1000 samplings). Also, I'm not sure why I would use the cross-validated fit to do this vs. the fit using lambda obtain from cross-validation. Maybe I'm not interpreting their words correctly, but I did write them down as they were speaking.

Any suggestions are appreciated.

Answer 1

Your supervisor(s) does not want you to test your approach on a 200 sample holdout set, most probably because it is small. Instead, they want this procedure repeated multiple times (here 1000) and see how the MSE fluctuates between the trials.

So, each of these 1000 trials will select a quarter of the dataset as test data; perform hyper-parameter tuning with cross-validation on the training set as you've done before, and test on the holdout quarter.

This way, you'd be able to "test the success of your cross validation fit across all samples", since repeatedly sampling a quarter of your dataset will leave no samples out with very high probability.

An alternate approach would be bootstrapping the whole process as also mentioned in the comments although this is not exactly what your supervisor(s) want.