I performed 10-fold cross-validation (cv.glmnet) on training data (80/20 split), total data consisting of 1000 samples. Next, I took the lambda.1se value and then fit my model (glmnet) on all the training data. Finally, I test the model on the test data and record the MSE.

I'm being asked by my supervisor to "test the success of my cross validation fit across all samples" since that one holdout test set "meant nothing". My instructions are to 1) choose a quarter of the samples for testing 2) do it 1000 times 3) record MSE for each of those 4) create a histogram of the MSEs.

I don't quite understand how to approach this since they want me to use the same data to cross-validate and test? They said this is internal validation to choose the model by using test sets (1000 samplings). Also, I'm not sure why I would use the cross-validated fit to do this vs. the fit using lambda obtain from cross-validation. Maybe I'm not interpreting their words correctly, but I did write them down as they were speaking.

Any suggestions are appreciated.

  • 1
    $\begingroup$ I'm not sure what you're supervisor is asking. If they are meaning for you to perform a kind of interval validation, I would suggest an alternate approach (the bootstrap as described by Frank Harrell and Ewout Steyerberg) $\endgroup$ Feb 18, 2022 at 20:11

1 Answer 1


Your supervisor(s) does not want you to test your approach on a 200 sample holdout set, most probably because it is small. Instead, they want this procedure repeated multiple times (here 1000) and see how the MSE fluctuates between the trials.

So, each of these 1000 trials will select a quarter of the dataset as test data; perform hyper-parameter tuning with cross-validation on the training set as you've done before, and test on the holdout quarter.

This way, you'd be able to "test the success of your cross validation fit across all samples", since repeatedly sampling a quarter of your dataset will leave no samples out with very high probability.

An alternate approach would be bootstrapping the whole process as also mentioned in the comments although this is not exactly what your supervisor(s) want.

  • $\begingroup$ Thanks for helping me to understand. Just to clarify on your second paragraph: I should use the obtained tuning parameter (lambda) from CV, fit model on all training data, then test on holdout quarter. Each time I will get a different tuning parameter, but the idea is to test cross validation fit across all samples so it wouldn't matter - is this correct? $\endgroup$
    – sumthymes
    Feb 19, 2022 at 5:32
  • $\begingroup$ Correct, that is what I believe your supervisors meant. $\endgroup$
    – gunes
    Feb 19, 2022 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.