So, when performing CV we target two goals:

  1. Build the best possible model
  2. Estimate model performance

I have read about nested CV and also about Tibshirani-Tibshirani method. In either case, is clear to me that we test against several hyper-parameter settings, choose the best and build a model with entire training set using the best hyper-parameter setting. While the final build model from goal 1 is the same, the performance estimation of the model is different.

Having this in mind, I know that when using conventional CV without hyper-parameters we perform several re-runs, hence for example 5-folds 10-cross-validation indicates to repeat 10 times the 5-fold algorithm and average the performance of each re-run. So my question is:

When several re-runs are performed with nested CV or Tibshirani-Tibshirani method we'll probably get different "best hyper-parameters" in each run, which one should I report? Are every hyper-parameter setting tested again in each re-run? Am I missing something here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.