1
$\begingroup$

So, when performing CV we target two goals:

  1. Build the best possible model
  2. Estimate model performance

I have read about nested CV and also about Tibshirani-Tibshirani method. In either case, is clear to me that we test against several hyper-parameter settings, choose the best and build a model with entire training set using the best hyper-parameter setting. While the final build model from goal 1 is the same, the performance estimation of the model is different.

Having this in mind, I know that when using conventional CV without hyper-parameters we perform several re-runs, hence for example 5-folds 10-cross-validation indicates to repeat 10 times the 5-fold algorithm and average the performance of each re-run. So my question is:

When several re-runs are performed with nested CV or Tibshirani-Tibshirani method we'll probably get different "best hyper-parameters" in each run, which one should I report? Are every hyper-parameter setting tested again in each re-run? Am I missing something here?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.