So, when performing CV we target two goals:
- Build the best possible model
- Estimate model performance
I have read about nested CV and also about Tibshirani-Tibshirani method. In either case, is clear to me that we test against several hyper-parameter settings, choose the best and build a model with entire training set using the best hyper-parameter setting. While the final build model from goal 1 is the same, the performance estimation of the model is different.
Having this in mind, I know that when using conventional CV without hyper-parameters we perform several re-runs, hence for example 5-folds 10-cross-validation indicates to repeat 10 times the 5-fold algorithm and average the performance of each re-run. So my question is:
When several re-runs are performed with nested CV or Tibshirani-Tibshirani method we'll probably get different "best hyper-parameters" in each run, which one should I report? Are every hyper-parameter setting tested again in each re-run? Am I missing something here?