I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a brick wall in terms of progress.
As so, another view on the subject would be very appreciated.
For what I understood, we use nested cross validation when we have several models, each of those with some amount to hyper-parameters to tune. Here is a an example where, as far I can tell, nested cross validation would be used:
Lets say we have a data set composed by 100 observations and we want to see which of the following models(with hyper-parameters in parenthesis) is the best one:
-Neural Network(number of hidden layers, number of neurons in each hidden layer, activation function)
-KNN(Number of neighbors, distance measure)
-SVM(Type of kernel function, kernel function parameters)
As the number of observations is small we can't afford to separate three disjoints sets (training, validation, test) because we will probably end up with an worst model since it was trained with less data than is available. To fix that we will implement a cross validation strategy. (is this correct so far?)
In the inner loop of the nested cross validation, we would choose the best hyper-parameters combination for each model. after that we would train each model with the combined data from the inner loop, and then compare then in the outer loop cross-validation. the model with smaller error measure would be considered the best. This model will then be trained with the whole data set and be used for future predictions.
My main doubt is: why can't I do repeated cross validation to get the same results? As an example lets say I used repeated k-fold. In that case I would train every model and hyper-parameter combination in the training set and evaluate its performance in the test set for each k-fold split. I would then choose the model and hyper-parameters that gave the smaller mean error across all repetitions. Finally The best model is fitted to the whole data similar to the nested cross validation example.
As I understand, the estimated error would be biased in the repeated cross validation example. This happens because there is some leakage of information since we are using the same data for model selection and hyper-parameters tuning, and model assessment . However if I am only interested in choosing the best model is nested cross validation really necessary?
Feel free to correct any wrong assumption or improper terminology usage as I am fairly new to this field and any help would be greatly appreciated.