I know there are many topics(1,2,3), papers(1,2,3) and websites(1) that discuss this topic at length. However for the past two days I am reading all I can find about the subject and seems I hit a brick wall in terms of progress.

As so, another view on the subject would be very appreciated.

For what I understood, we use nested cross validation when we have several models, each of those with some amount to hyper-parameters to tune. Here is a an example where, as far I can tell, nested cross validation would be used:

Lets say we have a data set composed by 100 observations and we want to see which of the following models(with hyper-parameters in parenthesis) is the best one:

-Neural Network(number of hidden layers, number of neurons in each hidden layer, activation function)

-KNN(Number of neighbors, distance measure)

-SVM(Type of kernel function, kernel function parameters)

As the number of observations is small we can't afford to separate three disjoints sets (training, validation, test) because we will probably end up with an worst model since it was trained with less data than is available. To fix that we will implement a cross validation strategy. (is this correct so far?)

In the inner loop of the nested cross validation, we would choose the best hyper-parameters combination for each model. after that we would train each model with the combined data from the inner loop, and then compare then in the outer loop cross-validation. the model with smaller error measure would be considered the best. This model will then be trained with the whole data set and be used for future predictions.

My main doubt is: why can't I do repeated cross validation to get the same results? As an example lets say I used repeated k-fold. In that case I would train every model and hyper-parameter combination in the training set and evaluate its performance in the test set for each k-fold split. I would then choose the model and hyper-parameters that gave the smaller mean error across all repetitions. Finally The best model is fitted to the whole data similar to the nested cross validation example.

As I understand, the estimated error would be biased in the repeated cross validation example. This happens because there is some leakage of information since we are using the same data for model selection and hyper-parameters tuning, and model assessment . However if I am only interested in choosing the best model is nested cross validation really necessary?

Feel free to correct any wrong assumption or improper terminology usage as I am fairly new to this field and any help would be greatly appreciated.

  • After reading a ton of posts and papers on the topic, I'm stuck with exactly this question. I would love to have someone provide insight into this. Would especially appreciate it if either @cbeleites or Dikran Marsupial (or both) can give some input. – amanbirs Nov 16 '17 at 17:54
  • Tagging @Dikran Marsupial as well, since I can only tag one external user per comment! – amanbirs Nov 16 '17 at 17:54

As I understand, the estimated error would be biased in the repeated cross validation example.... However if I am only interested in choosing the best model is nested cross validation really necessary?

Short Answer: Your understanding is correct. Nested cross-validation is not necessary to pick the best model (out of the models considered). However, are you sure that this is what you want to do?

Long Answer: As discussed here in similar terms, you can simplify your question by thinking of hyperparameter selection and model selection (i.e., Neural Network (NN), KNN, or SVM) as the same question; that is, you can think of selecting a single model from the big set $$\left\{ NN_1, ...., NN_{p_{NN}}, KNN_1,...,KNN_{p_{KNN}}, ... SVM_1,...,SVM_{p_{SVM}} \right\}$$ (where the subscript is used to index the different hyperparameter combinations).

This should make it clear why you only need one cross-validation loop to select the best model. As you suspected, nested cross-validation is needed only to obtain an unbiased estimate of error.

That said, I think it's very unlikely that you would want to do this in practice (except as a (nested) step in some larger validation procedure), as this leaves you vulnerable to over-fitting (especially if the number of candidate hyperparameters is large relative to the sample size). Regardless of the estimated error, your final model, fit to the whole data, could perform absolutely terribly when confronted with new data -- you would have no way of knowing.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.