I came across many posts on CrossValidated discussing cross-validation and nested cross-validation as an alternative (e.g. here or here). I don't quite understand why 'ordinary' K-fold cross-validation gives biased estimates of error which is the reason why we need nested cross-validation to evaluate performance of the chosen model in a reliable (unbiased) way.

In all these posts about cross-validation there is emphasis on the difference between hyperparameter tuning (model selection) and estimation of generalization performance. But what is the difference here? Why can't I use 'ordinary' K-fold cross-validation for the two tasks of model selection and estimation at once? The way I understand it is that model selection is related to estimation of performance because choosing the best model we base our assessment on some metrics such as MSE which is used to assess performance.

Where is that bias coming from? We train different models on $K-1$ sets and then evaluate performance on the remaining set which wasn't used for training so it should give us a good estimate of performance, shouldn't it? All $K$ sets used for validation are independent. We don't use the same set for training and validation. I see that in case we perform repeated cross-validation the validation sets are not independent in different repetitions and standard errors of the mean error will be too low but I can't really see how that approach would give us biased estimates.

Is there anything wrong with this reasoning? If so, why? Maybe the source of bias is somewhat tricky and I can't see it.

  • 4
    $\begingroup$ Think about tuning the random seed: if you choose the one with highest cross-validation performance, then the cross-validation performance will certainly be overstated just by chance. $\endgroup$ – Michael M Jul 31 '20 at 17:29
  • 2
    $\begingroup$ @MichaelM But let’s say we don’t tune any hyperparameters and we have some model we have to fit to the data. We just want to check its performance. Can we use k-fold CV? Will it give unbiased estimate of error if I calculate the mean from all K validation sets? It’s the same situation but without any tuning. Maybe the problem is that to me there is no difference between model selection and performance because we select the best model assessing its performance. I just can’t see the difference. $\endgroup$ – treskov Jul 31 '20 at 18:56
  • $\begingroup$ Intuitively, I'd say the x-validation performance would be unbiased for independent rows and without tuning. However, I remember an issue with leave-one-out x-validation, tending to biased results. I have never understood why, I must admit. $\endgroup$ – Michael M Jul 31 '20 at 19:10
  • 1
    $\begingroup$ imagine you don't do training or have any hyper parameters, but randomly initialise model each time. then in cross validation you choose the (randomly initialised) model that gives the best performance on your crossvalidation set. is the performance estimate for this model (on crossvalidation set) a good estimate of future performance? can you see that the act of choosing the best model on the crossvalidation set biases the estimation of generalisation error? $\endgroup$ – seanv507 Aug 9 '20 at 18:17

Can it be related with the fact that your observations may not be independent? Imagine you are predicting if a given chemical mixture of two components will explode or not, based on the properties of the two components. A certain component A may appear in diverse observations: you can have it in a mixture of A+B, A+C, A+D, etc. Now, imagine that you use k-fold validation. When the model is predicting for the A+C mixture, maybe it was already trained with the observation "A+B", therefore, it will be biased towards the output of that observation (because half of the variables of the two observations are the same: in one you have the properties of A and the properties of C, and in the other one you gave the properties of A and the properties of B).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.