36
$\begingroup$

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k values and report cross validation results of the best hyperparameter configuration.

To my knowledge, this result is biased, and they should have retained a separate test set to obtain an accuracy estimate on samples not used to perform hyperparameter optimization.

Am I right? Can you provide some references (preferably research papers) that describe this misuse of cross validation?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
34
$\begingroup$

Yes, there are issues with reporting only k-fold CV results. You could use e.g. the following three publications for your purpose (though there are more out there, of course) to point people towards the right direction:

I personally like those because they try to state the issues more in plain English than in Math.

Improve this answer
$\endgroup$
2
  • 4
    $\begingroup$ More precisely, the issue is not reporting cross validation results, but reporting performance estimates that have been part of the selection/optimization process. $\endgroup$
    – cbeleites
    Commented Jul 19, 2016 at 8:07
  • 1
    $\begingroup$ Also note that the Bengio & Grandvalet paper is somewhat less relevant if the issue here is the performance of a specific model trained on a particular data set - they discuss performance for the same trainig algorithm applied to new data sets from the same population (which needs to include variance between different data sets of the same size sampled from the same source - which is not an issue if we're talking about the prediction performance of a model trained on a specific data set). $\endgroup$
    – cbeleites
    Commented Jul 19, 2016 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.