31
$\begingroup$

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k values and report cross validation results of the best hyperparameter configuration.

To my knowledge, this result is biased, and they should have retained a separate test set to obtain an accuracy estimate on samples not used to perform hyperparameter optimization.

Am I right? Can you provide some references (preferably research papers) that describe this misuse of cross validation?

$\endgroup$
  • 3
    $\begingroup$ Note that instead of a separate test set one can use so-called nested cross-validation. If you search for this term on this site, you will find a lot of discussions. Look in particular for the answers by @DikranMarsupial who is one of the authors of the second paper cited in the accepted answer. $\endgroup$ – amoeba says Reinstate Monica Jul 18 '16 at 21:05
30
$\begingroup$

Yes, there are issues with reporting only k-fold CV results. You could use e.g. the following three publications for your purpose (though there are more out there, of course) to point people towards the right direction:

I personally like those because they try to state the issues more in plain English than in Math.

$\endgroup$
  • 2
    $\begingroup$ More precisely, the issue is not reporting cross validation results, but reporting performance estimates that have been part of the selection/optimization process. $\endgroup$ – cbeleites supports Monica Jul 19 '16 at 8:07
  • 1
    $\begingroup$ Also note that the Bengio & Grandvalet paper is somewhat less relevant if the issue here is the performance of a specific model trained on a particular data set - they discuss performance for the same trainig algorithm applied to new data sets from the same population (which needs to include variance between different data sets of the same size sampled from the same source - which is not an issue if we're talking about the prediction performance of a model trained on a specific data set). $\endgroup$ – cbeleites supports Monica Jul 19 '16 at 8:12
  • 1
    $\begingroup$ @cbeleites Correctly spotted: in my first draft of the answer I accidentally picked the third reference instead of the second one, but later didn't want to remove any information from the already accepted answer anymore - which is why I instead added the second in between (see versions of answer). Nevertheless, I think the question was mainly around the reported error, and those papers point out some of the things that one can do wrong with CV in this regard very well IMHO. $\endgroup$ – geekoverdose Jul 20 '16 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.