In many machine learning research papers, the authors state, "We performed k-fold cross validation and report the average performance metrics on the test set across all folds". No mention of model selection or hyperparameter tuning.

To my understanding, this would be ok (not lead to a biased metric) if there was no hyperparameter tuning or model selection done as part of the cross validation. In other words, they randomly/manually chose hyperparameters, performed k-fold cross validation once, and reported average test set performance metrics.

However, I find it hard to believe that the authors of these papers are not hyperparameter tuning. Are they simply excluding this and claiming that hyperparameters that they chose manually on the first go work well, or is hyperparameter tuning on a k-fold cross validation to find the best hyperparameters and reporting the biased average test set performance metrics so common that it is just accepted now?

  • 1
    $\begingroup$ I think it's the latter and this makes me furious, too! It's data dredging and I do not understand that reviewers do not complain about this. I often see that they show 10 different models and then print the score of the best model bold, and claim that this is an unbiased empirical risk estimate. No, it's not, since they did model choice on the test set. AAAARGH. $\endgroup$
    – PascalIv
    Jan 18 at 14:13
  • $\begingroup$ I'm not sure if your claim is true or not, but you are asking a speculative and opinion-based question that doesn't have a good answer. Maybe you could give examples of such papers and ask about those papers in particular? $\endgroup$
    – Tim
    Jan 18 at 17:12