Worthwhile to do k-fold cross-validation AND a holdout/test set?

Question

I'm relatively new to machine learning, and most of my experience at this stage comes from working with an automated machine learning tool called DataRobot. In their tool, and in their documentation and tutorials, they promote the idea of using 5-fold cross-validation AND a holdout set; that is, by default about 20% of the data is set aside to be tested against later (it is never used in the training process), and the remaining 80% is split into 5 partitions and run through cross-validation.

I've looked high and low for other references that recommend doing this, and can't find any. Generally, people seem to say that cross-validation is enough, and that there's nothing to be gained from having a holdout set if you're already cross-validating. On the other hand, I could see an argument being made that, because we're fine-tuning hyperparameters, and in DataRobot's case, comparing many different kinds of models, it is possible that the models, as assessed by cross-validation scores, may be overfitting to the data, so it is useful to have a truly independent set (the holdout) to evaluate once model selection is complete.

So, is DataRobot's approach recommended? Is it overly conservative, or widely applicable? Anything else I should know?

---

Answer 1

In short, yes; I would recommend it. Cross-validation is usually used for tuning hyperparameters (e.g. penalty term in lasso/ridge regression; tree depth and fraction of parameters per split in random forest; etc.). Once you use CV for tuning, it can no longer reliably be used to assess how well your method will perform on out-of-sample data. That's what the hold-out/test set is for, although it is admittedly suboptimal to use only a single sample to evaluate performance; some kind of hierarchical cross-validation procedure might be preferable.

e.g. from the scikit-learn documentation on cross-validation:

[When using CV a] test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

(The second clause refers to splitting a data set, a single time, into train/validate/test components.)