Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In practice, how does cross validation work, (say k-fold or leave one out) if we want to have a validation set as well as a test set?
Would we separate the data, say 80/20, training/test. And then within the training data, do cross validation? Most texts just say we just do CV on the entire sample, not making the distinction between validation and test.
Typically you would split your sample into three blocks: 70% (train + validation sets) 30% test set, run k-fold cross-validation on 70% block to tune and estimate regression, and use test set for computing out-of-sample prediction performance. See cross-validation section in the book [1]. As you mention, you can cross-validate on the whole sample (so using only train + validation sets) and report prediction errors computed on the validation set. Clearly, there is a trade-off between these two approaches: blocking in three parts may reduce overfitted issues, but will suffer from the smaller samples (increased bias & reduced variance) while the opposite holds for the second approach.
Ultimately, it depends on the sensitivity of your loss function to overfitting and on the data (e.g. presence of heavy-tails). Best would be to look for a study that used a similar approach on similar data as in your study and check what works in that study, or to compute both to see what works better for your case. You could also run a small simulation study to compare both methods under different DGPs and use the winner on your real dataset.
Hope this helps!
Refs:
[1] Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.
