0
$\begingroup$

In practice, how does cross validation work, (say k-fold or leave one out) if we want to have a validation set as well as a test set?

Would we separate the data, say 80/20, training/test. And then within the training data, do cross validation? Most texts just say we just do CV on the entire sample, not making the distinction between validation and test.

$\endgroup$
0
$\begingroup$

Typically you would split your sample into three blocks: 70% (train + validation sets) 30% test set, run k-fold cross-validation on 70% block to tune and estimate regression, and use test set for computing out-of-sample prediction performance. See cross-validation section in the book [1]. As you mention, you can cross-validate on the whole sample (so using only train + validation sets) and report prediction errors computed on the validation set. Clearly, there is a trade-off between these two approaches: blocking in three parts may reduce overfitted issues, but will suffer from the smaller samples (increased bias & reduced variance) while the opposite holds for the second approach.

Ultimately, it depends on the sensitivity of your loss function to overfitting and on the data (e.g. presence of heavy-tails). Best would be to look for a study that used a similar approach on similar data as in your study and check what works in that study, or to compute both to see what works better for your case. You could also run a small simulation study to compare both methods under different DGPs and use the winner on your real dataset.

Hope this helps!

Refs:

[1] Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.