1
$\begingroup$

I understand that to get an unbiased estimate of performance you need some sort of outer CV. However, this assumes you already have dealt with hyperparameter tuning.

Nested CV suggests CV for the performance estimation in the outer loop, and CV for model selection/hyperparam tuning in the inner loop. I'm confused why an inner CV is necessary at all for hyperparameter tuning. Couldn't you do an inner train/val/test split? What do you gain by using CV for model selection? It just seems more computationally expensive.

[edit to clarify]: I mean it seems overkill to use nested k-folds, rather than just an outer k-folds and an inner train/val/test split.

Improve this question
$\endgroup$
2
  • 5
    $\begingroup$ Train/validation/test split also is a form of cross-validation, same as $k$-fold cross-validation. $\endgroup$
    – Tim
    Commented Feb 10, 2020 at 20:58
  • $\begingroup$ Thank you for the clarification! I've regarded them as kind-of different things (considering train/val/test doesn't technically use the whole dataset for evaluation). I should have clarified that I meant more computationally expensive forms such as k-folds. $\endgroup$
    – davzaman
    Commented Feb 11, 2020 at 3:01

1 Answer 1

Reset to default
4
$\begingroup$

It's for the same reasons you use cross-validation in any situation, as opposed to a single train/test split. You're able to leverage the entire dataset for both training and testing, which provides more robust performance estimates (as it's calculated over a larger N), and protects against "unlucky" train/test splits (which becomes more likely with smaller datasets). Since computational power is rather cheap, it's often preferable to spend a bit more computing power to get a more robust and better-characterized result.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Why would you need this for hyperparameter tuning? So that you pick the best hyperparameters that are the best not by a chance split but consistently do better than the other configurations across multiple folds? [edit to add] isn't it possible to modify it to use k-2 folds for training, 1 fold for validation, and 1 for testing, and in that sense you're still using the whole dataset for validation/HT but never leaking into performance evaluation? That seems less expensive than nested k-folds? $\endgroup$
    – davzaman
    Commented Feb 11, 2020 at 2:59
  • $\begingroup$ @davzaman If you use 2 of the folds for testing and validation, that's 2 folds worth of data that aren't being used by your model building method at all. With a train/test split, you're typically reducing your available training data by 10-20%, but CV lets you get around that. If a single held-out test set contains "important" samples that the classifier really needs to see, it won't perform well on the test set. Train/test absolutely is less expensive than CV, but performance gains are usually worth some extra runtime. $\endgroup$
    – Nuclear Hoagie
    Commented Feb 11, 2020 at 14:29
  • $\begingroup$ I may be misunderstanding you, in the framework I suggested I believe you'd still technically be able to use the whole dataset for training; if it missed those important samples in one fold it would surely see it in another down the line. I am still confused how doing nested k-folds instead of an outer k-folds and an alternative inner process for hyperparam tuning isn't as good? When you do the inner k-fold loop you'd still be holding out validation, maybe a little less than a whole fold (depending on the choice of k_inner), so it's still withholding datapoints from training. $\endgroup$
    – davzaman
    Commented Feb 11, 2020 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.