2
$\begingroup$

I understand that for small sample sizes its best not to opt for the traditional hold out validation but rather use the entire dataset and perform a k-fold cross-validation. However in my case i want to find the best hyper-parameters and select the best model, i understand that in this case a nested cross validation may be the better option.

If I understand the method of nested cross validation, i.e. take your entire dataset and split it into 2 folds. You then take each fold and perform a k-fold cross validation to determine best hyper-parametes on one fold. And for the other fold you perform a k-fold cross validation to determine the best model. Is my understanding of cross validation correct and if so how is this any different to hold one validation?

$\endgroup$
  • $\begingroup$ I still didnt get the second part. You are splitting your data into 2 sets to do a k-fold cv on each? This would not be "nested" CV, only 2 CVs. In nested CV, there is an outer loop and an inner loop. The inner part is usually to fit a model and the outer is for hyperparameter tuning. In contrast, the Hold-Out method reserves a certain amount for testing (1/3) and uses the remainder for training (2/3). $\endgroup$ – Laksan Nathan Sep 5 '17 at 15:27
  • $\begingroup$ Thanks @Inathan, i understand that but i fail to see how nested cross validation methods works to ensure that the inner loop does not leak data into the outer loop during training - does that make sense? $\endgroup$ – Martin Sep 5 '17 at 15:44
1
$\begingroup$

Your fundamentally misunderstood the concept of cross validation.

  • $k$-fold cross validation: Have 1 loop

    • Equally divide dataset into $k$ folds.
    • For $i \in [1;k]$, test models on the $i$-th fold and train on the remaining folds.
    • The final result is the average results of $k$ testing folds.
  • nested $k$-fold cross validation: Have 2 loops.

    • For every $i$ above, we have a $k$-fold cross validation nested inside.
  • Holdout: literally hold a set of data out for testing.

In your case, when you split the data into 2 folds and perform two different tasks on each fold, you are not doing cross validation. The key thing to remember that whatever you want to do, you have perform it independently for every $i \in [1;k]$.

Thus, in your case, the correct way should be:

  • In the inner loop, you select best hyper-parameters by subsequently using different hyper-params to train model on the inner training folds and test on the inner testing fold.
  • Then use the best models selected in the inner loop to test on the outer loop.
  • The average results of the outer loop is the estimated performance of your model.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.