2
$\begingroup$

So I have (mostly) implemented K fold cross validation, with the help of this answer:

K fold cross validation; How many epochs to train for?

I am at a point where I have done a grid search and calculated the average error of each hyper parameter set against k folds. Great! I know the optimal parameters for my model.

I now want to train my model against the entire dataset. For the case where there is no additional holdout data, how does one know when to stop training?

Is it reasonable to train for the average number of epochs that each fold was trained for (when using the optimal hyper parameters)?

$\endgroup$
3
$\begingroup$

Yes this is reasonable. However, since the number of epochs is just another hyper-parameter, the CV results will be slightly optimistic compared to the one you'll see in your future data. This is because you've used the 'epoch' number that was optimized for each fold.

In order to calculate the 'actual' CV score, you'd have to use nested CV. So, within each fold of the outer CV, you do a new inner CV procedure which you use for finding the 'average' epoch number. Then, you use this figure to train the fold of the outer CV. You repeat this procedure for all outer CV folds. This way, the final outer CV score will not be 'overfitted' to the train data.

EDIT:

Pseudocode for nested CV

for outer_train_set, outer_valid_set in train_set(K_fold):
    for inner_train_set, inner_valid_set in outer_train_set(K_fold):
        Train NN and find the optimal #epochs with early stopping based on the inner_valid_set 
    Train NN using the average #epochs found on the inner loop
    Evaluate performance on outer_valid_set 

This nested CV procedure will give you the optimal hyper-parameters (if you don't already have them) and will show you the expected error (e.g. log loss, RMSE) that the model will have if you use the average #epochs of each fold.

The final selection of #epochs can be either the average respective figures of the outer folds or the average of the figures in the inner folds (if I'm not mistaken, these two figures should be the same).

$\endgroup$
  • 1
    $\begingroup$ Thanks for this, do you have a pseudo code example of the nested CV? $\endgroup$ – nixon Apr 11 '18 at 23:41
  • 1
    $\begingroup$ Edited my answer above $\endgroup$ – Stergios Apr 12 '18 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.