0
$\begingroup$

I learnt about validation for early stopping by taking the course, and I have several questions.

  1. Question about validation for early stopping:

    We split the dataset $D$ into a training set $G$ and a test set $T$. The training set $G$ is then split into a validation set $V$ and a smaller training set $G^-$. The trained model on $G^-$ is called $g^-$. The idea of early stopping with a validation set is 'to get a better estimate $\Bbb E_{val}(g^-)$ of a worse quantity $\Bbb E_{out}(g^-)$'.

    My question is after we get a satisfying $\Bbb E_{val}(g^-)$ thus stop training, do we simply forget about the validation set and calculate the $\Bbb E_{out}(g^-)$ on the test set?

  2. Question about cross-validation for early stopping:

    I learnt that CV can also be used for early stopping. The key is to treat the training iteration as a hyperparameter, so we just perform a grid search for this hyperparameter with CV as we usually do. At the end of the day, the training iteration that yields the smallest averaged validation error will be chosen.

    Do we actually do this in practice? I think this approach is costly.

$\endgroup$
0
$\begingroup$

For you first question, I would think using CV would be a better choice.
Say you have 10 sets of hyperparameters to try, and set the CV fold to 5. For each set of hyperparameter, you will train 5 models, and have 5 scores on the each of the 5 validation sets. Then you can take the average (I think?) of these 5 validation scores, tag it as the "final score" of this hyperparameter. In this way, you can pick the best hyperparameter set out from 10.

Furthermore, you can use this best hyperparameter set to train on all the data (without test set), and evaluate this model on the test set, just to give you some idea. In this CV way, you avoid fitting a model that is best on a single validation set. In each CV-fold, you can still use early-stopping while training.

For your second question, it should be partially illustrated above. Yes, there are computational cost involved at evaluating the validation scores, but you can probably specify only to check the validation scores every N rounds (or number of trees in the case of boosting trees algorithm)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @ Thank you for your answer! But I think the first answer that you give only rephrase the second question that I ask. $\endgroup$ – meTchaikovsky Aug 30 '18 at 0:19
  • $\begingroup$ Ah sorry. To the first question, I would say yes: the validation set has done its job :). But as I alluded to, doing this might run into the risk of overfitting to that single validation set, so if possible, might as well just do CV. $\endgroup$ – ccy Aug 30 '18 at 2:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.