16
$\begingroup$

I'm currently using early stopping in my work to prevent over fitting. Specifically those taken form Early Stopping But When?.

I'm now wanting to compare to other classification algorithms where it appears that 10 fold cross validation is widely used.

However I'm confused about whether cross validation is a method for preventing over fitting or selecting good parameters. (or maybe this is one and the same?). I'm also confused whether early stopping methods and cross validation can be used in place of one another or in combination.

So the question is: what is the relationship between early stopping and cross validation?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
7
$\begingroup$

Cross Validation is a method for estimating the generalisation accuracy of a supervised learning algorithm.

Early stopping is a method for avoiding overfitting and requires a method to assess the relationship between the generalisation accuracy of the learned model and the training accuracy.

So you could use cross validation to replace the validation set, mentioned in the paper you cite, within an early stopping framework. Ten fold cross validation for instance would be more accurate than using a single validation set, and would normally be a better estimate of generalisation error.

So to summarise, cross validation is a generalisation accuracy measure which could be used as part of an early stopping framework.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ That all makes sense, cheers. But form what I can tell 10 found cross validation seems to be used to choose the parameters for a given method, not to stop early when using that method. So I think I must still be missing something. $\endgroup$
    – Andy T
    Mar 16, 2015 at 11:58
  • 2
    $\begingroup$ @AndyT Maybe the thing to focus on here is generalisation estimation. Both parameter selection and early stopping require a good estimator of generalisation error. N fold cross validation is one such estimator. There are others, such as repeated hold out and bootstrapping, or a simple validation set. What you are trying to do in both cases, parameter selection and early stopping is assess how the model will perform on unseen data so you can choose a good model. To do this, in both cases you use a generalisation estimator. Does that help ? $\endgroup$
    – image_doctor
    Mar 16, 2015 at 12:03
  • 1
    $\begingroup$ Yes that's clear. Thank you very much for your time! It's much appreciated. $\endgroup$
    – Andy T
    Mar 16, 2015 at 13:13
  • $\begingroup$ @AndyT No trouble, I hope your project goes well! $\endgroup$
    – image_doctor
    Mar 16, 2015 at 13:25
2
$\begingroup$

In addition to the two generalization approaches you mention, there are many others.

  • adding regularization terms in your loss (cost) function that minimize the number and magnitude of your nonzero model parameters
  • randomly dropping (zeroing out) some portion of the weights/parameters in your model with each training epoch
  • adding a layer of stochastic nodes in your model (e.g., sampling from the "probabilities" given by the outputs of logistic functions)

Many of these approaches (including your cross-validation and early-stopping approaches) can be combined together to maximize model performance on unseen data (generalization performance).

One note on the early-stopping approach. For neural nets, Geoffrey Hinton recommends stopping training when the test set accuracy reaches its maximum (test set loss, excluding regularization terms, is at a minimum). One additional "tweak" to Hinton's approach is to not stop if the test_set accuracy is better (loss is smaller) than for your training set, even if the test set accuracy has stopped improving (test set loss has stopped declining). This is unlikely to gain you more than one epoch of training, but sometimes that can help a bit, especially for small test_sets. Don't do this for extremely small test sets (smaller than a representative sample set, like is sometimes used in K-folds training and cross-validation).

Improve this answer
$\endgroup$
0
$\begingroup$

you cannot use early stopping and K-fold cross validation in combination. because the early stopping select best model from validation set, the performance needs to be verified by test set. but in K-fold cross validation, there is not test set, if you using early stopping to select best model from validation set, and it will verified again in validation set. the K-fold cross validation is getting the average performance (measured by accuracy) of best model, and it is meanless.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The number of training iterations can be treated as a hyperparameter, and selected using cross validation, just as any other hyperparameter. One could reasonably call this "early stopping". $\endgroup$
    – user20160
    Aug 8, 2018 at 12:28
  • $\begingroup$ when using early stopping in K fold cross validation, the number of epoch is fixed by validation set and different for each split. This will make the network choose best model in each splits, which is not stand for the average performance. $\endgroup$
    – tianyu zhou
    Aug 8, 2018 at 14:07
  • 2
    $\begingroup$ The way to do it would be 1) Train for some number of iterations on every fold (e.g. up to a given maximum). 2) Compute the average validation set error (across folds) as a function of the number of iterations. 3) Select the number of iterations that minimizes the average validation set error. $\endgroup$
    – user20160
    Aug 8, 2018 at 14:36
  • 1
    $\begingroup$ Here is my solution:1. shuffle all data, 2. do 10-fold by sklearn, kfold = StratifiedKFold(n_splits=10, shuffle=False) 3, set callback: callbacks_list = [EarlyStopping(monitor='val_loss', patience=50, verbose=0, mode='min')] 4. choose best model by early stopping. history = model.fit(X[train], Y[train], epochs=250, batch_size=512, verbose=1 ,callbacks=callbacks_list, validation_split=0.1,shuffle=False) set another auto split validation set from X[train],Y[train]. 5. test the performance by X[test]andY[test] : scores = model.evaluate(X[test], Y[test], verbose=0) $\endgroup$
    – tianyu zhou
    Aug 8, 2018 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.