Early stopping vs cross validation I'm currently using early stopping in my work to prevent over fitting. Specifically those taken form Early Stopping But When?.
I'm now wanting to compare to other classification algorithms where it appears that 10 fold cross validation is widely used.
However I'm confused about whether cross validation is a method for preventing over fitting or selecting good parameters. (or maybe this is one and the same?). I'm also confused whether early stopping methods and cross validation can be used in place of one another or in combination.
So the question is: what is the relationship between early stopping and cross validation?
 A: Cross Validation is a method for estimating the generalisation accuracy of a supervised learning algorithm.
Early stopping is a method for avoiding overfitting and requires a method to assess the relationship between the generalisation accuracy of the learned model and the training accuracy.
So you could use cross validation to replace the validation set, mentioned in the paper you cite,  within an early stopping framework. Ten fold cross validation for instance would be more accurate than using a single validation set, and would normally be a better  estimate of generalisation error.
So to summarise, cross validation is a generalisation accuracy measure which could be used as part of an early stopping framework.
A: In addition to the two generalization approaches you mention, there are many others.


*

*adding regularization terms in your loss (cost) function that minimize the number and magnitude of your nonzero model parameters

*randomly dropping (zeroing out) some portion of the weights/parameters in your model with each training epoch

*adding a layer of stochastic nodes in your model (e.g., sampling from the "probabilities" given by the outputs of logistic functions)


Many of these approaches (including your cross-validation and early-stopping approaches) can be combined together to maximize model performance on unseen data (generalization performance).
One note on the early-stopping approach. For neural nets, Geoffrey Hinton recommends stopping training when the test set accuracy reaches its maximum (test set loss, excluding regularization terms, is at a minimum). One additional "tweak" to Hinton's approach is to not stop if the test_set accuracy is better (loss is smaller) than for your training set, even if the test set accuracy has stopped improving (test set loss has stopped declining). This is unlikely to gain you more than one epoch of training, but sometimes that can help a bit, especially for small test_sets. Don't do this for extremely small test sets (smaller than a representative sample set, like is sometimes used in K-folds training and cross-validation).
A: you cannot use early stopping and K-fold cross validation in combination. because the early stopping select best model from validation set, the performance needs to be verified by test set. but in K-fold cross validation, there is not test set, if you using early stopping to select best model from validation set, and it will verified again in validation set. the K-fold cross validation is getting the average performance (measured by accuracy) of best model, and it is meanless. 
