0
$\begingroup$

I'm new to ANN and deep learning, and I have come across the term k-fold cross-validation, which from my understanding is, when you split your dataset into small parts, to find the best parameters to train your model from your dataset (because obviously, we can't use them all). This concept was a bit confusing to me because it sounded extremely similar to iterations when we talk about training a model.

iterations in the sense of

We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.

Are these concepts similar, and used in different scenarios, or are they completely separated?

$\endgroup$
1
  • $\begingroup$ These are different concepts. Batches are part of the parameter estimation, SGD for example. The parameters of the model are updated with each batch, instead of once for the whole dataset. K-fold cross-validation is a procedure that helps to fix hyper-parameters. It is a variation on splitting a data set into train and validation sets; this is done to prevent overfitting. Keywords are bias and variance there. $\endgroup$
    – spdrnl
    May 19, 2020 at 9:51

1 Answer 1

1
$\begingroup$

In the first place, cross validation is a technique to estimate certain aspects of predictive performance of a model.
These performance estimates can then be used either as estimate of generalization error xor to select among several "candidate" models (typically corresponding to different hyperparameter sets).

Cross validation works by training a number of surrogate models which are assumed to be approximately equal to the model (I'll refer to this as the model) whose performance we actually want to know. These surrogate models are trained on training data sets that are again approximately equal to the model's training data: they differ by leaving out a few cases (which are then used as test cases for this surrogate model). For almost all cross validation procedures* the training sets for the surrogate models have substantial overlap in cases (this is desired: they are all assumed to approximate the same model, so they should be similar among themselves).

Cross validation can have iterations (aka repetitions), too: in that case, the whole procedure from the splitting on is repeated. Again, we have substantial overlap between training subsets for the surrogate models.

In contrast, AFAIK iterations during a training epoch of a neural network use batches of disjoint training cases.


* the exception is 2-fold where we get two models trained and tested on mutually independent subsets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.