7
$\begingroup$

I understand the concept of k-fold cross validation, but I do not understand what a "fold" means. Quoting from the linked page on wikipedia:

The cross-validation process is then repeated k times (the folds)

This seems very vague. Does the 'fold' refer to each repeat of the process? Or is it a noun to refer to the paired training-testing dataset?

$\endgroup$
  • 1
    $\begingroup$ I confess that I don't even know what cross-validation is, but isn't this just the usual English-language meaning of "$k$-fold" meaning "$k$ times", as in "There has been a fourfold increase in violent crime since the legalization of hand-held nuclear weapons." $\endgroup$ – David Richerby Sep 5 '16 at 20:43
  • $\begingroup$ that is a very good point. yet, as you can see in the answer the folds can be used to refer to the data. $\endgroup$ – Alex Sep 6 '16 at 23:39
  • $\begingroup$ Yeah, though that sounds an awful lot like a misunderstanding by a non-native speaker that's caught on. $\endgroup$ – David Richerby Sep 7 '16 at 7:24
9
$\begingroup$

The wording is definitely awkward there.

Recall that cross-validation partitions a dataset into $K$ roughly equal "sub-datasets." Each one of these "sub-datasets" is called a "fold." $K$-fold cross validation requires re-fitting a model $K$ times, omitting exactly one fold from the data each time, so the term "fold" can also be used to refer to each repetition.

Since there is a one-to-one correspondence between folds and repetitions, there usually isn't a problem with this lax terminology. It is usually apparent from the context which usage is intended, and other times it doesn't make a difference.

$\endgroup$
  • $\begingroup$ Right, so this interpretation makes each of the disjoint test sets a 'fold'. Thus, the training data can be referred to as 'data not in the fold'. Do you have a reference for this? $\endgroup$ – Alex Sep 5 '16 at 3:53
  • 1
    $\begingroup$ And yes, "out-of-fold" is a valid term $\endgroup$ – shadowtalker Sep 5 '16 at 3:55
  • 1
    $\begingroup$ The $k$ models are sometimes referred to as surrogate models, reference e.g. Braga-Neto UM, Dougherty ER.: Is cross-validation valid for small-sample microarray classification? Bioinformatics. 2004 Feb 12;20(3):374-80. dx.doi.org/10.1093/bioinformatics/btg419. "folds" is often used in distinction to a "run" (iteration/repetition) of the cross validation (a run then consists of $k$ folds in the "procedure" meaning) $\endgroup$ – cbeleites Sep 5 '16 at 11:03
  • 1
    $\begingroup$ +1 but "data not in the fold" phrase sounds very awkward and extremely unclear @Alex. Don't use it. $\endgroup$ – amoeba Sep 6 '16 at 13:26
  • 1
    $\begingroup$ I often use "fold" lazily to mean each chunk of the data set. As in "fold 5 is imbalanced compared to the rest of the data" $\endgroup$ – shadowtalker Sep 7 '16 at 0:29
2
$\begingroup$

"Fold" refers to a partition (in the set-theoretic meaning of the word) of the sample, $S$, into a training set, $T_j$, and validation set, $V_j$. This means:

  1. $T_j \cap V_j = \emptyset$,
  2. $T_j \cup V_j = S$,

($1 \leq j \leq k$).

Note that in "classic" $k$-fold cross-validation (CV) an additional condition is placed on the validation sets:

  1. $V_i \cap V_j = \emptyset$ ($i \neq j$).

Finally, note that the $k$ in classic $k$-fold CV controls both the number of times the train-validate procedure is carried out, as well as the size of the validation and training sets: $|V_j| \approxeq \frac{1}{k} |S|$, therefore $|T_j| \approxeq \frac{k-1}{k} |S|$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.