# What is a cross validation fold, or does this phrase not make sense?

I understand the concept of k-fold cross validation, but I do not understand what a "fold" means. Quoting from the linked page on wikipedia:

The cross-validation process is then repeated k times (the folds)

This seems very vague. Does the 'fold' refer to each repeat of the process? Or is it a noun to refer to the paired training-testing dataset?

• I confess that I don't even know what cross-validation is, but isn't this just the usual English-language meaning of "$k$-fold" meaning "$k$ times", as in "There has been a fourfold increase in violent crime since the legalization of hand-held nuclear weapons." – David Richerby Sep 5 '16 at 20:43
• that is a very good point. yet, as you can see in the answer the folds can be used to refer to the data. – Alex Sep 6 '16 at 23:39
• Yeah, though that sounds an awful lot like a misunderstanding by a non-native speaker that's caught on. – David Richerby Sep 7 '16 at 7:24

The wording is definitely awkward there.

Recall that cross-validation partitions a dataset into $K$ roughly equal "sub-datasets." Each one of these "sub-datasets" is called a "fold." $K$-fold cross validation requires re-fitting a model $K$ times, omitting exactly one fold from the data each time, so the term "fold" can also be used to refer to each repetition.

Since there is a one-to-one correspondence between folds and repetitions, there usually isn't a problem with this lax terminology. It is usually apparent from the context which usage is intended, and other times it doesn't make a difference.

• Right, so this interpretation makes each of the disjoint test sets a 'fold'. Thus, the training data can be referred to as 'data not in the fold'. Do you have a reference for this? – Alex Sep 5 '16 at 3:53
• And yes, "out-of-fold" is a valid term – shadowtalker Sep 5 '16 at 3:55
• The $k$ models are sometimes referred to as surrogate models, reference e.g. Braga-Neto UM, Dougherty ER.: Is cross-validation valid for small-sample microarray classification? Bioinformatics. 2004 Feb 12;20(3):374-80. dx.doi.org/10.1093/bioinformatics/btg419. "folds" is often used in distinction to a "run" (iteration/repetition) of the cross validation (a run then consists of $k$ folds in the "procedure" meaning) – cbeleites Sep 5 '16 at 11:03
• +1 but "data not in the fold" phrase sounds very awkward and extremely unclear @Alex. Don't use it. – amoeba Sep 6 '16 at 13:26
• I often use "fold" lazily to mean each chunk of the data set. As in "fold 5 is imbalanced compared to the rest of the data" – shadowtalker Sep 7 '16 at 0:29

"Fold" refers to a partition (in the set-theoretic meaning of the word) of the sample, $S$, into a training set, $T_j$, and validation set, $V_j$. This means:

1. $T_j \cap V_j = \emptyset$,
2. $T_j \cup V_j = S$,

($1 \leq j \leq k$).

Note that in "classic" $k$-fold cross-validation (CV) an additional condition is placed on the validation sets:

1. $V_i \cap V_j = \emptyset$ ($i \neq j$).

Finally, note that the $k$ in classic $k$-fold CV controls both the number of times the train-validate procedure is carried out, as well as the size of the validation and training sets: $|V_j| \approxeq \frac{1}{k} |S|$, therefore $|T_j| \approxeq \frac{k-1}{k} |S|$.