2
$\begingroup$

I have some data sets that goes from $n=100$ to $n=700$. How can you choose an adequate K number to do a repeated k-fold cross validation. Also which will be a good number for the number of repetitions. So far the $R^2$ of my linear regressions using all the data set are good (<0.7), I though about using this kind of validation so I can use the model in new data sets. I will appreciate any input

Improve this question
$\endgroup$
2
  • $\begingroup$ Hi Ursula, welcome to cross validated! I think your question may be answered already: stats.stackexchange.com/q/27730/4598 and stats.stackexchange.com/questions/61546/…. I'll vote to close it as duplicate since we try to avoid saying the same all over again and instead prefer to ultimately point new (future) users with sufficiently similar questions to the same answer. $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:17
  • $\begingroup$ That being said, n between 100 and 700 is not so extremely small that the more general "choice of k" would not apply because of small sample size. $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:19

1 Answer 1

Reset to default
1
$\begingroup$

For smaller datasets usually the recomendation is to use Leave One Out Cross Validation (LOOCV). In this case we have $K=1$, but we repeat this "$N$" times (repetitions) throughout the data. I will a link here for further detail:

https://en.wikipedia.org/wiki/Cross-validation_(statistics)

Note: there is no "correct" answer for the size of $K$ or $N$, the main point of cross-validation is to ensure that your training/validation splits are as representative of the variety in the underlying population distribution as possible. e.g. if your samples are all biased compare to the population distribution no amount of cross validation will help -> you need to go back and collect more data which is unbiased.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ sorry, but: with LOO, k = N. Also, careful about the term repetitions: this is often used to denote new splits into another k subsets/folds. In that sense, repetitions are not possible for LOO. $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:13
  • 1
    $\begingroup$ In addition, while I agree that sometimes data sets are so small that LOO is the only option, the known drawbacks (see the q&a I linked above) of LOO are sufficiently serious that I do LOO pretty much only with independent sample size <= 5 or so (in that case, independent sample size is typically a high-level clustering in the data that creates dependence between more observations/cases). $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:21
  • $\begingroup$ isn't LOOCV not just repeating cross validation "N" times? I'm not quite sure why the repetition part is important in this regard, I thought k-fold CV implies repeating 1 split validation K times $\endgroup$
    – tisPrimeTime
    Commented Apr 17, 2020 at 14:21
  • $\begingroup$ The usual description is that one iterates over k folds in k-fold cross validation. r repetitions then means doing a total of r * k folds. The difference is that the k folds of the same repetition have disjunct test sets, whereas of the folds of 2 different repetitios exactly one from the one repetition and one from the other repetition share any given case as test case. $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:23
  • $\begingroup$ Also, one repetition (run) of cross validation comprises evaluating all k folds. (It's just terminology, your repetitions are usually called folds, but repetitions are used for something else) $\endgroup$
    – cbeleites
    Commented Apr 17, 2020 at 14:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.