25
$\begingroup$

This is similar to question Caret re-sampling methods, although that really never answered this part of the question in an agreed upon way.

caret's train function offers cv and repeatedcv. What is the difference in say doing:

MyTrainControl=trainControl(
    method = "cv",
    number=5,
    repeats=5
)

vs

MyTrainControl=trainControl(
   method = "repeatedcv",
   number=5,
   repeats=5
)

I understand cv breaks the set into k-folds (parameter number), and then starts over and runs it parameters repeats number of times.

The only thing I could think of is that maybe regular cv with repeats uses the same exact indexes for the folds each time? essentially running the cv on the same exact folds each time, vs perhaps repeatedcv selects new folds each times?

Can someone clarify?

$\endgroup$
  • $\begingroup$ I wonder there are some more methods as well.. need some basic level understanding of each, is there anywhere I could find that? thanks. $\endgroup$ – Manoj Kumar Jun 19 '18 at 14:04
  • $\begingroup$ In create multi fold, code iterates over multiple times (given by repeats in train Control() syntax in R) for each of the k cross fold (given by number). In cross fold, while using CV, it is a one time process on each of the fold (set by using numbers in train control()). $\endgroup$ – Nitesh Jindal Apr 2 at 5:42
29
$\begingroup$

According to the caret manual, page 22, the parameter repeats only applies when the method is set to repeatedcv, so no repetition is performed when the method is set to cv. So the difference between both methods is indeed that repeatedcv repeats and cv does not.


Aside: Repeating a crossvalidation with exactly the same splitting will yield exactly the same result for every repetition (assuming that the model is trained in a deterministic manner), which is not only inefficient, but also dangerous when it comes to comparing the validation results for different model algorithms in a statistical manner. So be aware of this if you ever have to program a validation yourself.

$\endgroup$
3
$\begingroup$

The actual code behind these parameters can be found in the selectByFilter.R and createDataPartition.R (formerly createFolds.R) source files in the `caret/R/' folder of the package.

See these files for e.g. here and here (beware these permalinks may eventually point to older version of the code). For convenience the relevant snippets (as of version 6.0-78 c. Nov 2017) are show below

In selectByFilter.R c. line 157

sbf <- function (x, ...) UseMethod("sbf")
... 

"sbf.default" <-
  function(x, y,
           sbfControl = sbfControl(), ...)
  {
    ...

    if(is.null(sbfControl$index)) sbfControl$index <- switch(
      tolower(sbfControl$method),
      cv = createFolds(y, sbfControl$number, returnTrain = TRUE),
      repeatedcv = createMultiFolds(y, sbfControl$number, sbfControl$repeats),
      loocv = createFolds(y, length(y), returnTrain = TRUE),
      boot =, boot632 = createResample(y, sbfControl$number),
      test = createDataPartition(y, 1, sbfControl$p),
      lgocv = createDataPartition(y, sbfControl$number, sbfControl$p))
...

In createDataPartition.R c. line 227

createMultiFolds <- function(y, k = 10, times = 5) {
  if(class(y)[1] == "Surv") y <- y[,"time"]
  prettyNums <- paste("Rep", gsub(" ", "0", format(1:times)), sep = "")
  for(i in 1:times) {
    tmp <- createFolds(y, k = k, list = TRUE, returnTrain = TRUE)
    names(tmp) <- paste("Fold",
                        gsub(" ", "0", format(seq(along = tmp))),
                        ".",
                        prettyNums[i],
                        sep = "")
    out <- if(i == 1) tmp else c(out, tmp)

  }
  out
}
$\endgroup$
  • $\begingroup$ take a look at their functions....github.com/tonglu/caret/blob/master/pkg/caret/R/… $\endgroup$ – user3466398 Sep 10 '14 at 23:19
  • 7
    $\begingroup$ Could you please give more context in your answer? Links are good, but we try to avoid answers that won't stand on their own -- links can disappear. $\endgroup$ – Glen_b Sep 11 '14 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.