2
$\begingroup$

I want to use caret to compare two different classification algorithms. For example SVM and Elastic net.

I want to put aside some samples for test set and then use the rest of the samples for training the model, which involves tuning some parameters (like alpha and lambda for elastic net) for which I use cross validation as well. But once I fixed the model, I want to calculate it's performance on the test set (that was not used for parameter tuning).

I want to do 10 fold cross validation to calculate performance of each method (SVM vs Enet). Note that this cross validation should be in an outer loop than the cross validation used for parameter tuning for each method.

I was wondering if I can use Caret to do that.

In the pseudo code given by Caret (http://caret.r-forge.r-project.org/training.html) the first loop (line 2) is for each parameter set, and the next loop (line 3) is for each resampling iteration. Is it possible to add another outer loop such as (Line 0: for each resampling iteration) to this schema ?


EDIT: @topepo: yes we have a limited data, so we want to have multiple test sets. Use 90% data for training (including parameter tuning) and then calculate the performance on the 10% remaining data. then repeat this 10 times with different partitionings of the data.

$\endgroup$
1
  • $\begingroup$ Do you want multiple test sets? Otherwise, I don't see what the outer loop does. Pseudo-code would help. $\endgroup$
    – topepo
    May 20, 2014 at 0:41

1 Answer 1

6
$\begingroup$

Give this a try (modify the details as needed)

library(caret)

library(mlbench)
data(Sonar)

set.seed(1)
splits <- createFolds(Sonar$Class, returnTrain = TRUE)
    results <- lapply(splits, 
                      function(x, dat) {
                        holdout <- (1:nrow(dat))[-unique(x)]
                        data.frame(index = holdout, 
                                   obs = dat$Class[holdout])
                  },
                  dat = Sonar)
mods <- vector(mode = "list", length = length(splits))

## foreach or lapply would do this faster
for(i in seq(along = splits)) {
  in_train <- unique(splits[[i]])
  set.seed(2)
  mod <- train(Class ~ ., data = Sonar[in_train, ],
               method = "svmRadial",
               preProc = c("center", "scale"),
               tuneLength = 8)
  results[[i]]$pred <- predict(mod, Sonar[-in_train, ])
  mods[[i]] <- mod
}

lapply(results, defaultSummary)
$\endgroup$
1
  • $\begingroup$ Sorry to ask what is probably a very simple question but why is it required to pass x and splits[[i]] to the method unique() in your example? Why not simply use x or splits[[i]] on its own? As far as I understand unique returns the same vector but without duplicates. Also, I thought that each of the ten folds in splits is a list of row position integers for the training data (given returnTrain = TRUE) which would not contain duplicates? Sorry again if I am not understanding. My knowledge of R is not great. Thanks again for your Caret package! $\endgroup$ Mar 28, 2017 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.