2
$\begingroup$

I want to use caret to compare two different classification algorithms. For example SVM and Elastic net.

I want to put aside some samples for test set and then use the rest of the samples for training the model, which involves tuning some parameters (like alpha and lambda for elastic net) for which I use cross validation as well. But once I fixed the model, I want to calculate it's performance on the test set (that was not used for parameter tuning).

I want to do 10 fold cross validation to calculate performance of each method (SVM vs Enet). Note that this cross validation should be in an outer loop than the cross validation used for parameter tuning for each method.

I was wondering if I can use Caret to do that.

In the pseudo code given by Caret (http://caret.r-forge.r-project.org/training.html) the first loop (line 2) is for each parameter set, and the next loop (line 3) is for each resampling iteration. Is it possible to add another outer loop such as (Line 0: for each resampling iteration) to this schema ?


EDIT: @topepo: yes we have a limited data, so we want to have multiple test sets. Use 90% data for training (including parameter tuning) and then calculate the performance on the 10% remaining data. then repeat this 10 times with different partitionings of the data.

$\endgroup$
  • $\begingroup$ Do you want multiple test sets? Otherwise, I don't see what the outer loop does. Pseudo-code would help. $\endgroup$ – topepo May 20 '14 at 0:41
6
$\begingroup$

Give this a try (modify the details as needed)

library(caret)

library(mlbench)
data(Sonar)

set.seed(1)
splits <- createFolds(Sonar$Class, returnTrain = TRUE)
    results <- lapply(splits, 
                      function(x, dat) {
                        holdout <- (1:nrow(dat))[-unique(x)]
                        data.frame(index = holdout, 
                                   obs = dat$Class[holdout])
                  },
                  dat = Sonar)
mods <- vector(mode = "list", length = length(splits))

## foreach or lapply would do this faster
for(i in seq(along = splits)) {
  in_train <- unique(splits[[i]])
  set.seed(2)
  mod <- train(Class ~ ., data = Sonar[in_train, ],
               method = "svmRadial",
               preProc = c("center", "scale"),
               tuneLength = 8)
  results[[i]]$pred <- predict(mod, Sonar[-in_train, ])
  mods[[i]] <- mod
}

lapply(results, defaultSummary)
$\endgroup$
  • $\begingroup$ Sorry to ask what is probably a very simple question but why is it required to pass x and splits[[i]] to the method unique() in your example? Why not simply use x or splits[[i]] on its own? As far as I understand unique returns the same vector but without duplicates. Also, I thought that each of the ten folds in splits is a list of row position integers for the training data (given returnTrain = TRUE) which would not contain duplicates? Sorry again if I am not understanding. My knowledge of R is not great. Thanks again for your Caret package! $\endgroup$ – samleighton87 Mar 28 '17 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.