I am computing a SVM-RFE model with the rfe function of the caret package, but I am a bit confused about the results. My code is:

fiveStats <- function(...) c(twoClassSummary(...), defaultSummary(...))
svmFuncs <- caretFuncs
svmFuncs$summary <- fiveStats

FSctrl <- rfeControl(method = "repeatedcv",
                   repeats = 5,
                   verbose = TRUE,
                   functions = svmFuncs,
                   index = createMultiFolds(TrData[, 1], times = 5),
                   saveDetails = TRUE)

TRctrl = trainControl(method = "LGOCV",
                      number = 50, p = 0.7,
                      savePredictions = TRUE,
                      classProbs = TRUE,
                      verboseIter = FALSE)

svmRFE_NG <- rfe(x = TrData[, 2:43],
               y = TrData[, 1],
               sizes = seq(1,42),
               metric = "ROC",
               rfeControl = FSctrl,
               ## Options to train()
               method = "svmLinear",
               tuneGrid = expand.grid(C = 10.^(-2:2)),
               preProc = c("center", "scale"),
               ## Inner resampling process
               trControl = TRctrl)

I would like to compute some average metrics (ROC curve, AUC, sensitivity...) from the cross-validation data (training), but I am not sure where to look at:


> head(svmRFE_NG$pred)
              pred    BREAST      LUNG    obs Variables    Resample rowIndex
predictions.1 LUNG 0.3075494 0.6924506   LUNG        42 Fold01.Rep1       33
predictions.2 LUNG 0.1106591 0.8893409   LUNG        42 Fold01.Rep1       37
predictions.3 LUNG 0.2504079 0.7495921 BREAST        42 Fold01.Rep1       41
predictions.4 LUNG 0.1174505 0.8825495   LUNG        42 Fold01.Rep1       44
predictions.5 LUNG 0.1238329 0.8761671 BREAST        42 Fold01.Rep1       46
predictions.6 LUNG 0.2917743 0.7082257   LUNG        41 Fold01.Rep1       33

or svmRFE_NG$fit$pred:

> head(svmRFE_NG$fit$pred)
    pred    obs    BREAST      LUNG rowIndex    C   Resample
1 BREAST BREAST 0.7434318 0.2565682        4 0.01 Resample01
2   LUNG   LUNG 0.2731751 0.7268249        6 0.01 Resample01
3   LUNG BREAST 0.4431675 0.5568325        8 0.01 Resample01
4 BREAST BREAST 0.8306861 0.1693139       11 0.01 Resample01
5 BREAST BREAST 0.8404291 0.1595709       15 0.01 Resample01
6   LUNG   LUNG 0.3936469 0.6063531       19 0.01 Resample01

To my knowledge, the final model is stored in svmRFE_NG$fit. Should I take these results (for C = best tuning parameter) or should I work with the svmRFE_NG$pred results (for Variables = optimal size)?


From looking at the RFE examples at Max's page, svmRFE_NG$resample and svmRFE_NG$pred$Resample (and their counterparts in svmRFE_NG$fit), I'd say this depends on which characteristics you want to look at.

svmRFE_NG seems to contain cross validation results of using different variables, so could be used for statistics about using different variables (consider e.g. svmRFE_NG$variables too). Not all information seems to be preserved here though, like the performance of a specific combination of variables, if I didn't just overlook this.

In contrast, svmRFE_NG$fit seems to contain cross validation results for different hyperparameters of the "final model" (the best performing combination of features and hyperparameters). So those can be used for the more classic statistic about the final model you obtained from the whole process.

  • $\begingroup$ That link is dead. $\endgroup$ – Ekaba Bisong Nov 25 '16 at 6:12

Not directly relevant to your question but I'm not sure if you need trControl since you already used rfeControl. According to the difference between train and rfe (https://stackoverflow.com/a/32968300/6650689), since you are already using rfe (a wrapper around train) it may not be necessary to have trControl because you are essentially using rfe rather than a generalized train.

  • $\begingroup$ Should be a comment rather than an answer. $\endgroup$ – Michael Chernick Oct 31 '17 at 2:38
  • 1
    $\begingroup$ This is not correct. You can mix trControl and rfe/rfeControl $\endgroup$ – Jerry T May 28 '18 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.