I run caret's recursive feature selection with randomForest. While running rfe function with method repeatedcv, I had parameter maximize = TRUE. Thus, optimal set of variables is decided based on the best RMSE metrics.

However, I would like to see the minimum "tolerable" set of predictor variables without rerunning rfe with parameter maximize = FALSE. It takes 24 hours to rerun my analysis.

Appearently, caret's function pickSizeTolerance does the trick, as is described on caret's webpage: http://caret.r-forge.r-project.org/featureselection.html

How to use the existing rfe object to get the "tolerable" set of variables?

Reproducible code:

inTrain <- createDataPartition(y = iris[,4],
                           p = .66,
                           list = FALSE)
training <- iris[ inTrain,]
testing <- iris[-inTrain,]
ctrl <- rfeControl(functions = rfFuncs, method = "repeatedcv", repeats = 5,
               verbose = TRUE, returnResamp = "all")
rfProfile <- rfe(training[,-4], training[,4], sizes = c(2,3), rfeControl = ctrl, newdata = testing[,-4])

Object rfProfile$resample includes all the metrics, but how to calculate?


locked by whuber Jun 27 '17 at 18:28

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

Read more about locked posts here.


Ok, functions pickSizeTolerance and pickSizeBest are well documented in caret's ?rfFuncs

The above written reproducible code can be further used following the documentation's example :

example <- data.frame(RMSE =rfProfile$results$RMSE, Variables = c(2:4))

## Percent Loss in performance (positive)
example$PctLoss <- (example$RMSE - min(example$RMSE))/min(example$RMSE)*100

xyplot(RMSE ~ Variables, data= example)
xyplot(PctLoss ~ Variables, data= example)

absoluteBest <- pickSizeBest(example, metric = "RMSE", maximize = FALSE)
within5Pct <- pickSizeTolerance(example, metric = "RMSE", maximize = FALSE)

cat("numerically optimal:",
        "RMSE in position",
        absoluteBest, "\n")
cat("Accepting a 1.5 pct loss:",
"RMSE in position",
within5Pct, "\n")

Not the answer you're looking for? Browse other questions tagged or ask your own question.