I'm faced with a high dimensional (samples=148, features=20000), supervised binary classification problem. Which I would like to approach with an ensemble of classifiers, that will classify using a majority vote. To avoid serious overfitting, I would like to use the recursive feature elimination protocol (optionally: some univariate selection) in caret. However it seems this will result in relatively high runtime. I've ran the following code on a node with 16 cores for about 10 days.

trainRows         <- createDataPartition(classes, p=.80, list=FALSE)
trainData         <- X[trainRows, ]
testData          <- X[-trainRows, ]
trainClasses      <- classes[trainRows]
testClasses       <- classes[-trainRows]

ctrl_rfe <- rfeControl(method = "repeatedcv",
                       repeats = 5,
                       verbose = TRUE,
                       functions = caretFuncs,
                       allowParallel = TRUE)
ctrl_train <- trainControl(method = "adaptive_cv",
                           classProbs = TRUE,
                           repeats = 5,
                           summaryFunction = twoClassSummary,
                           adaptive = list(min = 5, alpha = 0.05, 
                                           method = "gls", complete = TRUE),
                           allowParallel = TRUE)

radCostSVM_rfe <- rfe(trainData, trainClasses, 
                      sizes      = seq(1, ncol(trainData), 100), 
                      rfeControl = ctrl_rfe,
                      trControl  = ctrl_train,
                      method     = "svmRadialCost",
                      tuneLength = 10)

To speed up things, I've used adaptive resampling and select 100 features in each rfe iteration. Is there a way of speeding this up any further, other than decreasing fold size and number of repeats in my cross-validation? How would SBF (Selection By Filtering) compare to that approach? A simple dimensionality reduction technique like a PCA, is no option for me, because apart from being unsupervised, it would represent my features in linear combinations, which would decrease the interpretability of my results.

Any help and sharing of experiences would be highly appreciated.


1 Answer 1


You are fitting a @#$^ ton of models, even with the adaptive resampling. You can do it, but the tuning will take a lot of time regardless.

You would probably be better off fitting a model with built-in feature selection instead of using a feature selection wrapper.

sbf could work, and it probably wouldn't be hard to try, but there is still model tuning.

  • $\begingroup$ The link is dead! $\endgroup$ Commented Oct 8, 2019 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.