I'm faced with a high dimensional (samples=148, features=20000), supervised binary classification problem. Which I would like to approach with an ensemble of classifiers, that will classify using a majority vote. To avoid serious overfitting, I would like to use the recursive feature elimination protocol (optionally: some univariate selection) in caret. However it seems this will result in relatively high runtime. I've ran the following code on a node with 16 cores for about 10 days.
trainRows <- createDataPartition(classes, p=.80, list=FALSE) trainData <- X[trainRows, ] testData <- X[-trainRows, ] trainClasses <- classes[trainRows] testClasses <- classes[-trainRows] ctrl_rfe <- rfeControl(method = "repeatedcv", repeats = 5, verbose = TRUE, functions = caretFuncs, allowParallel = TRUE) ctrl_train <- trainControl(method = "adaptive_cv", classProbs = TRUE, repeats = 5, summaryFunction = twoClassSummary, adaptive = list(min = 5, alpha = 0.05, method = "gls", complete = TRUE), allowParallel = TRUE) radCostSVM_rfe <- rfe(trainData, trainClasses, sizes = seq(1, ncol(trainData), 100), rfeControl = ctrl_rfe, trControl = ctrl_train, method = "svmRadialCost", tuneLength = 10)
To speed up things, I've used adaptive resampling and select 100 features in each rfe iteration. Is there a way of speeding this up any further, other than decreasing fold size and number of repeats in my cross-validation? How would SBF (Selection By Filtering) compare to that approach? A simple dimensionality reduction technique like a PCA, is no option for me, because apart from being unsupervised, it would represent my features in linear combinations, which would decrease the interpretability of my results.
Any help and sharing of experiences would be highly appreciated.