Using caret::sbf to apply feature selection and classification

Question

I'm aiming to use caret::sbf to filter a large number of predictors before using different machine learning models to predict a binary outcome. I would also like to optimise tuning parameters and do some internal cross-validation. Below is an example code using SVM with a simple grid search and cv:

data = twoClassSim(n = 100, linearVars = 300)

fit <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid=expand.grid(C = 2^c(seq(-25,10,.1))),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "none"),
  sbfControl = sbfControl(functions = caretSBF, verbose = FALSE, method = 'cv', number = 10))

However, I get the following error code:

Error in { : 
  task 1 failed - "Only one model should be specified in tuneGrid with no resampling"

Can anyone explain why this is happening? I have been through the caret vignettes and can't seem to find an answer. Any help would be greatly appreciated.

Answer 1

With sbf, you specify the method to estimate what is the variability in filtering / selecting variables. In your example, you used 10 fold cv.

The variables are selected over the whole training dataset, and to tune your parameters, you need to specify again the method and number.

So something like below:

library(caret)

set.seed(100)
data = twoClassSim(n = 100, linearVars = 300)

mdl <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid = data.frame(C=c(0.1,1,10)),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "cv",number=3),
  sbfControl = sbfControl(functions = caretSBF, verbose = FALSE, method = 'cv', number = 3))

Then you can see the selected variables under 3 fold cv:

mdl$variables
$selectedVars
 [1] "Linear030"  "Linear041"  "Linear062"  "Linear103"  "Linear105" 
 [6] "Linear154"  "Linear160"  "Linear163"  "Linear197"  "Linear219" 
[11] "Linear224"  "Linear227"  "Linear242"  "Linear254"  "Linear258" 
[16] "Linear260"  "Linear270"  "Linear275"  "Linear288"  "Nonlinear1"
[21] "Nonlinear3"

$selectedVars
 [1] "Linear017" "Linear020" "Linear026" "Linear030" "Linear050" "Linear073"
 [7] "Linear096" "Linear126" "Linear129" "Linear134" "Linear153" "Linear180"
[13] "Linear181" "Linear193" "Linear194" "Linear196" "Linear211" "Linear223"
[19] "Linear258" "Linear260" "Linear275"

$selectedVars
 [1] "Linear008" "Linear030" "Linear058" "Linear074" "Linear079" "Linear112"
 [7] "Linear148" "Linear186" "Linear203" "Linear247" "Linear288" "Linear295"

The model trained uses, as mentioned before, filter on the whole train dataset, which you can find:

mdl$optVariables
 [1] "Linear017"  "Linear026"  "Linear027"  "Linear030"  "Linear041" 
 [6] "Linear073"  "Linear074"  "Linear103"  "Linear148"  "Linear160" 
[11] "Linear163"  "Linear196"  "Linear223"  "Linear247"  "Linear258" 
[16] "Linear260"  "Linear270"  "Linear275"  "Linear288"  "Nonlinear1"

And, the results of cv to find the best parameter:

Support Vector Machines with Linear Kernel 

100 samples
 20 predictor
  2 classes: 'Class1', 'Class2' 

Pre-processing: centered (20), scaled (20) 
Resampling: Cross-Validated (3 fold) 
Summary of sample sizes: 67, 66, 67 
Resampling results across tuning parameters:

  C     Accuracy   Kappa    
   0.1  0.6996435  0.3837282
   1.0  0.6693405  0.3171290
  10.0  0.6200238  0.2190648

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was C = 0.1.