Using caret::sbf to apply feature selection where features are selected over different threshold scores

Question

I'm aiming to use caret::sbf to filter a large number of predictors before using different machine learning models to predict a binary outcome. I would like to filter for variables that are identified as significant in at least 70% of LGOCV instances. However, I am unsure how to articulate this in the caretSBF score function. Below is an example of how it might work.

svmSBF <- caretSBF
svmSBF$summary <- function(...) c(twoClassSummary(...), defaultSummary(...))
#svmSBF$score <- ??
svmSBF$filter <- function(score, x, y) score > 70

data <- twoClassSim(n = 100, linearVars = 300)

fit <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid=expand.grid(C = 2^c(seq(-25,10,.1))),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "repeatedcv",
                  number = 10,
                  repeats = 10,
                  classProbs = TRUE,
                  savePredictions = TRUE),
  sbfControl = sbfControl(method = "LGOCV",
                 number = 100,
                 p = .8,
                 functions = svmSBF,
                 saveDetails = TRUE))

Any help would be greatly appreciated.

Answer 1

We can set up the model using a dataset Sonar:

library(caret)
library(mlbench)

svmSBF <- caretSBF
data(Sonar)
data <- Sonar

With that settings, SBF filters the values using anovaScores if it is a classification or gamScores if it's continuous. The filter is based on p-value (see $score below):

caretSBF$score
function (x, y) 
{
    if (is.factor(y)) 
        anovaScores(x, y)
    else gamScores(x, y)
}

caretSBF$filter
function (score, x, y) 
score <= 0.05

For what you need, you need to run SBF first to get the filtered variables over the train set and use that to retrain your model:

sbf_fit <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid=data.frame(C=1),
  trControl = trainControl(),
  sbfControl = sbfControl(method = "LGOCV",
                 number = 20,
                 p = .8,
                 functions = svmSBF,
                 saveDetails = TRUE))

We calculate the proportion of times each variable is kept:

prop_included = rowMeans(sapply(sbf_fit$variables,function(i)sbf_fit$coefnames %in% i))

selected = sbf_fit$coefnames[prop_included > 0.7]

And you train your model:

mdl <- train(
       form = Class ~ .,
       data = data[,c(selected,"Class")], 
       method = "svmLinear", 
      trControl = trainControl(method="cv",number=5))