# Using caret::sbf to apply feature selection where features are selected over different threshold scores

I'm aiming to use caret::sbf to filter a large number of predictors before using different machine learning models to predict a binary outcome. I would like to filter for variables that are identified as significant in at least 70% of LGOCV instances. However, I am unsure how to articulate this in the caretSBF score function. Below is an example of how it might work.

svmSBF <- caretSBF
svmSBF$$summary <- function(...) c(twoClassSummary(...), defaultSummary(...)) #svmSBF$$score <- ??
svmSBF$filter <- function(score, x, y) score > 70 data <- twoClassSim(n = 100, linearVars = 300) fit <- sbf( form = Class ~ ., data = data, method = "svmLinear", tuneGrid=expand.grid(C = 2^c(seq(-25,10,.1))), preProc = c("center", "scale"), trControl = trainControl(method = "repeatedcv", number = 10, repeats = 10, classProbs = TRUE, savePredictions = TRUE), sbfControl = sbfControl(method = "LGOCV", number = 100, p = .8, functions = svmSBF, saveDetails = TRUE))  Any help would be greatly appreciated. • the scores doesn't work this way. If you look at svmSBF$score you see that if it is a factor, an anova is used else it's a gam. They go by p-value – StupidWolf Apr 7 '20 at 23:23
• I kind of understand what you to take what is selected 70% of the time, for this you have to run it independently, like run sbf and ignoring the train results. Then run train on the variables you select – StupidWolf Apr 7 '20 at 23:25

We can set up the model using a dataset Sonar:

library(caret)
library(mlbench)

svmSBF <- caretSBF
data(Sonar)
data <- Sonar


With that settings, SBF filters the values using anovaScores if it is a classification or gamScores if it's continuous. The filter is based on p-value (see $score below): caretSBF$score
function (x, y)
{
if (is.factor(y))
anovaScores(x, y)
else gamScores(x, y)
}

caretSBF$filter function (score, x, y) score <= 0.05  For what you need, you need to run SBF first to get the filtered variables over the train set and use that to retrain your model: sbf_fit <- sbf( form = Class ~ ., data = data, method = "svmLinear", tuneGrid=data.frame(C=1), trControl = trainControl(), sbfControl = sbfControl(method = "LGOCV", number = 20, p = .8, functions = svmSBF, saveDetails = TRUE))  We calculate the proportion of times each variable is kept: prop_included = rowMeans(sapply(sbf_fit$$variables,function(i)sbf_fit$$coefnames %in% i)) selected = sbf_fit$coefnames[prop_included > 0.7]


mdl <- train(