1
$\begingroup$

I'm aiming to use caret::sbf to filter a large number of predictors before using different machine learning models to predict a binary outcome. I would like to filter for variables that are identified as significant in at least 70% of LGOCV instances. However, I am unsure how to articulate this in the caretSBF score function. Below is an example of how it might work.

svmSBF <- caretSBF
svmSBF$summary <- function(...) c(twoClassSummary(...), defaultSummary(...))
#svmSBF$score <- ??
svmSBF$filter <- function(score, x, y) score > 70

data <- twoClassSim(n = 100, linearVars = 300)

fit <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid=expand.grid(C = 2^c(seq(-25,10,.1))),
  preProc = c("center", "scale"),
  trControl = trainControl(method = "repeatedcv",
                  number = 10,
                  repeats = 10,
                  classProbs = TRUE,
                  savePredictions = TRUE),
  sbfControl = sbfControl(method = "LGOCV",
                 number = 100,
                 p = .8,
                 functions = svmSBF,
                 saveDetails = TRUE))

Any help would be greatly appreciated.

$\endgroup$
  • $\begingroup$ the scores doesn't work this way. If you look at svmSBF$score you see that if it is a factor, an anova is used else it's a gam. They go by p-value $\endgroup$ – StupidWolf Apr 7 at 23:23
  • 1
    $\begingroup$ I kind of understand what you to take what is selected 70% of the time, for this you have to run it independently, like run sbf and ignoring the train results. Then run train on the variables you select $\endgroup$ – StupidWolf Apr 7 at 23:25
1
$\begingroup$

We can set up the model using a dataset Sonar:

library(caret)
library(mlbench)

svmSBF <- caretSBF
data(Sonar)
data <- Sonar

With that settings, SBF filters the values using anovaScores if it is a classification or gamScores if it's continuous. The filter is based on p-value (see $score below):

caretSBF$score
function (x, y) 
{
    if (is.factor(y)) 
        anovaScores(x, y)
    else gamScores(x, y)
}

caretSBF$filter
function (score, x, y) 
score <= 0.05

For what you need, you need to run SBF first to get the filtered variables over the train set and use that to retrain your model:

sbf_fit <- sbf(
  form = Class ~ .,
  data = data, 
  method = "svmLinear", 
  tuneGrid=data.frame(C=1),
  trControl = trainControl(),
  sbfControl = sbfControl(method = "LGOCV",
                 number = 20,
                 p = .8,
                 functions = svmSBF,
                 saveDetails = TRUE))

We calculate the proportion of times each variable is kept:

prop_included = rowMeans(sapply(sbf_fit$variables,function(i)sbf_fit$coefnames %in% i))

selected = sbf_fit$coefnames[prop_included > 0.7]

And you train your model:

mdl <- train(
       form = Class ~ .,
       data = data[,c(selected,"Class")], 
       method = "svmLinear", 
      trControl = trainControl(method="cv",number=5))
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Again, great concise answer. Thank you kindly. $\endgroup$ – Taylor Braund Apr 9 at 0:27
  • $\begingroup$ you're welcome :) delayed answer, glad it was useful $\endgroup$ – StupidWolf Apr 9 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.