How to do backward subset selection on a random forest model for classification?

Question

I'd like to identify the most predictive features for my classification model. I'm using this data.

Here is a sample.

This is my code. I'd like to use the predictors to predict loan status.

library(caret)

folds <- 10
cvIndex <- createFolds(factor(Vc$Loan_Status), folds, returnTrain = T)
tc <- trainControl(index = cvIndex,
                   method = 'cv', 
                   number = folds)


preProcValues <- preProcess(Vc, method = c("center","scale"))
train_processed <- predict(preProcValues, Vc)
train_processed$Loan_Status <- ifelse(train_processed$Loan_Status==FALSE,0,1)

index <- createDataPartition(train_processed$quoteCreated, p=0.75, list=FALSE)
trainSet <- train_processed[ index,]
testSet <- train_processed[-index,]

control <- rfeControl(functions = rfFuncs,
                      method = "repeatedcv",
                      repeats = 3,
                      verbose = FALSE)

outcomeName<-'Loan_Status'
predictors<-names(trainSet)[!(names(trainSet) %in% outcomeName | names(trainSet) == "Loan_ID")]
Quote_Pred_Profile <- rfe(trainSet[,predictors], trainSet[,outcomeName],
                          metric = "Accuracy",
                          maximizize = TRUE,
                         rfeControl = control)
plot(Quote_Pred_Profile)

I get the warning message:

In randomForest.default(x, y, importance = TRUE, ...) : The response has five or fewer unique values. Are you sure you want to do regression?

Why is it serving this error? Is this the right way to do subset selection to prep for classification?

Also, when I run plot(Quote_Pred_Profile), it serves a plot with RMSE on the y-axis.

Answer 1

Ah, I figured it out. It was a data-type issue. I coded my response variable as 0-1, under the impression that was necessary for caret. I just gave it another try with this code.

Quote_Pred_Profile <- rfe(trainSet[,predictors], as.factor(trainSet[,outcomeName]),
                          metric = "Accuracy",
                          maximizize = TRUE,
                         rfeControl = control)

And it is working just I expected, along with a chart that shows the accuracy for each possible number of variables.