For a regression problem, I'm comparing two randomforest (RF) models:

(a) A resulted model after doing feature selection by caret::rfe().

As I know, the recursive feature elimination (RFE) algorithm automatically build a tuned model with the optimal feature subset. This resulted model can be used directly for prediction.

(b) A RF model built by randomForest::randomForest() using the same parameter value and feature subset with those of model (a).

Above two RF models were used to predict numeric data. I used airquality dataset in R to create an example code.

I only used numeric columns of the dataset and this set was split by 10-fold CV method (train:test=9:1).

The R2 of (a) was about 0.94 whereas that of (b) was about 0.52.
I don't expect that the models show the same performance, but I at least expected similar one.

Why these models have so different results?

Thank you

# load library


# dataset
data = 
  airquality[complete.cases(airquality), ] %>%

# run RFE  


index_train = createMultiFolds(data$Ozone, times = 5, k = 10)
pred_vars = names(data)[!(names(data) %in% c("Ozone"))]

ctrl_rf = rfeControl(method = "repeatedcv",
                     repeats = 5,
                     verbose = TRUE,
                     functions = rfFuncs,
                     index = index_train)

rf_rfe = rfe(x = data[, pred_vars],
             y = data$Ozone,
             size = c(1:length(pred_vars)),
             rfeControl = ctrl_rf,
             ntree = 1000)

# build RF manually

rf_var_selectd = rf_rfe$optVariables

train_idx = sample(1:dim(data)[1], dim(data)[1] * 0.1) # 10-fold cv
rf_manual = randomForest(Ozone ~ .,
                         data = data[, c("Ozone", rf_var_selectd)],
                         subset = train_idx,
                         mtry = 1, # default: p/3 = 6
                         ntree = 1000,
                         importance = TRUE)

# predict
rf_rfe_test = predict(rf_rfe,
                      newdata = data[-train_idx, ])
rf_manual_test = predict(rf_manual,
                         newdata = data[-train_idx, ])

summary(lm(rf_rfe_test ~ data[-train_idx, "Ozone"])) # R2 0.94
summary(lm(rf_manual_test ~ data[-train_idx, "Ozone"])) # R2 0.52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.