For a regression problem, I'm comparing two randomforest (RF) models:
(a) A resulted model after doing feature selection by
As I know, the recursive feature elimination (RFE) algorithm automatically build a tuned model with the optimal feature subset. This resulted model can be used directly for prediction.
(b) A RF model built by
randomForest::randomForest() using the same parameter value and feature subset with those of model (a).
Above two RF models were used to predict numeric data. I used
airquality dataset in R to create an example code.
I only used numeric columns of the dataset and this set was split by 10-fold CV method (train:test=9:1).
The R2 of (a) was about 0.94 whereas that of (b) was about 0.52.
I don't expect that the models show the same performance, but I at least expected similar one.
Why these models have so different results?
# load library library(caret) library(randomForest) library(dplyr) # dataset data = airquality[complete.cases(airquality), ] %>% select(Ozone:Temp) # run RFE set.seed(100) index_train = createMultiFolds(data$Ozone, times = 5, k = 10) pred_vars = names(data)[!(names(data) %in% c("Ozone"))] ctrl_rf = rfeControl(method = "repeatedcv", repeats = 5, verbose = TRUE, functions = rfFuncs, index = index_train) rf_rfe = rfe(x = data[, pred_vars], y = data$Ozone, size = c(1:length(pred_vars)), rfeControl = ctrl_rf, ntree = 1000) # build RF manually rf_rfe$fit rf_var_selectd = rf_rfe$optVariables train_idx = sample(1:dim(data), dim(data) * 0.1) # 10-fold cv rf_manual = randomForest(Ozone ~ ., data = data[, c("Ozone", rf_var_selectd)], subset = train_idx, mtry = 1, # default: p/3 = 6 ntree = 1000, importance = TRUE) # predict rf_rfe_test = predict(rf_rfe, newdata = data[-train_idx, ]) rf_manual_test = predict(rf_manual, newdata = data[-train_idx, ]) summary(lm(rf_rfe_test ~ data[-train_idx, "Ozone"])) # R2 0.94 summary(lm(rf_manual_test ~ data[-train_idx, "Ozone"])) # R2 0.52