I've been googling and reading alot about my issue but couldn't find a clear answer.

In order to prevent data leakage, I use caret RFE for feature elimination:

rfFuncs$summary <- twoClassSummary
#---- rfe control
rfe_ctrl <- rfeControl(functions = rfFuncs,
                       method = "repeatedcv",
                       number = 5, # folds
                       repeats = 3, # iterations
                       verbose = TRUE,
                       returnResamp = "all",
                       saveDetails = TRUE)

#---- rfe
rfe_profile <- rfe(df_train[,features_name], 
                  sizes = subsets,
                  rfeControl = rfe_ctrl,
                  metric = "ROC",
                  scale = TRUE)

#---- select final features
features_rfe <- predictors(rfe_profile)[1:30]

This works well and tells me how many predictors I should use. Let's say I get the result that 60 features works best. However, according to plots and further analysis (e.g. pickSizeTolerance) I know I can use the first 30 features and still having good performance of predictions.

Now, can I use these 30 features (=features_rfe) in a subsequent model training step (see code below) since I don't touch the test set until after this step? Or would I introduce data leakage with this step because the model gets trained with pre-selected best features?

#---- random forest control
rf_ctrl <- trainControl(method = "cv",
                        number = 5, # folds
                        verbose = TRUE,
                        returnResamp = "all",
                        classProbs = TRUE,
                        summaryFunction = twoClassSummary)

#---- random forest grid search
rf_grid <-  expand.grid(mtry = c(1, 5, 10, 15, 20))

#---- random forest
rf_caret_train <- train(df_train[,features_rfe], 
                method = "rf", 
                trControl = rf_ctrl,
                tuneGrid = rf_grid,
                metric = "ROC",
                verbose = TRUE)
#---- make predictions
rf_caret_test_raw <- predict(rf_caret_train, df_test[,features_rfe], type='raw', scale=TRUE)

According to this answer here caret rfe variable selection and test prediction it seems to be ok to "Using feature selection on the training set and predicting the test set is fine.". But my feelings tell me "no".

I hope you cann help me.

Best, Troji

  • $\begingroup$ don't quite understand the issue you have. Ultimately you want a model with predictive power. If you select the features based on only the training set, the test set has not been used in training the model $\endgroup$
    – StupidWolf
    Jun 16, 2020 at 21:37
  • $\begingroup$ thank you for your answer. My question was based on e.g. this blog post. But I guess, I missed the point that this problem only arise if I want to predict test error rate based on the training set only. As long as I use a test set which has not been used during training then I'm fine. Am I right? $\endgroup$
    – Trojii
    Jun 18, 2020 at 9:48
  • $\begingroup$ yes thats correct. lol the post is from a SO contributer $\endgroup$
    – StupidWolf
    Jun 18, 2020 at 13:30


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.