Correct process for feature selection with cross validation with R Caret and Boruta

Question

I have a specific question relating to the use of the two R packages in building a prediction model; namely caret (training) and boruta (feature selection).

Firstly; I have a small data set ~ circa 2000 records. I have chosen to split this immediately in to 1600 ish records for training and 400 records for a holdout set.

Within the 1600 records, I have used caret to train and optimise the mtry parameter on a random forest model using 5-fold cross validation.

I've then applied this model against my holdout set to measure its AUC.

So far so good. However, having come across Boruta I want to try reducing the number of features in the data to hopefully improve this AUC fit. I realise random forest does this to some extent, but the Boruta model uses randomised significance testing to avoid overfitting, and this "feels" like a worthy improvement.

My question therefore is: where in this process is it correct to run Boruta? I'm wary from other posts here that it is incorrect to apply feature selection prior to CV, but does this still apply to my case where I've held out a clean, unseen 400 records?

Am I duplicating effort in applying Boruta? Is it worth doing CV when I have a holdout set?

Any advice would be gratefully received! Thanks.

Answer 1

Since you created a separate test set prior to feature selection / model training, any error estimates for that test set would still be valid.

That said, the biggest positive change that you could make that comes to mind would be to use some variant of K-fold CV (10-fold, LOO, Repeated K-fold CV, or Stratified Repeated K-fold CV) to repeat your entire process on different splits of the data, to ensure that your results aren't overly specific for the particular test set you generated.

Caret and scikit-learn both provide functionality to help you generate indices for all of the splits, so it just takes a little bit of time to use these to generate the various splits and repeat your entire process for each partitioning.

Regarding the use of Boruta followed by RF, it's less obvious how necessary the feature selection step is, and most likely it will depend on the nature and scale (# features) of your data. Random forest seems to perform well even when the number of features is much larger than the number of samples you have, but at some point it will start to be affected by the disproportionate number of features. As such, I personally tend to combine the two approaches, using feature selection as a first-pass method to eliminate the least informative variables, and then training/optimizing a final model on the remaining subset.

Ultimately though, the best way to find out is to try both approaches, using proper CV, and compare the impact of each on downstream model performance.