I have a specific question relating to the use of the two R packages in building a prediction model; namely caret (training) and boruta (feature selection).

Firstly; I have a small data set ~ circa 2000 records. I have chosen to split this immediately in to 1600 ish records for training and 400 records for a holdout set.

Within the 1600 records, I have used caret to train and optimise the mtry parameter on a random forest model using 5-fold cross validation.

I've then applied this model against my holdout set to measure its AUC.

So far so good. However, having come across Boruta I want to try reducing the number of features in the data to hopefully improve this AUC fit. I realise random forest does this to some extent, but the Boruta model uses randomised significance testing to avoid overfitting, and this "feels" like a worthy improvement.

My question therefore is: where in this process is it correct to run Boruta? I'm wary from other posts here that it is incorrect to apply feature selection prior to CV, but does this still apply to my case where I've held out a clean, unseen 400 records?

Am I duplicating effort in applying Boruta? Is it worth doing CV when I have a holdout set?

Any advice would be gratefully received! Thanks.

  • $\begingroup$ I'm surprised that nobody answers the question! $\endgroup$
    – zesla
    Commented Jan 19, 2019 at 2:47
  • $\begingroup$ Note that feature selection may make generalisation performance worse rather than better, especially if you have a lot of features and not very many samples. The reason is that the more features you have, the greater the degrees of freedom available to minimise the feature selection criterion, and so the greater the likelihood of over-fitting the feature selection criterion. The smaller the dataset, the greater the variance of the feature selection criterion and the easier it is to overfit. So if improving performance is the only aim, regularisation is likely to work better. $\endgroup$ Commented Aug 6, 2023 at 16:27
  • $\begingroup$ What are your constraints? If you have enough computational time, I'd 1) use nested CV, as a single hold-out will be spoiled after a series of (indirect) attempts of improving metrics 2) run boruta inside CV and on the whole data, and compare results. Do you have time to sort out potential collisions? Eg if you get different importance with boosting/lasso and boruta, which one you'd be ready to rationalize? Will you have time for further experiments? If no, I'd go with nested CV + model's native feature selection. $\endgroup$
    – forveg
    Commented Dec 12, 2023 at 5:55

1 Answer 1


Since you created a separate test set prior to feature selection / model training, any error estimates for that test set would still be valid.

That said, the biggest positive change that you could make that comes to mind would be to use some variant of K-fold CV (10-fold, LOO, Repeated K-fold CV, or Stratified Repeated K-fold CV) to repeat your entire process on different splits of the data, to ensure that your results aren't overly specific for the particular test set you generated.

Caret and scikit-learn both provide functionality to help you generate indices for all of the splits, so it just takes a little bit of time to use these to generate the various splits and repeat your entire process for each partitioning.

Regarding the use of Boruta followed by RF, it's less obvious how necessary the feature selection step is, and most likely it will depend on the nature and scale (# features) of your data. Random forest seems to perform well even when the number of features is much larger than the number of samples you have, but at some point it will start to be affected by the disproportionate number of features. As such, I personally tend to combine the two approaches, using feature selection as a first-pass method to eliminate the least informative variables, and then training/optimizing a final model on the remaining subset.

Ultimately though, the best way to find out is to try both approaches, using proper CV, and compare the impact of each on downstream model performance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.