I have a specific question relating to the use of the two
R packages in building a prediction model; namely
caret (training) and
boruta (feature selection).
Firstly; I have a small data set ~ circa 2000 records. I have chosen to split this immediately in to 1600 ish records for training and 400 records for a holdout set.
Within the 1600 records, I have used
caret to train and optimise the
mtry parameter on a random forest model using 5-fold cross validation.
I've then applied this model against my holdout set to measure its AUC.
So far so good. However, having come across Boruta I want to try reducing the number of features in the data to hopefully improve this AUC fit. I realise random forest does this to some extent, but the Boruta model uses randomised significance testing to avoid overfitting, and this "feels" like a worthy improvement.
My question therefore is: where in this process is it correct to run Boruta? I'm wary from other posts here that it is incorrect to apply feature selection prior to CV, but does this still apply to my case where I've held out a clean, unseen 400 records?
Am I duplicating effort in applying Boruta? Is it worth doing CV when I have a holdout set?
Any advice would be gratefully received! Thanks.