0
$\begingroup$

For what I know RF can be used as a model for feature importance or feature selection. Also, that RF can be used as a prediction and classification model. The question that I have is if its possible for a classification task, to use first RF as a feature selection method. So that with the most important features selected, to apply again a RF, but as a classifier. What would be the drawbacks to make this methodology? Is it a correct one? I have been searching some theory, and it seems it can be done, but I do not know if it is advisable to use this model. Any help or explanation about this issue would be great.

Thanks

$\endgroup$
  • $\begingroup$ I don't see any problem with that except you always have to make a subjective choice about the cutoff for 'inclusion' if you want to select features from the variable importance measures. Why don't you use a LASSO first to select the variables and then apply random forest if you want? $\endgroup$ – Blain Waan Jan 26 at 1:45
0
$\begingroup$

Random Forest is a classification/regression algorithm. It can be used as a "feature selection" method in the sense that -once it has been trained for classification- it provides some Feature Importances based on the information that was gained when making splits on each variable.

So technically yes, you can train your Random Forest on the full data and then retrain it only on the important variables. Given the interaction between the features, however, it would be best to do so in a stepwise fashion, removing only one feature at the time. Also, it is important to be wary of RF feature importances, as they can be quite misleading. In particular, they can be strongly affected by correlated features, and they have a strong bias towards numerical or high cardinality features.

$\endgroup$
  • $\begingroup$ If you use the bootstrap to compute confidence intervals for the importance measures you'll be shocked how little information they provide unless $N >> p$. $\endgroup$ – Frank Harrell Jan 28 at 14:33
  • 1
    $\begingroup$ I know, that's why I wrote to be very careful with them and linked to an article. They are often taken as gold but can be very misleading $\endgroup$ – Davide ND Jan 28 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.