1
$\begingroup$

For what I know RF can be used as a model for feature importance or feature selection. Also, that RF can be used as a prediction and classification model. The question that I have is if its possible for a classification task, to use first RF as a feature selection method. So that with the most important features selected, to apply again a RF, but as a classifier. What would be the drawbacks to make this methodology? Is it a correct one? I have been searching some theory, and it seems it can be done, but I do not know if it is advisable to use this model. Any help or explanation about this issue would be great.

Improve this question
$\endgroup$
1
  • $\begingroup$ I don't see any problem with that except you always have to make a subjective choice about the cutoff for 'inclusion' if you want to select features from the variable importance measures. Why don't you use a LASSO first to select the variables and then apply random forest if you want? $\endgroup$
    – Blain Waan
    Commented Jan 26, 2020 at 1:45

1 Answer 1

Reset to default
-2
$\begingroup$

Random Forest is a classification/regression algorithm. It can be used as a "feature selection" method in the sense that -once it has been trained for classification- it provides some Feature Importances based on the information that was gained when making splits on each variable.

So technically yes, you can train your Random Forest on the full data and then retrain it only on the important variables. Given the interaction between the features, however, it would be best to do so in a stepwise fashion, removing only one feature at the time. Also, it is important to be wary of RF feature importances, as they can be quite misleading. In particular, they can be strongly affected by correlated features, and they have a strong bias towards numerical or high cardinality features.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If you use the bootstrap to compute confidence intervals for the importance measures you'll be shocked how little information they provide unless $N >> p$. $\endgroup$
    – Frank Harrell
    Commented Jan 28, 2020 at 14:33
  • 1
    $\begingroup$ I know, that's why I wrote to be very careful with them and linked to an article. They are often taken as gold but can be very misleading $\endgroup$
    – Davide ND
    Commented Jan 28, 2020 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.