0
$\begingroup$

There are many different ways to selection features in modeling process.

One way is to first select all-relevant features (like Boruta algorithm). And then develop model upon those those selected features.

Another way is minimum optimal feature selection methods. For example, recursive feature selection using random forest (or other algorithms). Build models on subsets of features and select the subset based on estimated model performance in cross-validation process.

My question is in what situations/applications one has advantages over the other? Any guidance and advice? How we can choose one over the other? Thanks.

$\endgroup$
1
$\begingroup$

The Boruta algorithm is built as a wrapper around Random Forest, but with a more complex way of ranking the feature's importance, based on merging the original dataset with its copy, where the column values are shuffled. This supposedly lessen the possibility of overfitting. As a consequence, the computational cost gap is not irrelevant here.

So, in a way, it is alright to say that the Boruta, in the context of feature selection, is an improvement on Random Forest.

Nonetheless, my poor analogy is a consequence of the fact that the algorithm itself is pretty straightforward to understand, so I'd ultimately suggest you try and read the paper: Feature Selection with the Boruta Package.

By the way, if you're on Python, apart from a nice explanation this blog post points to a really nice and fast implementation.

$\endgroup$
  • $\begingroup$ I don't think the user is asking if Boruta is better than Random Forest for feature selection, I think they are wondering when it is appropriate to use a wrapper type feature selection method vs. a filter type. To which I would say that a properly performed wrapper will outperform a filter method, but is in general a lot costlier in computing performance. So which to use really depends on computing cost vs. algorithm performance $\endgroup$ – astel Feb 1 at 23:25
  • $\begingroup$ I also don't believe he's asking that, that's why in my first paragraph I answered the same thing you wrote. I just added an observation that I consider reasonable. $\endgroup$ – Lucas Farias Feb 2 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.