all-relevant feature selection vs minimum optimum feature selection

Question

There are many different ways to selection features in modeling process.

One way is to first select all-relevant features (like Boruta algorithm). And then develop model upon those those selected features.

Another way is minimum optimal feature selection methods. For example, recursive feature selection using random forest (or other algorithms). Build models on subsets of features and select the subset based on estimated model performance in cross-validation process.

My question is in what situations/applications one has advantages over the other? Any guidance and advice? How we can choose one over the other? Thanks.

Answer 1

The Boruta algorithm is built as a wrapper around Random Forest, but with a more complex way of ranking the feature's importance, based on merging the original dataset with its copy, where the column values are shuffled. This supposedly lessen the possibility of overfitting. As a consequence, the computational cost gap is not irrelevant here.

So, in a way, it is alright to say that the Boruta, in the context of feature selection, is an improvement on Random Forest.

Nonetheless, my poor analogy is a consequence of the fact that the algorithm itself is pretty straightforward to understand, so I'd ultimately suggest you try and read the paper: Feature Selection with the Boruta Package.

By the way, if you're on Python, apart from a nice explanation this blog post points to a really nice and fast implementation.