I have a general question about boruta. I know that it is not necessarily going to improve accuracy, as by design it is not meant to do this. However, what about using it before a regularization method such as Lasso, which further reduces our feature set? Should we expect any interesting properties to emerge? I tried applying lasso to a data set both with and without pre- application of boruta. Using 5-fold cross validation the test set accuracy without Boruta was slightly higher than with Boruta
Boruta, based on RF, doesn't care about scale. It permutes a column several times and looks at the distribution of permuted importance vs. the non-permuted. It only labels as reject those columns that are more likely in the distribution than not. Things that are questionable are retained, but have an estimated importance value.
Boruta looks at nonlinear interactions in its importance estimate. The regularization does not necessarily do that. It can allow you to set columns to zero before applying the regularization, so it acts as if you set a smaller regularization parameter.
So I'm borrowing images from this site:
In the image, the point at the center of the ellipse is the non-regularized fit (metaphor). You can see that the values for the 2 axes are both non-zero. For the L1 regularized, preference is given to values that are along the axis AND closer to zero, aka where the other axes tend to be zero. You can also see that in L2 regularization they only want points whose values are closer to zero.