4
$\begingroup$

I have a general question about boruta. I know that it is not necessarily going to improve accuracy, as by design it is not meant to do this. However, what about using it before a regularization method such as Lasso, which further reduces our feature set? Should we expect any interesting properties to emerge? I tried applying lasso to a data set both with and without pre- application of boruta. Using 5-fold cross validation the test set accuracy without Boruta was slightly higher than with Boruta

$\endgroup$
2
  • 2
    $\begingroup$ Two-stage variable selection methods are common, such as 2 rounds of lasso estimating 2 different penalties, or lasso-then-ridge, or lasso-then-elastic net, etc. I'm not sure if anyone has studied Boruta-then-lasso, but it sounds like an interesting research direction. One caveat is that the features that are important according to nonlinear models like random forest might not be important to a linear model like lasso. Here's an example: stats.stackexchange.com/questions/164048/… $\endgroup$
    – Sycorax
    May 29, 2020 at 14:19
  • $\begingroup$ One fun thing with the nonlinear-then-linear is that, impo, you get better projection of the nonlinear into the linear space than you would with linear-linear. You can think of it like piecewise ensembles of Taylor series of the nonlinear terms where the order is truncated based on the variables of the simpler fit. $\endgroup$ Jun 8, 2020 at 15:35

1 Answer 1

3
$\begingroup$

Boruta, based on RF, doesn't care about scale. It permutes a column several times and looks at the distribution of permuted importance vs. the non-permuted. It only labels as reject those columns that are more likely in the distribution than not. Things that are questionable are retained, but have an estimated importance value.

Boruta looks at nonlinear interactions in its importance estimate. The regularization does not necessarily do that. It can allow you to set columns to zero before applying the regularization, so it acts as if you set a smaller regularization parameter.

So I'm borrowing images from this site:
https://kevinbinz.com/2019/06/09/regularization/

You can think of regularized regression like this:
enter image description here

In the image, the point at the center of the ellipse is the non-regularized fit (metaphor). You can see that the values for the 2 axes are both non-zero. For the L1 regularized, preference is given to values that are along the axis AND closer to zero, aka where the other axes tend to be zero. You can also see that in L2 regularization they only want points whose values are closer to zero.

In an exercise of fun they also provide this:
enter image description here

Update:
Some folks say you can take the power of a random forest and encode it into a weighted linear model (link, link) if you use the RF weights in it. You could take Boruta feature importance for retained features, and get an effect like Lasso, where some parameter values are set to zero, and an effect like Ridge where the parameter values are regularized. It should act like a robust hybrid of both. It might be interesting to compare it to a Huber-regularization analog of Ridge(link, link).

$\endgroup$
2
  • $\begingroup$ Does Boruta also inflate the p-values?, Do we need to adjust them with some kind of Bonferroni correction? $\endgroup$
    – skan
    Jun 4, 2023 at 19:36
  • $\begingroup$ Random Forests have that correction in their importance values anyway, but (as I understand it) Boruta goes a step more and augments with multiple true-random columns and looks at the distributions of importance coming from those to determine if the importance from a non-synthetic column, including a randomly permuted column, is likely to be within that distribution. $\endgroup$ Jun 5, 2023 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.