1
$\begingroup$

I have a general question about boruta. I know that it is not necessarily going to improve accuracy, as by design it is not meant to do this. However, what about using it before a regularization method such as Lasso, which further reduces our feature set? Should we expect any interesting properties to emerge? I tried applying lasso to a data set both with and without pre- application of boruta. Using 5-fold cross validation the test set accuracy without Boruta was slightly higher than with Boruta

$\endgroup$
  • 1
    $\begingroup$ Two-stage variable selection methods are common, such as 2 rounds of lasso estimating 2 different penalties, or lasso-then-ridge, or lasso-then-elastic net, etc. I'm not sure if anyone has studied Boruta-then-lasso, but it sounds like an interesting research direction. One caveat is that the features that are important according to nonlinear models like random forest might not be important to a linear model like lasso. Here's an example: stats.stackexchange.com/questions/164048/… $\endgroup$ – Sycorax May 29 at 14:19
  • $\begingroup$ One fun thing with the nonlinear-then-linear is that, impo, you get better projection of the nonlinear into the linear space than you would with linear-linear. You can think of it like piecewise ensembles of Taylor series of the nonlinear terms where the order is truncated based on the variables of the simpler fit. $\endgroup$ – EngrStudent Jun 8 at 15:35
2
$\begingroup$

Boruta, based on RF, doesn't care about scale. It permutes a column several times and looks at the distribution of permuted importance vs. the non-permuted. It only labels as reject those columns that are more likely in the distribution than not. Things that are questionable are retained, but have an estimated importance value.

Boruta looks at nonlinear interactions in its importance estimate. The regularization does not necessarily do that. It can allow you to set columns to zero before applying the regularization, so it acts as if you set a smaller regularization parameter.

So I'm borrowing images from this site:
https://kevinbinz.com/2019/06/09/regularization/

You can think of regularized regression like this:
enter image description here

In the image, the point at the center of the ellipse is the non-regularized fit (metaphor). You can see that the values for the 2 axes are both non-zero. For the L1 regularized, preference is given to values that are along the axis AND closer to zero, aka where the other axes tend to be zero. You can also see that in L2 regularization they only want points whose values are closer to zero.

In an exercise of fun they also provide this:
enter image description here

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.