2
$\begingroup$

Assume that we have a high-dimensional data with a few samples. We want to select a minimum set of best features from this dataset using LightGBM feature importance. This is because of an external restriction that we need to limit the number of features that are used in the final model. We want to select features using LightGBM feature importance vectors.

I see this question about applying Boruta before LASSO for feature selection. In the comments, someone referred to this question that shows that features that are important for a non-linear model (such as the random forest applied in Boruta) may not be important for a linear model like LASSO. What about if we use LightGBM as the second step of feature selection after running Boruta? Can this have any benefits compared to just running LGBM without running Boruta?

Improve this question
$\endgroup$
6
  • $\begingroup$ Why do you think you need feature selection here? $\endgroup$
    – Tim ♦
    Mar 28 at 10:19
  • 1
    $\begingroup$ As I mentioned, we have a very high dimensional data. We want to select a minimum set of influential features. $\endgroup$
    – ML Guy
    Mar 28 at 10:20
  • $\begingroup$ Do you have any reason to believe that you are getting bad results without it? $\endgroup$
    – Tim ♦
    Mar 28 at 10:22
  • 1
    $\begingroup$ Assume that we have a restriction on the number of features that our final system can work with. Also, we don't want to do feature extraction (like using PCA), but we want to select features from the current features. $\endgroup$
    – ML Guy
    Mar 28 at 10:26
  • 1
    $\begingroup$ Yes, we need to select features somehow and the question is this: Is there any benefits of using Boruta and then LGBM for feature selection rather than simply running LGBM and select based on feature importance? $\endgroup$
    – ML Guy
    Mar 28 at 10:31

1 Answer 1

Reset to default
1
$\begingroup$
  • Feature selection algorithms like Boruta, don't guarantee you to pick "universally the best" features. Each of the algorithms picks some definition of what they mean as importance and uses some algorithm for finding them, in some cases leading to picking different features depending on the choice of algorithm, or it's parameters. Saying it differently, the features picked by Boruta (or other algorithm) do not necessarily need to be the "best", different algorithm could pick different algorithm leading to better performance depending on the problem.
  • The same applies to feature importance. Usually there are more than one ways of calculating importances, that can give you different results.
  • Extracting a subset of "best" features and using them for a new model, will not necessarily lead to a model as good as the model using all the features. Different features may need different hyperparameters etc.

So both approaches can give you different results and neither is guaranteed to be the best. Probably the best you can do is try both and compare the results.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your reply. Yes, different methods can lead to different selected features. But can we come up with a case that running Boruta before LGBM results in a better selected set than running just LGBM. By "better" I mean selecting the features that are predictive of the target variable in a more generalizable way. Please note that here I use "LGBM" as the feature selector not the final classifier. $\endgroup$
    – ML Guy
    Mar 28 at 11:27
  • $\begingroup$ @MLGuy in some cases one approach would work better, in some the another, there is no rule. $\endgroup$
    – Tim ♦
    Mar 28 at 11:37
  • $\begingroup$ Does Boruta also inflate the p-values?, Do we need to adjust them with some kind of Bonferroni correction? $\endgroup$
    – skan
    Jun 4 at 19:38
  • $\begingroup$ Does Boruta take into account your model? For example if you want to use it for a Cox model. $\endgroup$
    – skan
    Jun 4 at 19:39
  • $\begingroup$ @skan I'd recommend you post those as questions. Boruta works as any other model-agnostic algorithm and is model agnostic. $\endgroup$
    – Tim ♦
    Jun 4 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.