My understanding is rudimentary and high level but it seems like Ridge Regression/LASSO/Elastic Net would be better when the data is linear and Random Forest is better when the data is nonlinear? Also Random Forest tends to be more resource intensive? Am I right? And is there more?

  • $\begingroup$ Are you asking about feature selection in particular, or for a general comparison between these methods? If the latter, is there anything in particular you'd like to know? Otherwise, it would be a bit broad. $\endgroup$ – user20160 Aug 24 '18 at 16:48
  • $\begingroup$ @user20160 I'm asking for a general comparison of these methods when it comes to feature selection. I'm looking for a high level overview and what would make one want to use one over the other. $\endgroup$ – TinderForMidgets Aug 24 '18 at 17:02
  • $\begingroup$ Ok, got it. Ridge regression and random forests don't actually perform feature selection. So, that just leaves elastic net and lasso (which is a special case of elastic net with the L2 penalty set to zero). There are multiple posts on this site comparing them (e.g. see here and here, there are probably more) $\endgroup$ – user20160 Aug 24 '18 at 17:30
  • 1
    $\begingroup$ @user20160 Wait, random forest doesn't perform feature selection? Then why does it include something like feature importance. A quick search of the internet tells me that random forest is used for feature selection. $\endgroup$ – TinderForMidgets Aug 24 '18 at 17:52
  • $\begingroup$ RFs do select an individual feature for each split performed, which is a purely local decision. Hopefully uninformative features won't be chosen often. But, RFs don't impose any explicit limitation on the number of features used in the overall function (i.e. across all splits and trees). This is different than explicit feature selection methods, and sparsity-inducing regularization techniques like lasso, etc. The variable importance score measures the amount of influence each feature has on the learned function, but measuring is different than constraining. $\endgroup$ – user20160 Aug 24 '18 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.