1
$\begingroup$

I am working on binary classification with class proportion of 77:23 (977 records)

Currently, I am exploring the feature selection approaches and came across methods like below

a) Featurewiz

b) Sequential forward and backward feature selection

c) Borutapy

d) RFE etc

Now all the above methods use a ML model to find the best performing features.

Now my question is

a) Do we have to use the best parameters for getting the best features?

b) If yes, then once we select the features, do we have to again do a gridsearchCV and find the best parameters to fit and predict?

Or do you think it is suffice to just use default parameters for feature selection and for model building we can use best parameters?

$\endgroup$
6
  • 1
    $\begingroup$ I would agree with gunes. It is kind of a chicken and egg problem, and you won't find the global optimum. However, that's okay - you can still get good results. I would recommend using something like Boruta as a feature selection tool, then optimize hyperparams with gridsearchCV after using only good features (as selected by Boruta) $\endgroup$ Feb 22 at 18:18
  • $\begingroup$ Based on both of your responses, I have another related question digging deeper to understand why of certain decisions during ML model building. would you be interested to share your views on this - stats.stackexchange.com/questions/565454/… $\endgroup$
    – The Great
    Feb 23 at 1:59
  • $\begingroup$ To be honest, I couldn't quite grasp your question there. Could you try rephrasing? I'm not sure what you mean by "random", for example. $\endgroup$ Feb 23 at 15:17
  • $\begingroup$ @VladimirBelik - by random, I mean not consistent. And why we are not able to find out a consistent specific reasoning/explanations for why ML model does what it does. $\endgroup$
    – The Great
    Feb 24 at 0:22
  • $\begingroup$ If still not able to understand, the fault might be with my english skills. Not sure, how else can I phrase that. $\endgroup$
    – The Great
    Feb 24 at 0:23

1 Answer 1

1
$\begingroup$

Both feature selection and hyper-parameter (HP) optimization are sub-optimal. With infinite compute power, we could have done both at the same time. But we can't search the whole space, so we have approximate approaches.

Do we have to use the best parameters for getting the best features?

Typical practice is to use a good enough estimator. Usually, the best HPs are found with the complete feature set may not be the same as the ones found with a feature subset. It's a chicken-egg problem. So, you don't have to. These are all approximate approaches.

You can also use the features found by the above heuristics and include them in your HP search, e.g. include your best three feature sets and search best HPs together with these sets as well.

$\endgroup$
3
  • 1
    $\begingroup$ thanks for your help. upvoted $\endgroup$
    – The Great
    Feb 22 at 10:36
  • $\begingroup$ I think OP was asking a different question. Am I missing something? He was asking about HP search for estimator that is being used inside the Boruta algorithm. Even in this paper, the author mentions about one of the caveats of Boruta: "For datasets with lots of features, the default configuration of the importance source is likely insufficient; in the particular case of Random Forest the number of trees is often not large enough to allow the importance scores to stabilise..." $\endgroup$
    – ARAT
    2 days ago
  • $\begingroup$ I don't think the question is specific to Boruta or any other feature selection method. It is about how HP searching and feature selection interplay. $\endgroup$
    – gunes
    12 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.