0
$\begingroup$

Is it advisable to use RFE for linear or logistic regression when we have some dummy features. Reason I am asking this is: in RFE we will eliminate some features which will also include dummy features and as per this answer for a categorical variable which is dummy encoded we should not keep some features & drop others, we should keep all dummy features of a categorical feature.

Reasoning given is : "You should leave all five indicator variables in. Dropping predictors because they are non-significant leads to biased estimates for regression coefficients and inflated p-values."

"The problem with dropping the indicator is that you'll change the p-values of the remaining levels as well, as you're shifting the intercept (aka the reference group.)"

If ideal thing to do is keeping all levels/dummy features of a categorical variable then how do you use RFE for dummy features or how do you eliminate unimportant dummy or categorical features?

$\endgroup$
  • 1
    $\begingroup$ Dummy variables are one set, they represent the categorical variable which is parental to them. Normally one should drop or leave the complete set. Dropping just one dummy of the set is essentially merging of this category with the reference category. $\endgroup$ – ttnphns Sep 3 '19 at 22:54
  • $\begingroup$ @ttnphns If say 5 dummy variables of a parental categorical variable are highly significant & have high coefficients and 5 are insignificant, then wouldn't it be wrong to drop the complete set. $\endgroup$ – Vikrant Arora Sep 3 '19 at 23:38
  • $\begingroup$ @ttnphns "Dropping just one dummy of the set is essentially merging of this category with the reference category">>>>I agree, so to overcome this merging of dropped dummy features wouldn't it be better to use full encoding i.e. not dropping a level of categorical variable as reference category. If that's done then even if you drop 1 or more dummy features it would not create an issue as all dummy variables derived from a particular categorical variable would be independent in the sense that there is no reference cateory $\endgroup$ – Vikrant Arora Sep 3 '19 at 23:43
0
$\begingroup$

I don't think you need to always keep all indicators. The ones that are dropped because of lack of significance (after dummying) can technically be considered as collapsed to one.

The inflation of p values happens to all features as a result of RFE.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Good point "The inflation of p values happens to all features as a result of RFE.". $\endgroup$ – Vikrant Arora Sep 3 '19 at 23:34
  • 1
    $\begingroup$ @ldin If you drop some dummy variables then coefficients of remaining dummy variables would not be accurate and no inferences about features importance could be done or ranking of features to select the most important ones based on coefficients would not be possible. That's my concern – $\endgroup$ – Vikrant Arora Sep 7 '19 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.