0
$\begingroup$

Is it advisable to use RFE for linear or logistic regression when we have some dummy features. Reason I am asking this is: in RFE we will eliminate some features which will also include dummy features and as per this answer for a categorical variable which is dummy encoded we should not keep some features & drop others, we should keep all dummy features of a categorical feature.

Reasoning given is : "You should leave all five indicator variables in. Dropping predictors because they are non-significant leads to biased estimates for regression coefficients and inflated p-values."

"The problem with dropping the indicator is that you'll change the p-values of the remaining levels as well, as you're shifting the intercept (aka the reference group.)"

If ideal thing to do is keeping all levels/dummy features of a categorical variable then how do you use RFE for dummy features or how do you eliminate unimportant dummy or categorical features?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Dummy variables are one set, they represent the categorical variable which is parental to them. Normally one should drop or leave the complete set. Dropping just one dummy of the set is essentially merging of this category with the reference category. $\endgroup$
    – ttnphns
    Commented Sep 3, 2019 at 22:54
  • $\begingroup$ @ttnphns If say 5 dummy variables of a parental categorical variable are highly significant & have high coefficients and 5 are insignificant, then wouldn't it be wrong to drop the complete set. $\endgroup$
    – Vikrant Arora
    Commented Sep 3, 2019 at 23:38
  • $\begingroup$ @ttnphns "Dropping just one dummy of the set is essentially merging of this category with the reference category">>>>I agree, so to overcome this merging of dropped dummy features wouldn't it be better to use full encoding i.e. not dropping a level of categorical variable as reference category. If that's done then even if you drop 1 or more dummy features it would not create an issue as all dummy variables derived from a particular categorical variable would be independent in the sense that there is no reference cateory $\endgroup$
    – Vikrant Arora
    Commented Sep 3, 2019 at 23:43

2 Answers 2

Reset to default
0
$\begingroup$

This is a fairly old question but since it has no accepted answer I will give my two cents.

In logistic regression, it is typical to have a reference group, meaning we drop one category from the variable and lump it into the interpretation of the intercept. Consider a categorical variable with three levels: a,b, and c. Suppose that by recursive feature elimination that you determine category c should be dropped. What is the effect of this?

By removing category c, you automatically change all observations of level c to level a. Since the variables are all dummied, the observations from category c have 0s in columns for a and b. Remove the column c, and you automatically make all observations of category c into category a. This can result in large bias.

If you are going to select categorical variables you need to select/eliminate groups of dummy variables at a time. Either the model has all the variables from that category, or it doesn’t.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Rather than answering a FAQ for the n'th time, it would be better to search for duplicates (which there are), a good start is to look at the side bar under "Linked" and "Related" $\endgroup$
    – kjetil b halvorsen ♦
    Commented Dec 25, 2020 at 2:25
-1
$\begingroup$

I don't think you need to always keep all indicators. The ones that are dropped because of lack of significance (after dummying) can technically be considered as collapsed to one.

The inflation of p values happens to all features as a result of RFE.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Good point "The inflation of p values happens to all features as a result of RFE.". $\endgroup$
    – Vikrant Arora
    Commented Sep 3, 2019 at 23:34
  • 1
    $\begingroup$ @ldin If you drop some dummy variables then coefficients of remaining dummy variables would not be accurate and no inferences about features importance could be done or ranking of features to select the most important ones based on coefficients would not be possible. That's my concern – $\endgroup$
    – Vikrant Arora
    Commented Sep 7, 2019 at 12:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.