1
$\begingroup$

When using RFE in linear regression and logistic regression, do we one-hot encode the features (K levels and K dummy features) or dummy-encode the features (K levels and K-1 dummy features leaving one out).

As per a comment by @Matthew Drury in an answer (URL below), one hot encoding is applied for a regularized linear model and for unregularized linear model dummy encoding. My doubt is what type of encoding when using RFE without any L1/L2 penalties.

Problems with one-hot encoding vs. dummy encoding

My understanding is since in RFE some features gets eliminated so if for a categorical variable with say 4 levels we do dummy encoding and have 3 features/levels in model & RFE eliminated 1, we will only have 2 features/levels left and the interpretation of its coefficient would not make sense in absence of the one level which was left out as reference.

Whereas if we have done one-hot encoding and RFE considers 2 features as important and eliminates other 2 then we can very well judge/interpret the coefficients or importance of 2 features RFE keeps.

So question which type of encoding is needed to be done when using RFE with linear and logistic regression?

$\endgroup$
  • 1
    $\begingroup$ "dummy" coding and "one-hot" coding are complete synonyms, the first term being used in statistics and the second - in machine learning. The third synonym is "indicator" coding. As for whether one has or wants to keep all the k these elementary variables in the set or just k-1 variables out of it - is another question. This topic is related to multicollinearity and the type of the analysis or even the algorithmic realization of the analysis. $\endgroup$ – ttnphns Sep 3 '19 at 23:05
  • $\begingroup$ Sorry about the terminology but I clarified in my question what i mean. And about multicollinearity: I know full encoding (not leaving out a reference level) causes it i.e. dummy variable trap. But my question is when RFE eliminates some dummy variables/levels while keeping others of a categorical variable the interpretation of coefficients will be a problem. So what type of encoding is to be used when applying RFE and dropping some levels while keeping others is not recommended as per shorturl.at/knAGO $\endgroup$ – Vikrant Arora Sep 3 '19 at 23:25
  • $\begingroup$ How does this differ from your earlier Q stats.stackexchange.com/questions/424804/…? $\endgroup$ – kjetil b halvorsen Sep 6 '19 at 11:51
1
$\begingroup$

This is not really about the kind of encoding, as already explained in comment by @ttnphns. The kind (dialect) of encoding is more of an algorithm/implementation detail. For variable reduction (feature elimination) in logistic regression see How to reduce predictors the right way for a logistic regression model and this list of answered questions about rfe.

If the question is about dropping/joining some of the levels of the categorical variable, that is a very different question. Mostly the answer is: don't do it, it changes the definition of the variable. The categorical variable with all of its levels is te variable, and should be dropt or kept as is. See Does it make sense to apply recursive feature elimination on one-hot encoded features?.

If the problem is very may levels, see Principled way of collapsing categorical variables with many levels?.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks Kjetil for your inputs, a question: in your answer "Does it make sense to apply recursive feature elimination on one-hot encoded features?" you suggest don't drop dummy features of a particular categorical variable if you have dropped a level/category while encoding. I ask if I fully encode a categorical variable i.e. I don't drop a level, then will it make sense to drop some levels and keep important ones only since there won't be any dependcy in the sense that there isn't any dropped level or category. I guess in tree based models dropping unimportant dummy variables is fine? $\endgroup$ – Vikrant Arora Sep 7 '19 at 12:30
  • $\begingroup$ by above comment all I mean to ask is : In tree based models since we full encode a categorical feature I mean we don't drop a level as reference, can we drop unimportant dummy features and keep only important dummies? $\endgroup$ – Vikrant Arora Sep 7 '19 at 12:41
  • $\begingroup$ Your question didn't mention trees ... so if the Q is about how trees handles categorical variables, you should ask explicitly for that. In general, make clear what you mean by "unimportant dummy features", as what could look like unimportant by standard tests printed by default, could depend entirely upon the coding scheme used! $\endgroup$ – kjetil b halvorsen Sep 7 '19 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.