1
$\begingroup$

When using RFE in linear regression and logistic regression, do we one-hot encode the features (K levels and K dummy features) or dummy-encode the features (K levels and K-1 dummy features leaving one out).

As per a comment by @Matthew Drury in an answer (URL below), one hot encoding is applied for a regularized linear model and for unregularized linear model dummy encoding. My doubt is what type of encoding when using RFE without any L1/L2 penalties.

Problems with one-hot encoding vs. dummy encoding

My understanding is since in RFE some features gets eliminated so if for a categorical variable with say 4 levels we do dummy encoding and have 3 features/levels in model & RFE eliminated 1, we will only have 2 features/levels left and the interpretation of its coefficient would not make sense in absence of the one level which was left out as reference.

Whereas if we have done one-hot encoding and RFE considers 2 features as important and eliminates other 2 then we can very well judge/interpret the coefficients or importance of 2 features RFE keeps.

So question which type of encoding is needed to be done when using RFE with linear and logistic regression?

$\endgroup$
3
  • 1
    $\begingroup$ "dummy" coding and "one-hot" coding are complete synonyms, the first term being used in statistics and the second - in machine learning. The third synonym is "indicator" coding. As for whether one has or wants to keep all the k these elementary variables in the set or just k-1 variables out of it - is another question. This topic is related to multicollinearity and the type of the analysis or even the algorithmic realization of the analysis. $\endgroup$
    – ttnphns
    Sep 3, 2019 at 23:05
  • $\begingroup$ Sorry about the terminology but I clarified in my question what i mean. And about multicollinearity: I know full encoding (not leaving out a reference level) causes it i.e. dummy variable trap. But my question is when RFE eliminates some dummy variables/levels while keeping others of a categorical variable the interpretation of coefficients will be a problem. So what type of encoding is to be used when applying RFE and dropping some levels while keeping others is not recommended as per shorturl.at/knAGO $\endgroup$ Sep 3, 2019 at 23:25
  • $\begingroup$ How does this differ from your earlier Q stats.stackexchange.com/questions/424804/…? $\endgroup$ Sep 6, 2019 at 11:51

2 Answers 2

1
$\begingroup$

This is not really about the kind of encoding, as already explained in comment by @ttnphns. The kind (dialect) of encoding is more of an algorithm/implementation detail. For variable reduction (feature elimination) in logistic regression see How to reduce predictors the right way for a logistic regression model and this list of answered questions about rfe.

If the question is about dropping/joining some of the levels of the categorical variable, that is a very different question. Mostly the answer is: don't do it, it changes the definition of the variable. The categorical variable with all of its levels is the variable, and should be dropt or kept as is. See Does it make sense to apply recursive feature elimination on one-hot encoded features?.

If the problem is very many levels, see Principled way of collapsing categorical variables with many levels?.

Part of the problem comes from a lack of abstraction. When the linear model is presented, with matrix language, as $$ Y=X\beta +\epsilon $$ this is not really where modeling starts. Some of the columns of $X$ really represent one 1D-variable, maybe a continuous variable like age, but others come from multi-df variables, maybe a spline or polynomial in age, maybe a factor, ... The matrix language used above forgets about this relations, so the multiple columns representing some logical variable are "forgotten", this relationship is unrepresented in the matrix formulation, which is a loss. Some modeling languages, like R, preserve this relationship, in R with terms objects. So, if RFE is used with one-hot coded columns, it should be done not at the column level, but at the terms level. With R, if you do the one-hot encoding not "by hand", yourself, but by declaring a factor variable and leaving the actual coding to R, the R in-built functions for stepwise modeling will use the terms structure and so do the right thing. If RFE is a good idea at all, is another question, see Are there any circumstances where stepwise regression should be used?

$\endgroup$
3
  • $\begingroup$ Thanks Kjetil for your inputs, a question: in your answer "Does it make sense to apply recursive feature elimination on one-hot encoded features?" you suggest don't drop dummy features of a particular categorical variable if you have dropped a level/category while encoding. I ask if I fully encode a categorical variable i.e. I don't drop a level, then will it make sense to drop some levels and keep important ones only since there won't be any dependcy in the sense that there isn't any dropped level or category. I guess in tree based models dropping unimportant dummy variables is fine? $\endgroup$ Sep 7, 2019 at 12:30
  • $\begingroup$ by above comment all I mean to ask is : In tree based models since we full encode a categorical feature I mean we don't drop a level as reference, can we drop unimportant dummy features and keep only important dummies? $\endgroup$ Sep 7, 2019 at 12:41
  • $\begingroup$ Your question didn't mention trees ... so if the Q is about how trees handles categorical variables, you should ask explicitly for that. In general, make clear what you mean by "unimportant dummy features", as what could look like unimportant by standard tests printed by default, could depend entirely upon the coding scheme used! $\endgroup$ Sep 7, 2019 at 17:35
0
$\begingroup$

Kaggle has a really good answer to this. In short:

Use k-1:

For algorithms that look at ALL the features at the same time, for example linear regression, support vector machines, neural networks, clustering. BUT, this is true for training, for feature selection see below

Use k:

For algorithm that look only at a subset of features, for example tree algorithms (random forest)

Feature selection = k

Finally, if you are planning to do feature selection, you will also need the entire set of binary variables (k) to let the machine learning model select which ones have the most predictive power.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.