# Recursive feature elimination and one-hot & dummy encoding?

When using RFE in linear regression and logistic regression, do we one-hot encode the features (K levels and K dummy features) or dummy-encode the features (K levels and K-1 dummy features leaving one out).

As per a comment by @Matthew Drury in an answer (URL below), one hot encoding is applied for a regularized linear model and for unregularized linear model dummy encoding. My doubt is what type of encoding when using RFE without any L1/L2 penalties.

Problems with one-hot encoding vs. dummy encoding

My understanding is since in RFE some features gets eliminated so if for a categorical variable with say 4 levels we do dummy encoding and have 3 features/levels in model & RFE eliminated 1, we will only have 2 features/levels left and the interpretation of its coefficient would not make sense in absence of the one level which was left out as reference.

Whereas if we have done one-hot encoding and RFE considers 2 features as important and eliminates other 2 then we can very well judge/interpret the coefficients or importance of 2 features RFE keeps.

So question which type of encoding is needed to be done when using RFE with linear and logistic regression?

• "dummy" coding and "one-hot" coding are complete synonyms, the first term being used in statistics and the second - in machine learning. The third synonym is "indicator" coding. As for whether one has or wants to keep all the k these elementary variables in the set or just k-1 variables out of it - is another question. This topic is related to multicollinearity and the type of the analysis or even the algorithmic realization of the analysis. Sep 3 '19 at 23:05
• Sorry about the terminology but I clarified in my question what i mean. And about multicollinearity: I know full encoding (not leaving out a reference level) causes it i.e. dummy variable trap. But my question is when RFE eliminates some dummy variables/levels while keeping others of a categorical variable the interpretation of coefficients will be a problem. So what type of encoding is to be used when applying RFE and dropping some levels while keeping others is not recommended as per shorturl.at/knAGO Sep 3 '19 at 23:25
• How does this differ from your earlier Q stats.stackexchange.com/questions/424804/…? Sep 6 '19 at 11:51

Part of the problem comes from a lack of abstraction. When the linear model is presented, with matrix language, as $$Y=X\beta +\epsilon$$ this is not really where modeling starts. Some of the columns of $$X$$ really represent one 1D-variable, maybe a continuous variable like age, but others come from multi-df variables, maybe a spline or polynomial in age, maybe a factor, ... The matrix language used above forgets about this relations, so the multiple columns representing some logical variable are "forgotten", this relationship is unrepresented in the matrix formulation, which is a loss. Some modeling languages, like R, preserve this relationship, in R with terms objects. So, if RFE is used with one-hot coded columns, it should be done not at the column level, but at the terms level. With R, if you do the one-hot encoding not "by hand", yourself, but by declaring a factor variable and leaving the actual coding to R, the R in-built functions for stepwise modeling will use the terms structure and so do the right thing. If RFE is a good idea at all, is another question, see Are there any circumstances where stepwise regression should be used?