0
$\begingroup$

In a regular regression or logistic regression, if I have a categorical variable, say [Red, Blue, Green], I can only include two of them in my model so that Y = C + Blue + Green.

Do I also need to leave one out in LASSO or can I include all 3 colors? If I include all 3, can I still have an intercept? Or do I need to get rid of it?

Thanks!

$\endgroup$
1
$\begingroup$

When using LASSO, you do not need to drop one level from your categorical variable.

The practice of dropping a level is known as dummy coding. Here are a couple of references that say that dummy coding is not necessary when you are using regularization (for example, when using LASSO):

From the book Machine Learning Design Patterns

Because dummy coding is a more compact representation, it is preferred in statistical models that perform better when the inputs are linearly independent.

Modern machine learning algorithms, though, don’t require their inputs to be linearly independent and use methods such as L1 regularization to prune redundant inputs

From the blog post Think twice before dropping that first one-hot encoded column

Consequently, if we apply the tiniest bit of regularization (whether it's ℓ2, ℓ1, or elastic net), we can handle features that are perfectly correlated without removing any columns. Regularization also innately addresses the effects of multicollinearity—it's pretty awesome.

But if you are regularizing, there's no need to drop one of the one-hot encoded columns from each categorical feature—math's got your back.

$\endgroup$
1
  • $\begingroup$ Thanks, keeping it in would make life easy. Do I need to drop the intercept then for LASSO if I keep it in? Seems like in the article they don't have an intercept. $\endgroup$
    – confused
    Sep 19 at 16:23
0
$\begingroup$

The thing is that not "including" the omited variable its because its the reference level. Think of it as the control one.

Mostly people do a dummify and leave one out. But in using Lasso, as far as im concerned, with factors, you should do a grouped lasso. (Wich i have to look it up too)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.