Skip regularization for some features for generalized linear models in SkLearn

Question

I'm working on a time-series forecasting project, where we found regularized linear models to work well. But we have a feature that has small but important impact for some period. We want to be sure this column is not regularized to 0.

Is there any way to skip regularization for generalized linear models, such as Lasso, Ridge, ElasticNet in the SkLearn library? I can't find the option in the documentation. What would be a good workaround if this option is not available?

Answer 1

You can completely skip regularization for all features using LinearRegression, or setting alpha=0 in Ridge, Lasso or ElasticNet.

You can't skip regularization for only a subset of features with scikit-learn's API.

---

Note that using Lasso leads to many coefficients to exactly zero, whereas Ridge leads to small values, but with almost no zeros, so more features are kept in the game. ElasticNet is a mix of both regularization, balanced with l1_ratio parameter. So try using Ridge, or decrease the regularization parameter alpha.

As an experimental workaround, you may also try to balance regularization between features by scaling the features of interest (and obviously use normalize=False): Multiplying a feature by 0.1 should increase the corresponding coefficient by 10 (with no regularization), so the regularization should be stronger on this feature. I am curious if it yields the desired effect.