# How avoid regularizing intercept in scikit's LogisticRegression

Scikit's LogisticRegression regularizes the intercept term.

Is it possible to treat these coefficient separately (as far a regularization goes).

There are two scenario I can see that may be useful:

• the intercept term is set to the value that predicts the population mean
• the intercept term is un-regularized (it is allowed to have large values).

(Would this first follow from the second for LR?)

Why. If the purpose of regularization is to reduce the dependence of predictions on any one factor, we should also like to encourage (and force) a stronger default bias towards 'not explained by any of the factors'. Because the intercept is not a (variable) input factor, I should prefer to allow greater magnitude weights on the intercept than the other factors.

While this might cost us training loss, we may see lower variance in the coefficient due to training set variance.

• sklearn regularizing the intercept is a design mistake, plain and simple. It's a pain in the ass. – Matthew Drury Sep 13 '17 at 16:44
• I wouldn't go that far -- if your goal is only loss minimization, it may make sense. I agree its a PITA though. – user48956 Sep 13 '17 at 16:45
• It think it's definitely a design mistake. The default and natural use should support the most common case. It is by far the most common case to not regularize the intercept. – Matthew Drury Sep 13 '17 at 17:24
• Perhaps -- I think the bigger design mistake is to assume a common use case, instead of allowing for more use cases. I see this a lot in scikit -- a specific implementation is hard-baked into many parts. Want special handling of NULL attributes in decision trees? Nope. – user48956 Sep 13 '17 at 17:32

Use a solver other than liblinear. The liblinear solver (formerly the default for LogisticRegression) regularizes the intercept, but the others do not (see https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression). If you must use liblinear, see the intercept_scaling argument, which presumably is included to mitigate this issue.

• Yes. This is the right think. Note ‘liblinear’ was the default (and its limitations poorly documented when I first posted there. They've since made ‘lbfgs' the default, which is an improvement. – user48956 Apr 17 '20 at 21:20

There is one quick and dirty fix to make less regularization on intercept.

To do it, we can append a huge number in intercept column say $1 \times 10^{9}$, instead of expand the data frame with all $1$ column.

For example, with mtcars data, if we want to do logistic regression respect to weight, we do

> head(cbind(1e9,mtcars\$wt))
[,1]  [,2]
[1,] 1e+09 2.620
[2,] 1e+09 2.875
[3,] 1e+09 2.320
[4,] 1e+09 3.215
[5,] 1e+09 3.440
[6,] 1e+09 3.460


> head(model.matrix(am~wt,mtcars))