Scikit's LogisticRegression regularizes the intercept term.

Is it possible to treat these coefficient separately (as far a regularization goes).

There are two scenario I can see that may be useful:

  • the intercept term is set to the value that predicts the population mean
  • the intercept term is un-regularized (it is allowed to have large values).

(Would this first follow from the second for LR?)

Why. If the purpose of regularization is to reduce the dependence of predictions on any one factor, we should also like to encourage (and force) a stronger default bias towards 'not explained by any of the factors'. Because the intercept is not a (variable) input factor, I should prefer to allow greater magnitude weights on the intercept than the other factors.

While this might cost us training loss, we may see lower variance in the coefficient due to training set variance.

  • 4
    $\begingroup$ sklearn regularizing the intercept is a design mistake, plain and simple. It's a pain in the ass. $\endgroup$ Sep 13 '17 at 16:44
  • $\begingroup$ I wouldn't go that far -- if your goal is only loss minimization, it may make sense. I agree its a PITA though. $\endgroup$
    – user48956
    Sep 13 '17 at 16:45
  • 2
    $\begingroup$ It think it's definitely a design mistake. The default and natural use should support the most common case. It is by far the most common case to not regularize the intercept. $\endgroup$ Sep 13 '17 at 17:24
  • $\begingroup$ Perhaps -- I think the bigger design mistake is to assume a common use case, instead of allowing for more use cases. I see this a lot in scikit -- a specific implementation is hard-baked into many parts. Want special handling of NULL attributes in decision trees? Nope. $\endgroup$
    – user48956
    Sep 13 '17 at 17:32

Use a solver other than liblinear. The liblinear solver (formerly the default for LogisticRegression) regularizes the intercept, but the others do not (see https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression). If you must use liblinear, see the intercept_scaling argument, which presumably is included to mitigate this issue.

  • 1
    $\begingroup$ Yes. This is the right think. Note ‘liblinear’ was the default (and its limitations poorly documented when I first posted there. They've since made ‘lbfgs' the default, which is an improvement. $\endgroup$
    – user48956
    Apr 17 '20 at 21:20

There is one quick and dirty fix to make less regularization on intercept.

To do it, we can append a huge number in intercept column say $1 \times 10^{9}$, instead of expand the data frame with all $1$ column.

For example, with mtcars data, if we want to do logistic regression respect to weight, we do

> head(cbind(1e9,mtcars$wt))
      [,1]  [,2]
[1,] 1e+09 2.620
[2,] 1e+09 2.875
[3,] 1e+09 2.320
[4,] 1e+09 3.215
[5,] 1e+09 3.440
[6,] 1e+09 3.460

instead of

> head(model.matrix(am~wt,mtcars))
                  (Intercept)    wt
Mazda RX4                   1 2.620
Mazda RX4 Wag               1 2.875
Datsun 710                  1 2.320
Hornet 4 Drive              1 3.215
Hornet Sportabout           1 3.440
Valiant                     1 3.460
  • 6
    $\begingroup$ This hack makes me so sad. $\endgroup$ Sep 13 '17 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.