5
$\begingroup$

Scikit's LogisticRegression regularizes the intercept term.

Is it possible to treat these coefficient separately (as far a regularization goes).

There are two scenario I can see that may be useful:

  • the intercept term is set to the value that predicts the population mean
  • the intercept term is un-regularized (it is allowed to have large values).

(Would this first follow from the second for LR?)

Why. If the purpose of regularization is to reduce the dependence of predictions on any one factor, we should also like to encourage (and force) a stronger default bias towards 'not explained by any of the factors'. Because the intercept is not a (variable) input factor, I should prefer to allow greater magnitude weights on the intercept than the other factors.

While this might cost us training loss, we may see lower variance in the coefficient due to training set variance.

$\endgroup$
  • 2
    $\begingroup$ sklearn regularizing the intercept is a design mistake, plain and simple. It's a pain in the ass. $\endgroup$ – Matthew Drury Sep 13 '17 at 16:44
  • $\begingroup$ I wouldn't go that far -- if your goal is only loss minimization, it may make sense. I agree its a PITA though. $\endgroup$ – user48956 Sep 13 '17 at 16:45
  • 2
    $\begingroup$ It think it's definitely a design mistake. The default and natural use should support the most common case. It is by far the most common case to not regularize the intercept. $\endgroup$ – Matthew Drury Sep 13 '17 at 17:24
  • $\begingroup$ Perhaps -- I think the bigger design mistake is to assume a common use case, instead of allowing for more use cases. I see this a lot in scikit -- a specific implementation is hard-baked into many parts. Want special handling of NULL attributes in decision trees? Nope. $\endgroup$ – user48956 Sep 13 '17 at 17:32
2
$\begingroup$

There is one quick and dirty fix to make less regularization on intercept.

To do it, we can append a huge number in intercept column say $1 \times 10^{9}$, instead of expand the data frame with all $1$ column.

For example, with mtcars data, if we want to do logistic regression respect to weight, we do

> head(cbind(1e9,mtcars$wt))
      [,1]  [,2]
[1,] 1e+09 2.620
[2,] 1e+09 2.875
[3,] 1e+09 2.320
[4,] 1e+09 3.215
[5,] 1e+09 3.440
[6,] 1e+09 3.460

instead of

> head(model.matrix(am~wt,mtcars))
                  (Intercept)    wt
Mazda RX4                   1 2.620
Mazda RX4 Wag               1 2.875
Datsun 710                  1 2.320
Hornet 4 Drive              1 3.215
Hornet Sportabout           1 3.440
Valiant                     1 3.460
$\endgroup$
  • 5
    $\begingroup$ This hack makes me so sad. $\endgroup$ – Matthew Drury Sep 13 '17 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.