Scikit's LogisticRegression regularizes the intercept term.
Is it possible to treat these coefficient separately (as far a regularization goes).
There are two scenario I can see that may be useful:
- the intercept term is set to the value that predicts the population mean
- the intercept term is un-regularized (it is allowed to have large values).
(Would this first follow from the second for LR?)
Why. If the purpose of regularization is to reduce the dependence of predictions on any one factor, we should also like to encourage (and force) a stronger default bias towards 'not explained by any of the factors'. Because the intercept is not a (variable) input factor, I should prefer to allow greater magnitude weights on the intercept than the other factors.
While this might cost us training loss, we may see lower variance in the coefficient due to training set variance.