Regularization methods for logistic regression

Regularization using methods such as Ridge, Lasso, ElasticNet is quite common for linear regression. I wanted to know the following: Are these methods applicable for logistic regression? If so, are there any differences in the way they need to be used for logistic regression? If these methods are not applicable, how does one regularize a logistic regression?

• Are you looking at a particular data set, and thus need to consider making the data tractable for computation, e.g. selecting, scaling and offsetting the data so that the initial computaion tends to succeed. Or is this a more general look at the hows and whys (without a specific data set to compute against0? – Philip Oakley Aug 8 '16 at 14:04
• This is a more general look at the hows and whys of regularization. Introductory texts for regularization methods (ridge, Lasso, Elasticnet etc.) that I came across specifically mentioned linear regression examples. Not a single one mentioned logistic specifically, hence the question. – TAK Aug 8 '16 at 18:49
• Logistic Regression is a form of GLM using a non-identity link function, almost everything applies. – Firebug Aug 8 '16 at 19:29
• Have you stumbled upon Andrew Ng's video on the topic? – Antoni Parellada Aug 17 '16 at 19:15
• Ridge, lasso and elastic net regression are popular options, but they aren't the only regularization options. For example, smoothing matrices penalize functions with large second derivatives, so that the regularization parameter allows you to "dial in" a regression which is a nice compromise between over- and under-fitting the data. As with ridge/lasso/elastic net regression, these can also be used with logistic regression. – Sycorax Jan 18 '17 at 15:03

Yes, Regularization can be used in all linear methods, including both regression and classification. I would like to show you that there are not too much difference between regression and classification: the only difference is the loss function.

Specifically, there are three major components of linear method, Loss Function, Regularization, Algorithms. Where loss function plus regularization is the objective function in the problem in optimization form and the algorithm is the way to solve it (the objective function is convex, we will not discuss in this post).

In loss function setting, we can have different loss in both regression and classification cases. For example, Least squares and least absolute deviation loss can be used for regression. And their math representation are $L(\hat y,y)=(\hat y -y)^2$ and $L(\hat y,y)=|\hat y -y|$. (The function $L( \cdot )$ is defined on two scalar, $y$ is ground truth value and $\hat y$ is predicted value.)

On the other hand, logistic loss and hinge loss can be used for classification. Their math representations are $L(\hat y, y)=\log (1+ \exp(-\hat y y))$ and $L(\hat y, y)= (1- \hat y y)_+$. (Here, $y$ is the ground truth label in $\{-1,1\}$ and $\hat y$ is predicted "score". The definition of $\hat y$ is a little bit unusual, please see the comment section.)

In regularization setting, you mentioned about the L1 and L2 regularization, there are also other forms, which will not be discussed in this post.

Therefore, in a high level a linear method is

$$\underset{w}{\text{minimize}}~~~ \sum_{x,y} L(w^{\top} x,y)+\lambda h(w)$$

If you replace the Loss function from regression setting to logistic loss, you get the logistic regression with regularization.

For example, in ridge regression, the optimization problem is

$$\underset{w}{\text{minimize}}~~~ \sum_{x,y} (w^{\top} x-y)^2+\lambda w^\top w$$

If you replace the loss function with logistic loss, the problem becomes

$$\underset{w}{\text{minimize}}~~~ \sum_{x,y} \log(1+\exp{(-w^{\top}x \cdot y)})+\lambda w^\top w$$

Here you have the logistic regression with L2 regularization.

This is how it looks like in a toy synthesized binary data set. The left figure is the data with the linear model (decision boundary). The right figure is the objective function contour (x and y axis represents the values for 2 parameters.). The data set was generated from two Gaussian, and we fit the logistic regression model without intercept, so there are only two parameters we can visualize in the right sub-figure.

The blue lines are the logistic regression without regularization and the black lines are logistic regression with L2 regularization. The blue and black points in right figure are optimal parameters for objective function.

In this experiment, we set a large $\lambda$, so you can see two coefficients are close to $0$. In addition, from the contour, we can observe the regularization term is dominated and the whole function is like a quadratic bowl. Here is another example with L1 regularization. Note that, the purpose of this experiment is trying to show how the regularization works in logistic regression, but not argue regularized model is better.

Here are some animations about L1 and L2 regularization and how it affects the logistic loss objective. In each frame, the title suggests the regularization type and $\lambda$, the plot is objective function (logistic loss + regularization) contour. We increase the regularization parameter $\lambda$ in each frame and the optimal solution will shrink to $0$ frame by frame.

Some notation comments. $w$ and $x$ are column vectors,$y$ is a scalar. So the linear model $\hat y = f(x)=w^\top x$. If we want to include the intercept term, we can append $1$ as a column to the data.

In regression setting, $y$ is a real number and in classification setting $y \in \{-1,1\}$.

Note it is a little bit strange for the definition of $\hat y=w^{\top} x$ in classification setting. Since most people use $\hat y$ to represent a predicted value of $y$. In our case, $\hat y = w^{\top} x$ is a real number, but not in $\{-1,1\}$. We use this definition of $\hat y$ because we can simplify the notation on logistic loss and hinge loss.

Also note that, in some other notation system, $y \in \{0,1\}$, the form of the logistic loss function would be different.

The code can be found in my other answer here.

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

• Nice answer, but I don't see anything (except notation) in the explanation that suggests you need to be limited to linear methods. Can you have the same answer but with w replaced by f, and $w^Tx$ replaced by $f(x)$? People certainly use regularization for nonlinear methods, such as neural networks. – DavidR Aug 9 '16 at 21:59
• Thanks for the answer @hxd1011, can you please explain what the solid black lines represent in the contour graph? To be more precise, I know that (as you explained) x and y axis show the 2 parameters we use. But what about the solid lines and their numbers like 8000, 10000, 12000. Thanks! – Jespar Jun 4 '17 at 17:23
• – Haitao Du Jun 4 '17 at 17:55

A shrinkage/regularization method that was originally proposed for logistic regression based on considerations of higher order asymptotic was Firth logistic regression... some while before all of these talks about lasso and what not started, although after ridge regression risen and subsided in popularity through 1970s. It amounted to adding a penalty term to the likelihood, $$l^*(\beta) = l(\beta) + \frac12 \ln |i(\beta)|$$ where $i(\beta) = \frac1n \sum_i p_i (1-p_i) x_i x_i'$ is the information matrix normalized per observation. Firth demonstrated that this correction has a Bayesian interpretation in that it corresponds to Jeffreys prior shrinking towards zero. The excitement it generated was due to it helping fixing the problem of perfect separation: say a dataset $\{(y_i,x_i)\| = \{(1,1),(0,0)\}$ would nominally produce infinite ML estimates, and glm in R is still susceptible to the problem, I believe.

• Yes you can use R packages logistf or brglm for this! Maybe worth mentioning... – Tom Wenseleers Apr 5 '17 at 16:20
• It's very elegant, but it's quite slow for optimisation isn't it? In the gradient you get the inverse of $i(\beta)$ which needs to be recomputed in every iteration... – appletree Feb 23 at 20:27
• It is painfully slow, indeed, @appletree – StasK Mar 4 at 15:18
• (+1) I had not heard of Firth's correction before. I would not expect the implied approach in the cited paper to add much time to the GLM solution? (You do not need to invert a matrix, just compute leverages. If solving the GLM via iteratively reweighted least squares, these are just the row-norms of the Q factor. The leverages then just add to the data and exposure, using $h/2$ instead of $1/2$ in the Jeffreys adjustment.) – GeoMatt22 Mar 19 at 2:39

Yes, it is applicable to logistic regression. In R, using glmnet, you simply specify the appropriate family which is "binomial" for logistic regression. There are a couple of others (poison, multinomial, etc) that you can specify depending on your data and the problem you are addressing.

• Downside of just using glmnet for this is that that approach would not give you significance levels. If you're interested in those, then R packages logistf or brglm would be a better way to go... – Tom Wenseleers Apr 5 '17 at 16:21
• @TomWenseleers there are also methods for bootstrapping glmnet to obtain p-values. However, it is a little complicated as "regular" bootstrap doesn't work for lasso coefficients – godspeed Apr 5 '17 at 16:24
• Thanks for letting me know about that, I've seen others mention this too, e.g. here: stats.stackexchange.com/questions/34859/…, but can't seem to find anything standard implemented in some R package. Would you happen to have any pointers? Or good primary literature on this? Downside of bootstrapping is also that it would typically be very slow for large datasets... – Tom Wenseleers Apr 5 '17 at 19:22
• Are you referring to methods such as those implemented in R package hdi, cran.r-project.org/web/packages/hdi/index.html ? – Tom Wenseleers Apr 6 '17 at 14:33