6
$\begingroup$

I understand how Ridge / Lasso / Elastic Net regression penalties are applied to the linear regression's cost function, but I am trying to figure out how they are applied to Logistic Regression's Maximum Likelihood cost function.

I've tried looking into pages through google, and it looks like it can be done (I believe Sci-Kit's logistic regression models accept L1 and L2 parameters, and I've seen some YouTube videos saying that the penalties can be applied in logistic models too) and I've found how they are added to the sum of squared residuals cost function, but I am curious on how the penalties are applied with the Maximum Likelihood cost function. Is it Maximum Likelihood minus the penalties?

Improve this question
$\endgroup$
1
  • $\begingroup$ The following section of this book has a description of maximum likelihood for logistic regression and likely answers your question. $\endgroup$
    – Adam Kells
    Commented Sep 23, 2021 at 12:46

2 Answers 2

Reset to default
7
$\begingroup$

The elastic net terms are added to the maximum likelihood cost function.i.e. the final cost function is:

$\sum_{i = 0}^{N}\bigg[- (y\log(p) + (1-y)\log(1-p))\bigg] + \lambda_1 \sum_{i=0}^{k}|w_i| + \lambda_2 \sum_{i=0}^{k}w_i^2$

The first term is the likelihood, the second term is the $l_1$ norm part of the elastic net, and the third term is the $l_2$ norm part.

i.e. the network is trying to minimize the negative log likelihood and also trying to minimize the weights.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Does it matter if the log loss sum divides by the sample size $N$? Certainly that does not matter in the non-regularized case, but I wonder if it matters when regularization is included. $\endgroup$
    – Dave
    Commented Sep 23, 2021 at 14:48
  • 1
    $\begingroup$ @Dave i think it does matter and it can affect the values we choose for $\lambda_1, \lambda_2$. At any rate $\lambda_1, \lambda_2$ are hyper parameters that are needed to be found using other evaluation metrics. $\endgroup$
    – ofer-a
    Commented Sep 23, 2021 at 15:21
  • $\begingroup$ Maybe that's the way it works: we will get different $\lambda$ values if we take the log loss part as a sum vs as mean, but will the cross validation resulting in those optimum $\lambda$ values give the same validation-set performance for the optimal $\lambda$ in each case? $\endgroup$
    – Dave
    Commented Sep 23, 2021 at 15:24
  • 1
    $\begingroup$ yes i believe that we will get very close results $\endgroup$
    – ofer-a
    Commented Sep 23, 2021 at 16:00
4
$\begingroup$

Yes, the penalties are simply added to the cost function (and negative/positive depending on whether you minimize or maximize the function).

You can view penalty terms in a cost function (e.g. costfunction like the likelihood) as being equivalent to the Lagrange multiplier equivalent of a problem like

$${\text{maximize} f(\beta) \text{ subject to $g(\beta) \leq t$ and $h(\beta) \leq t_2$}}$$

$$\begin{align} f(\beta) &= \mathcal{L}(\beta \vert x) \\ g(\beta) &= \vert\vert \beta \vert\vert_1\\ h(\beta) &= \vert\vert \beta \vert\vert^2 \end{align}$$

In simple words. You maximize the log likelihood function with restrictions on the size of the quadratic norm of the coefficients $\vert\vert \beta \vert\vert^2$ (equivalent to ridge regression) and the size of the $l_1$ norm of the coefficients $\vert\vert \beta \vert\vert_1$ (equivalent to lasso).

See also Equivalence between Elastic Net formulations

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.