Math behind applying elastic net penalties to logistic regression I understand how Ridge / Lasso / Elastic Net regression penalties are applied to the linear regression's cost function, but I am trying to figure out how they are applied to Logistic Regression's Maximum Likelihood cost function.
I've tried looking into pages through google, and it looks like it can be done (I believe Sci-Kit's logistic regression models accept L1 and L2 parameters, and I've seen some YouTube videos saying that the penalties can be applied in logistic models too) and I've found how they are added to the sum of squared residuals cost function, but I am curious on how the penalties are applied with the Maximum Likelihood cost function. Is it Maximum Likelihood minus the penalties?
 A: The elastic net terms are added to the maximum likelihood cost function.i.e. the final cost function is:
$\sum_{i = 0}^{N}\bigg[- (y\log(p) + (1-y)\log(1-p))\bigg] + \lambda_1 \sum_{i=0}^{k}|w_i| + \lambda_2 \sum_{i=0}^{k}w_i^2$
The first term is the likelihood, the second term is the $l_1$ norm part of the elastic net, and the third term is the $l_2$ norm part.
i.e. the network is trying to minimize the negative log likelihood and also trying to minimize the weights.
A: Yes, the penalties are simply added to the cost function (and negative/positive depending on whether you minimize or maximize the function).
You can view penalty terms in a cost function (e.g. costfunction like the likelihood) as being equivalent to the Lagrange multiplier equivalent of a problem like
$${\text{maximize} f(\beta) \text{ subject to $g(\beta) \leq t$ and $h(\beta) \leq t_2$}}$$
$$\begin{align} 
 f(\beta) &= \mathcal{L}(\beta \vert x) \\
 g(\beta) &= \vert\vert \beta \vert\vert_1\\
 h(\beta) &= \vert\vert \beta \vert\vert^2
\end{align}$$
In simple words. You maximize the log likelihood function with restrictions on the size of the quadratic norm of the coefficients $\vert\vert \beta \vert\vert^2$ (equivalent to ridge regression) and the size of the $l_1$ norm of the coefficients $\vert\vert \beta \vert\vert_1$ (equivalent to lasso).
See also Equivalence between Elastic Net formulations
