# Is logstic regression with elasticnet penalty a convex optimization problem?

Logistic regression is a convex optimization problem and adding elastic net penalties is adding convex elements. So am I right if I conclude that logistic regression with elastic net penalties is also convex, and does this only hold in the special cases of the L1 and L2 penalties?

• The sum of convex functions is a convex function. – Mark L. Stone May 23 '17 at 18:36
• @MarkL.Stone, why not turn that into an official answer? Then this won't be counted as unanswered. – gung - Reinstate Monica May 23 '17 at 18:53

Since the loss used in logistic regression, the $\ell_1$, and the $\ell_2$ norm are convex, any non-negative combination of those terms defines a convex problem. Since $0$ is technically non-negative, this also means you can remove any of the terms from the objective and still have a convex problem.

Here's Why.

Suppose that $\{f_1,\dots,f_n\}$ are a set of real-valued convex functions. Then,

$$f(x) = \sum_{i=1}^n \lambda_i f_i(x)$$

is convex in $x\in\mathbb{R}^n$ for scalars $\lambda_i \geq 0$. You can see why this is the case simply by applying Jensen's inequality. Specifically, we know that for any $x$ and $y$,

$$\lambda_i f_i\left(\theta x + (1-\theta)y\right) \leq \theta \lambda_i f_i(x) + (1-\theta) \lambda_i f_i(y)$$

for all $i \in \{1,\dots,n\}$, where $\theta \in (0,1)$. Substituting this into the sum, we have

\begin{align} f\left(\theta x + (1-\theta)y\right) &= \sum_i \lambda_i f_i\left(\theta x + (1-\theta)y\right) \\ &\leq \sum_i \theta\lambda_i f_i\left(x\right) + (1-\theta)\lambda_i f\left(y\right) \\ &= \theta f(x) + (1-\theta) f(y) \end{align}

meaning that $f$ satisfies Jensen's inequality and is therefore convex. This holds for the special case Mark mentioned in the comment, for which $\lambda_i = 1$ for all $i$.