Logistic regression is a convex optimization problem and adding elastic net penalties is adding convex elements. So am I right if I conclude that logistic regression with elastic net penalties is also convex, and does this only hold in the special cases of the L1 and L2 penalties?

  • 7
    $\begingroup$ The sum of convex functions is a convex function. $\endgroup$ – Mark L. Stone May 23 '17 at 18:36
  • 1
    $\begingroup$ @MarkL.Stone, why not turn that into an official answer? Then this won't be counted as unanswered. $\endgroup$ – gung - Reinstate Monica May 23 '17 at 18:53

Since the loss used in logistic regression, the $\ell_1$, and the $\ell_2$ norm are convex, any non-negative combination of those terms defines a convex problem. Since $0$ is technically non-negative, this also means you can remove any of the terms from the objective and still have a convex problem.

Here's Why.

Suppose that $\{f_1,\dots,f_n\}$ are a set of real-valued convex functions. Then,

$$f(x) = \sum_{i=1}^n \lambda_i f_i(x)$$

is convex in $x\in\mathbb{R}^n$ for scalars $\lambda_i \geq 0$. You can see why this is the case simply by applying Jensen's inequality. Specifically, we know that for any $x$ and $y$,

$$\lambda_i f_i\left(\theta x + (1-\theta)y\right) \leq \theta \lambda_i f_i(x) + (1-\theta) \lambda_i f_i(y)$$

for all $i \in \{1,\dots,n\}$, where $\theta \in (0,1)$. Substituting this into the sum, we have

\begin{align} f\left(\theta x + (1-\theta)y\right) &= \sum_i \lambda_i f_i\left(\theta x + (1-\theta)y\right) \\ &\leq \sum_i \theta\lambda_i f_i\left(x\right) + (1-\theta)\lambda_i f\left(y\right) \\ &= \theta f(x) + (1-\theta) f(y) \end{align}

meaning that $f$ satisfies Jensen's inequality and is therefore convex. This holds for the special case Mark mentioned in the comment, for which $\lambda_i = 1$ for all $i$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.