Logistic regression is a convex optimization problem and adding elastic net penalties is adding convex elements. So am I right if I conclude that logistic regression with elastic net penalties is also convex, and does this only hold in the special cases of the L1 and L2 penalties?
$\begingroup$
$\endgroup$
2
asked May 23, 2017 at 18:33
-
7$\begingroup$ The sum of convex functions is a convex function. $\endgroup$– Mark L. StoneCommented May 23, 2017 at 18:36
-
1$\begingroup$ @MarkL.Stone, why not turn that into an official answer? Then this won't be counted as unanswered. $\endgroup$– gung - Reinstate MonicaCommented May 23, 2017 at 18:53
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Since the loss used in logistic regression, the $\ell_1$, and the $\ell_2$ norm are convex, any non-negative combination of those terms defines a convex problem. Since $0$ is technically non-negative, this also means you can remove any of the terms from the objective and still have a convex problem.
Here's Why.
Suppose that $\{f_1,\dots,f_n\}$ are a set of real-valued convex functions. Then,
$$f(x) = \sum_{i=1}^n \lambda_i f_i(x)$$
is convex in $x\in\mathbb{R}^n$ for scalars $\lambda_i \geq 0$. You can see why this is the case simply by applying Jensen's inequality. Specifically, we know that for any $x$ and $y$,
$$\lambda_i f_i\left(\theta x + (1-\theta)y\right) \leq \theta \lambda_i f_i(x) + (1-\theta) \lambda_i f_i(y)$$
for all $i \in \{1,\dots,n\}$, where $\theta \in (0,1)$. Substituting this into the sum, we have
\begin{align} f\left(\theta x + (1-\theta)y\right) &= \sum_i \lambda_i f_i\left(\theta x + (1-\theta)y\right) \\ &\leq \sum_i \theta\lambda_i f_i\left(x\right) + (1-\theta)\lambda_i f\left(y\right) \\ &= \theta f(x) + (1-\theta) f(y) \end{align}
meaning that $f$ satisfies Jensen's inequality and is therefore convex. This holds for the special case Mark mentioned in the comment, for which $\lambda_i = 1$ for all $i$.