3
$\begingroup$

I couldn't find a better title, but here's the thing...

I was studying Elastic Net regularization and I found this function: $$ \text{Loss} = \sum_{i=0}^n \left(y_i - (wx_i + c)\right)^2 + \lambda_1 \sum_{j=0}^{m-1} \left|w_j\right| + \lambda_2 \sum_{j=0}^{m-1} w_j^2 \\ $$

Sorry about the image, but it's much easier this way. So I found this as being the loss function. However, there is no lambda 1, nor lambda 2 in Scikit-Learn. Instead, we find alpha and l1_ration. After studying more, I found that Scikit-Learn alpha is actually lambda and Scikit-Learn l1_ratio is actually alpha, both from this other equation:

$$ L_\text{enet} = \frac{1}{2n}\sum_{i=1}^n (y_i - x_i^T \beta)^2 + \lambda \left( \frac{1-\alpha}{2} \sum_{j=1}^m \beta_j^2+ \alpha \sum_{j=1}^m \left|\beta_j\right| \right) $$

So I guess the biggest question is: how do I go from the first equation to the last equation? How are these two equations connected?

I couldn't find a way to go from the first equation to the second one.

Again, sorry about the image, I know this isn't the best practice, but it was so much easier to just add them in here.

NOTE: Please consider w and beta as the same thing, the coefficients from the regression.

$\endgroup$
6
  • $\begingroup$ @Sycorax I tried writing the two equations and finding the relation, but how can I know the relation between these lambdas? Where is this established? I couldn't find. By writing both equations, the relationship between them is true if lambda 1 and lambda 2 are somehow related. $\endgroup$
    – Yuxxxxxx
    Aug 24 at 15:40
  • 1
    $\begingroup$ Ok, so edit your post to show your work & where you're stuck and we'll help you get un-stuck. $\endgroup$
    – Sycorax
    Aug 24 at 15:40
  • $\begingroup$ Well, this is the thing, I can't understand how the lambdas are related. Actually, as I said, I can't find how the two equations are related. $\endgroup$
    – Yuxxxxxx
    Aug 24 at 15:46
  • 1
    $\begingroup$ I tried to clean up the equations using Mathjax since I noticed that your images had $n$ used in two different ways, once for counting the number of observations and once for counting the number of coefficients, so I also fixed that; presumably, you are not restricted to having the same number of observations and coefficients. More information about Mathjax: math.meta.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    Aug 24 at 16:41
  • $\begingroup$ @Sycorax your efforts to replace the images with LaTeX do not go unnoticed. Thank you. $\endgroup$
    – mhdadk
    Aug 24 at 17:07
2
$\begingroup$

First, we're going to re-express these equations using common notation.

The coefficient vector $w$ does not include a constant element for an intercept. So we have to revise $x$ to contain a constant; likewise the relation between $w$ and $\beta$ is $\beta = [c ~ w]$ the concatenation of $w$ and $c$ the value of the intercept from the first equation. This means that $\beta$ has $m$ elements, indexed from 0: $0, 1, 2, \dots, m$ and we have $x_0=1$ and $\beta_0 =c$.

Rewriting the first equation, we have

$$\begin{align} \text{Loss} = L_A &= \sum_{i=0}^n \left(y_i - (w x_i + c)\right)^2 + \lambda_1 \sum_{j=0}^{m-1} \left|w_j\right| + \lambda_2 \sum_{j=0}^{m-1} w_j^2 \\ &= \sum_{i=0}^n \left(y_i - x_i^T\beta \right)^2 + \lambda_1 \sum_{j=1}^m \left|\beta_j\right| + \lambda_2 \sum_{j=1}^{m} \beta_j^2 \end{align} $$

Note that we've changed the indexing so that the intercept is not included in the penalty.

I think it's confusing to have $\lambda_1$, $\lambda_2$ and $\lambda$ all in the same context, so I'm going to rewrite $\lambda = \frac{1}{2n}\gamma$, which just re-scales $\lambda$ to be expressed units that depend on $n$. So our second equation is

$$ L_\text{enet} = L_B = \frac{1}{2n}\sum_{i=1}^n (y_i - x_i^T \beta)^2 + \frac{1}{2n} \gamma \left( \frac{1-\alpha}{2} \sum_{j=1}^m \beta_j^2+ \alpha \sum_{j=1}^m \left|\beta_j\right| \right) $$

When one chooses suitable hyperparameters, there is a unique global minimum for both equations. The first equation is strictly convex when at least one of $\lambda_1, \lambda_2$ is positive, and any remaining are 0. The second equation is strictly convex for $\gamma>0$ and $0 \le \alpha \le 1$.

We can show that this minimum has the same coefficient vector in either case when the hyper-parameters $\lambda_1, \lambda_2, \gamma, \alpha$ are well-chosen. This is not a coincidence -- it's because you can choose to re-write the expressions. In this sense, the equations are equivalent because they result in the same model, i.e. the same $\beta$.

Finally, because we only care about the the location of the minimum, but not the value of the minimum itself, these equations can be arbitrarily re-scaled. In other words, $L_A$ is proportional to $L_b$. I'll denote the scaling with some constant $C>0$.

$$ \begin{align} L_A &\propto L_B \\ L_A &= C L_B \\ \sum_{i=0}^n \left(y_i - x_i^T\beta \right)^2 + \lambda_1 \sum_{j=1}^n \left|\beta_j\right| + \lambda_2 \sum_{j=1}^n \beta_j^2 &= C \frac{1}{2n}\sum_{i=1}^n (y_i - x_i^T \beta)^2 + C \frac{1}{2n} \gamma \left( \frac{1-\alpha}{2} \sum_{j=1}^n \beta_j^2+ \alpha \sum_{j=1}^n \left|\beta_j \right| \right) \\ 2n\sum_{i=0}^n \left(y_i - x_i^T\beta \right)^2 + 2n\lambda_1 \sum_{j=1}^n \left|\beta_j\right| + 2n\lambda_2 \sum_{j=1}^n \beta_j^2 &= C \sum_{i=1}^n (y_i - x_i^T \beta)^2 + C \gamma \left( \frac{1-\alpha}{2} \sum_{j=1}^n \beta_j^2+ \alpha \sum_{j=1}^n \left|\beta_j\right| \right) \\ &= C \sum_{i=1}^n (y_i - x_i^T \beta)^2 + C\gamma \left( \frac{1-\alpha}{2} \sum_{j=1}^n \beta_j^2+ \alpha \sum_{j=1}^n \left|\beta_j\right| \right) \\ 2n\lambda_1 \sum_{j=1}^n \left|\beta_j\right| + 2n\lambda_2 \sum_{j=1}^n \beta_j^2 &= C\gamma \frac{1-\alpha}{2} \sum_{j=1}^n \beta_j^2+ C\gamma \alpha \sum_{j=1}^n \left|\beta_j\right| \end{align} $$ if we choose $C = 2n$. By inspection, we can now write $$ \lambda_1 = \gamma \alpha \\ \lambda_2 = \gamma\frac{1 -\alpha}{2} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.