1
$\begingroup$

In several questions [1,2] the graphical intuition of the L1/L2 regularization has been discussed. But, for example in [1], it has been stated that:

The solution to the constrained optimization lies at the intersection between the contours of the two functions

And this is basically what I wanted to understand. Why is that the case? In general, if you want to minimize $f(x) = g(x) + h(x)$ - like in regularization - the minimum of $f$ is not the sum of the minimums of $g$ and $h$. Why can we just make the statement quoted above in regularization, as $loss = mse + regularization\text{ }term$?

For example, if you have two weights $w_1, w_2$ in a neural network and $Loss(w_1,w_2)$, you can interpret the minimum as the intersection of the MSE function and the regularization function $\mathbb{R}^2 \rightarrow \mathbb{R}$ which is for me really unintuitive, as again, for me, in general, the minimum of the sum of two functions is not their intersection or something like that.

Thank you so much.

[1] The graphical intuiton of the LASSO in case p=2
[2] L1 L2 regularization

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think the confusion here is mostly from the two different forms of regularization. The images in both your linked questions represent the constrained optimization form; they are looking for $\min_x g(x) \text{ s.t. } h(x)\leq t$, not $\min_x g(x)+\lambda h(x)$. And for the former, it's clear from a picture that the minimum will occur where a contour of $g$ meets the contour $h=t$ tangentially (provided the minimum of $g$ doesn't already lie inside the region $h(x)\leq t$, that is, and using that the loss function is convex). In [1], the image shows multiple values of $t$ to demonstrate the "regularization path," that lasso brings and then keeps a coefficient at 0.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. Unfortunately, I do not see for example in [1] why they are discussing a constrained optimization problem. For example the accepted answer in [1] say that Lasso minimization is the minimization of $OLS + L_1$, which is also what I stated, isn't it? My question originates from Deep Learning, where we want to regularize the MSE. And there, we had the exact same picture, for mapping two weights $(w_1, w_2) \rightarrow MSE$ and $(w_1, w_2) \rightarrow L1$ and then took the intersection/tangential (if we project it onto a 2d plane)... $\endgroup$
    – Maxbit
    Commented Nov 12, 2020 at 14:34
  • $\begingroup$ The two forms are equivalent, as the answers in [2] observe. The graphics are useful (only?) when viewing the problem in the constrained form. $\endgroup$
    – Ben Reiniger
    Commented Nov 13, 2020 at 15:16
  • $\begingroup$ In the context of deep neural networks, the picture may be much more complicated: the loss contours needn't be such nice ellipses, I think. But the general idea is the same. $\endgroup$
    – Ben Reiniger
    Commented Nov 13, 2020 at 15:18
  • 1
    $\begingroup$ Thank you so much @Ben Reiniger. I think I finally understood. Am I correct in the assumption that in the case that the minimum of $g$ lies inside the region $h(x) \leq t$, the solution of the constraint optimization problem is just the minimum of $g$ itself? (For anyone else who had the same problem as I did, stats.stackexchange.com/questions/191603/… really helps in understanding the equivalence of the two optimization problems) $\endgroup$
    – Maxbit
    Commented Nov 23, 2020 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.