# Intersection of MSE-Loss and Regularization Term

In several questions [1,2] the graphical intuition of the L1/L2 regularization has been discussed. But, for example in [1], it has been stated that:

The solution to the constrained optimization lies at the intersection between the contours of the two functions

And this is basically what I wanted to understand. Why is that the case? In general, if you want to minimize $$f(x) = g(x) + h(x)$$ - like in regularization - the minimum of $$f$$ is not the sum of the minimums of $$g$$ and $$h$$. Why can we just make the statement quoted above in regularization, as $$loss = mse + regularization\text{ }term$$?

For example, if you have two weights $$w_1, w_2$$ in a neural network and $$Loss(w_1,w_2)$$, you can interpret the minimum as the intersection of the MSE function and the regularization function $$\mathbb{R}^2 \rightarrow \mathbb{R}$$ which is for me really unintuitive, as again, for me, in general, the minimum of the sum of two functions is not their intersection or something like that.

Thank you so much.

I think the confusion here is mostly from the two different forms of regularization. The images in both your linked questions represent the constrained optimization form; they are looking for $$\min_x g(x) \text{ s.t. } h(x)\leq t$$, not $$\min_x g(x)+\lambda h(x)$$. And for the former, it's clear from a picture that the minimum will occur where a contour of $$g$$ meets the contour $$h=t$$ tangentially (provided the minimum of $$g$$ doesn't already lie inside the region $$h(x)\leq t$$, that is, and using that the loss function is convex). In [1], the image shows multiple values of $$t$$ to demonstrate the "regularization path," that lasso brings and then keeps a coefficient at 0.
• Thank you for your answer. Unfortunately, I do not see for example in [1] why they are discussing a constrained optimization problem. For example the accepted answer in [1] say that Lasso minimization is the minimization of $OLS + L_1$, which is also what I stated, isn't it? My question originates from Deep Learning, where we want to regularize the MSE. And there, we had the exact same picture, for mapping two weights $(w_1, w_2) \rightarrow MSE$ and $(w_1, w_2) \rightarrow L1$ and then took the intersection/tangential (if we project it onto a 2d plane)... Nov 12 '20 at 14:34
• Thank you so much @Ben Reiniger. I think I finally understood. Am I correct in the assumption that in the case that the minimum of $g$ lies inside the region $h(x) \leq t$, the solution of the constraint optimization problem is just the minimum of $g$ itself? (For anyone else who had the same problem as I did, stats.stackexchange.com/questions/191603/… really helps in understanding the equivalence of the two optimization problems) Nov 23 '20 at 17:31