The tutorial says the intersection point for L1 and L2 regularization gives the minimum loss - But why the intersection gives the minimum loss? I cannot interpret the graph clearly.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Possible duplicate of Lasso regression feature selection or stats.stackexchange.com/questions/45643/… $\endgroup$– Sycorax ♦Commented Oct 31, 2018 at 15:28
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
One way to view both ridge and the lasso is as the solution to the constrained equation $$ \widehat \beta = \operatorname{arg min}_{\beta} \|Y - X\beta\|^2_2 \quad \text{subject to $\|\beta\| \le \eta$} $$ where $\|\cdot\|$ is, in either case, the $\ell_1$ or $\ell_2$ norm. The usual penalization characterization of this $$ \widehat \beta = \arg \min_\beta \|Y - X\beta\|^2_2 + \lambda \|\beta\|_p^p $$ ($p = 1$ for the lasso and $p = 2$ for ridge) is just the formulation of the optimization in terms of a Lagrangian, where $\lambda = \lambda(\eta)$ is a function of $\eta$. Anyway, once you have the constraint formulation, it should be clear why the solution is the intersection: we are finding the contour with the smallest value which satisfies the constraint. The points inside the $\ell_1$ or $\ell_2$ ball are the points which satisfy the constraint, and we want to pick the solution $\widehat \beta$ which is on the contour of smallest value. When the least squares solution does not lie in the $\ell_p$ ball itself, this will occur on the boundary, where one of the contours intersects the boundary of the ball.
-
$\begingroup$ Thanks, when they are separated say $$\widehat \beta = \arg \min_\beta \|Y - X\beta\|^2_2$$ and $$\widehat \beta = \lambda \|\beta\|_p^p$$ I suppose the solution will be the center for the square and ellipse (for L1), and the center for the circle and ellipse (for L2). What I am still not very sure is when they sum together, why the solution lies in their intersection and not some non-intersection point? $\endgroup$ Commented Oct 31, 2018 at 23:09
-
$\begingroup$ @william007 Again, you want to think of it as constrained minimization, not in terms of the Lagrangian, if you want to get this straight. There is no ball directly associated with the Lagrangian. If you give me the Lagrangian, I cannot give you the corresponding circle, at least without you also giving me the data. It is only through the formulation as a constrained optimization that you get this: there exists a $\lambda$ such that the constrained optimizer is the solution to the optimization. $\endgroup$– guyCommented Nov 1, 2018 at 0:43
-
$\begingroup$ Thanks guy, and thinking in terms of constrained optimization, why the solution lies on their intersection, but not non-intersection points? $\endgroup$ Commented Nov 1, 2018 at 0:50
-
$\begingroup$ @william007 it must lie in the ball by definition; that literally is what the constraint is saying. There are two possibilities: either the unconstrained optimizer is in the ball (in which case it is also the constrained optimizer and is not necessarily on the boundary), or it is not, in which you cannot use it. In the latter case, you need to find the value of the objective which is minimal and lands in the feasible region, which can be done by finding the first countour which intersects the ball. $\endgroup$– guyCommented Nov 1, 2018 at 1:08
$\begingroup$
$\endgroup$
Minimizing the loss function $f(\theta)$ with regularization function $g(\theta)$ can be viewed as minimizing a Lagrange Function.
$\mathcal{L}(\theta,\lambda) = f(\theta) - \lambda \cdot g(\theta)$
you can then minimize by taking the gradient
$\nabla_{\theta,\lambda} \mathcal{L}(\theta, \lambda)=0$.
In a Lagrange Function, the optimum occurs when the gradient of the loss function is perpendicular to the regularization function.
$\nabla_{\theta} f = \lambda \nabla_{\theta} g,$
Here is a much better explanation by Dikran Marsupial :What is the connection between regularization and the method of Lagrange multipliers ? Just a note it this is a very general explanation.
answered Oct 31, 2018 at 15:36