2
$\begingroup$

$\textrm{min} \ (\ \sum \limits_{i=1}^m (y_i - \boldsymbol x_i^T \boldsymbol \theta)^2 + \lambda \sum \limits_{j=1}^n \theta_j^2 \ )$

By searching around the internet I could figure out that the solution to the above Lagrange formulation (Ridge regression) lies at the intersection between the contours of the two summands.

I know that the above represents the Lagrangian formulation of the underlying minimization problem. Clearly, by considering the constrained formulation it makes sense for me why the solution is found at the intersection of the corresponding contours. That is also explained here: L1 L2 regularization

But, by just having the Lagrange formulation as above, how can it be seen that the minimum is found at the intersection? In general, the minimum of the sum of two functions is not always found at the intersection of the corresponding contours.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You have omitted the Lagrange multiplier from the equation. Please add it back in so that the question will make more sense. But note it would only be a Lagrangian formulation if the minimization was over $\lambda$ as well as $\theta_j$. $\endgroup$
    – Gordon Smyth
    Commented Jun 4, 2022 at 8:53
  • $\begingroup$ This is a general property of Lagrange multipliers. $\endgroup$
    – whuber ♦
    Commented Jun 4, 2022 at 11:54

1 Answer 1

Reset to default
2
$\begingroup$

Typically, those figures do not show the complete picture. It looks like the following:

Ridge Regression

There are an infinite number of contours belonging to both the MSE (elliptic ones) and the L2 regularization (circular ones). So, they don't intersect at a single point. Because for a given point on the ellipse, you can find a circle passing at that point.

They should be tangents. However, still, they don't intersect at a single point. Because, for a given MSE contour, you can gradually increase the size of the L2 circles and find their touch point.

The solution exists, for a given lambda, when the derivative of the Lagrangian is equal to zero. That is, when

$$L=f(\beta)+\lambda g(\beta)\rightarrow \nabla_\beta L = f'(\beta)+\lambda g'(\beta)=0\rightarrow f'(\beta)=-\lambda g'(\beta)$$

which means, the gradients at the solution should be multiples of each other. They point to opposite directions (due to $-\lambda$). This happens at tangent points. Because these gradients are normal vectors of these level curves. But, which tangent to choose depends on the value of $\lambda$. The above plot shows different solutions, i.e. tangent points, depending on the value of $\lambda$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Everything understandably explained - just one question: If $\lambda$ approaches infinity or accepts a very large value, the solution is found at the origin, that is, at (0,0). So, given your equation, both gradients should be very large as well. By looking at your sketch I can not see this property - the gradients of level curves close to the origin look similar as those obtained for larger beta's. So why result larger values of $\lambda$ in solution points closer to the origin? $\endgroup$
    – SolidMechanicsFan
    Commented Jun 4, 2022 at 19:59
  • $\begingroup$ No, both gradients won't be large. Gradient for $g$ will be $0$ and $\lambda g'$ will be $\infty \times 0$ which is undefined. So, the result can be anything. It can be $-f'$ as well. Experimentally, the red points I've plotted above are the solutions I've found by choosing lambda large. $\endgroup$
    – gunes
    Commented Jun 5, 2022 at 7:18
  • $\begingroup$ Makes sense: $\nabla g = 2 \beta$ is zero if beta is zero. But is there an intuitive explanation why the solution is found at the origin if we choose $\lambda$ large enough? $\endgroup$
    – SolidMechanicsFan
    Commented Jun 5, 2022 at 19:04
  • $\begingroup$ Of course. When $\lambda$ is very large, the L2 portion of the loss function gets very large, i.e. $\lambda ||\beta||^2$, and is small only when the L2 norm of $\beta$ is very small, i.e. $0$ when $\lambda$ is infinity. $\endgroup$
    – gunes
    Commented Jun 5, 2022 at 19:37
  • $\begingroup$ So if $\lambda$ is very large, $\lambda \| \boldsymbol \beta \| ^2$ has to go (faster as $\lambda$ increases) to $\boldsymbol 0$ in order to minimize the sum of the L2 portion and the least squares term. Is that true? $\endgroup$
    – SolidMechanicsFan
    Commented Jun 5, 2022 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.