From what I understood from this article, the blue circles are the level curves and the blue dot is the optimal solution that minimizes the cost function. The yellow circle is the L2-norm constraint.

The solution that we need is the one that minimizes the cost function as much as possible and also, at the same time, is within the circle. Meaning, the solution is the tangent point between the yellow circle and the level curve.

But, my question is how this can be the solution if the W values at the tangent point don't completely minimize the cost function? Only the blue dot is the one that minimizes the cost function. enter image description here

  • 2
    $\begingroup$ If you can only select $w$ values from within the yellow circle, then that tangent point is the best (minimal) value that can be achieved. So in this sense it is optimal. $\endgroup$ Jun 16 '18 at 20:03

The point on the circle $w*$ is the optimal parameter value among those that satisfy the constraint.

If I own a plot of land on the side of a hill and I ask what is the highest point on my property? it's no good you pointing over at the top of the hill and saying "it's over there in your neighbor's property".

That's a good answer to a different question than the one that was asked; "my property" is the constraint, you have to find a solution that satisfies it. The point $w$ doesn't satisfy the constraint (it's in my neighbor's yard, not mine, and is no use when I want to know the highest point on my land).

  • $\begingroup$ So, we want to find a balance where the yellow circle(the constraint) and the function we want minimize meet because If they meet near the blue dot, we might get overfitting. The $COST_{ridge}=COST+\lambda\beta^2$. Where do we control the radius of the yellow cricle in this lagrangian? $\endgroup$
    – theateist
    Jun 18 '18 at 18:07
  • $\begingroup$ Note that other questions on site may cover that question. e.g. see here and places linked to from that page. Try a few searches. $\endgroup$
    – Glen_b
    Jun 19 '18 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.