Geometric interpretation of penalized linear regression

Question

I know that linear regression can be thought as "the line that is vertically closest to all the points":

But there is another way to see it, by visualizing the column space, as "the projection at the space spanned by the columns of the coefficient matrix":

My question is: in these two interpretations, what happens when we use the penalized linear regression, like ridge regression and LASSO? What happens with the line in the first interpretation? And what happens with the projection in the second interpretation?

UPDATE: @JohnSmith in the comments brought up the fact that the penalty occurs in the space of the coefficients. Can we come up with an interpretation in this space also?

Answer 1

Sorry for my painting skills, I will try to give you the following intuition.

Let $f(\beta)$ be the objective function (for example, MSE in case of regression). Let's imagine the contour plot of this function in red (of course we paint it in the space of $\beta$, here for simplicity $\beta_1$ and $\beta_2$).

There is a minimum of this function, in the middle of the red circles. And this minimum gives us the non-penalized solution.

Now we add different objective $g(\beta)$ which contour plot is given in blue. Either LASSO regularizer or ridge regression regularizer. For LASSO $g(\beta) = \lambda (|\beta_1| + |\beta_2|)$, for ridge regression $g(\beta) = \lambda (\beta_1^2 + \beta_2^2)$ ($\lambda$ is a penalization parameter). Contour plots shows the area at which the function have the fixed values. So the larger $\lambda$ - the faster $g(x)$ growth, and the more "narrow" the contour plot is.

Now we have to find the minimum of the sum of this two objectives: $f(\beta) + g(\beta)$. And this is achieved when two contour plots meet each other.

The larger penalty, the "more narrow" blue contours we get, and then the plots meet each other in a point closer to zero. An vise-versa: the smaller the penalty, the contours expand, and the intersection of blue and red plots comes closer to the center of the red circle (non-penalized solution).

And now follows an interesting thing that greatly explains to me the difference between ridge regression and LASSO: in case of LASSO two contour plots will probably meet where the corner of regularizer is ($\beta_1 = 0$ or $\beta_2 = 0$). In case of ridge regression that is almost never the case.

That's why LASSO gives us sparse solution, making some of parameters exactly equal $0$.

Hope that will explain some intuition about how penalized regression works in the space of parameters.

Answer 2

The intuition I have is the following: In the least-squares case, the hat matrix is an orthogonal projection thus idempotent. In the penalized case, the hat matrix is no longer idempotent. Actually, applying it infinitely many times, will shrink the coefficients to the origin. On the other hand, the coefficients still have to lie in the span of the predictors, so it is still a projection, albeit not orthogonal. The magnitude of the penalizing factor and the type of norm control the distance and direction of shrinkage towards the origin.