Geometric interpretation of penalized linear regression

Question

I know that linear regression can be thought as "the line that is vertically closest to all the points":

But there is another way to see it, by visualizing the column space, as "the projection onto the space spanned by the columns of the coefficient matrix":

My question is: in these two interpretations, what happens when we use the penalized linear regression, like ridge regression and LASSO? What happens with the line in the first interpretation? And what happens with the projection in the second interpretation?

UPDATE: @JohnSmith in the comments brought up the fact that the penalty occurs in the space of the coefficients. Is there an interpretation in this space also?

Answer 1

Sorry for my painting skills, I will try to give you the following intuition.

Let $f(\beta)$ be the objective function (for example, MSE in case of regression). Let's imagine the contour plot of this function in red (of course we paint it in the space of $\beta$, here for simplicity $\beta_1$ and $\beta_2$).

There is a minimum of this function, in the middle of the red circles. And this minimum gives us the non-penalized solution.

Now we add different objective $g(\beta)$ which contour plot is given in blue. Either LASSO regularizer or ridge regression regularizer. For LASSO $g(\beta) = \lambda (|\beta_1| + |\beta_2|)$, for ridge regression $g(\beta) = \lambda (\beta_1^2 + \beta_2^2)$ ($\lambda$ is a penalization parameter). Contour plots shows the area at which the function have the fixed values. So the larger $\lambda$ - the faster $g(x)$ growth, and the more "narrow" the contour plot is.

Now we have to find the minimum of the sum of this two objectives: $f(\beta) + g(\beta)$. And this is achieved when two contour plots meet each other.

The larger penalty, the "more narrow" blue contours we get, and then the plots meet each other in a point closer to zero. An vise-versa: the smaller the penalty, the contours expand, and the intersection of blue and red plots comes closer to the center of the red circle (non-penalized solution).

And now follows an interesting thing that greatly explains to me the difference between ridge regression and LASSO: in case of LASSO two contour plots will probably meet where the corner of regularizer is ($\beta_1 = 0$ or $\beta_2 = 0$). In case of ridge regression that is almost never the case.

That's why LASSO gives us sparse solution, making some of parameters exactly equal $0$.

Hope that will explain some intuition about how penalized regression works in the space of parameters.

Answer 2

The intuition I have is the following: In the least-squares case, the hat matrix is an orthogonal projection thus idempotent. In the penalized case, the hat matrix is no longer idempotent. Actually, applying it infinitely many times, will shrink the coefficients to the origin. On the other hand, the coefficients still have to lie in the span of the predictors, so it is still a projection, albeit not orthogonal. The magnitude of the penalizing factor and the type of norm control the distance and direction of shrinkage towards the origin.

Answer 3

Second image

The geometric interpretation in the second figure will look as following:

With OLS the observations are projected onto a surface spanned by the regressor variables and this finds the point in the plane that has the shortest distance to the observed vector $(y_1,y_2,y_3)$.

With ridge regression we do not look for a closest point in the entire plane but for a closest point that is inside a circle on the plane (or sphere in higher dimensions and other shape for different types of regression).

Image from the question Why does regularization wreck orthogonality of predictions and residuals in linear regression?

First image

Regularisation will reduce the size of the parameters. So, the fitted line will have a smaller intercept and a smaller slope. In general this will mean that the line will be more flat and closer to the x-axis.