# Geometric interpretation of penalized linear regression

I know that linear regression can be thought as "the line that is vertically closest to all the points":

But there is another way to see it, by visualizing the column space, as "the projection at the space spanned by the columns of the coefficient matrix":

My question is: in these two interpretations, what happens when we use the penalized linear regression, like ridge regression and LASSO? What happens with the line in the first interpretation? And what happens with the projection in the second interpretation?

UPDATE: @JohnSmith in the comments brought up the fact that the penalty occurs in the space of the coefficients. Can we come up with an interpretation in this space also?

-
I am not sure that it is possible to come up with such an interpretation. Simply because what you provided are images in the original space of features and responses. And penalized regression involves the space of coefficients, which is very different. –  Dmitry Laptev Jun 14 '12 at 16:02
I will update my question, @JohnSmith. Thanks! –  Lucas Reis Jun 14 '12 at 16:30

Sorry for my painting skills, I will try to give you the following intuition.

Let $f(\beta)$ be the objective function (for example, MSE in case of regression). Let's imagine the contour plot of this function in red (of course we paint it in the space of $\beta$, here for simplicity $\beta_1$ and $\beta_2$).

There is a minimum of this function, in the middle of the red circles. And this minimum gives us the non-penalized solution.

Now we add different objective $g(\beta)$ which contour plot is given in blue. Either LASSO regularizer or ridge regression regularizer. For LASSO $g(\beta) = \lambda (|\beta_1| + |\beta_2|)$, for ridge regression $g(\beta) = \lambda (\beta_1^2 + \beta_2^2)$ ($\lambda$ is a penalization parameter). Contour plots shows the area at which the function have the fixed values. So the larger $\lambda$ - the faster $g(x)$ growth, and the more "narrow" the contour plot is.

Now we have to find the minimum of the sum of this two objectives: $f(\beta) + g(\beta)$. And this is achieved when two contour plots meet each other.

The larger penalty, the "more narrow" blue contours we get, and then the plots meet each other in a point closer to zero. An vise-versa: the smaller the penalty, the contours expand, and the intersection of blue and red plots comes closer to the center of the red circle (non-penalized solution).

And now follows an interesting thing that greatly explains to me the difference between ridge regression and LASSO: in case of LASSO two contour plots will probably meet where the corner of regularizer is ($\beta_1 = 0$ or $\beta_2 = 0$). In case of ridge regression that is almost never the case.

That's why LASSO gives us sparse solution, making some of parameters exactly equal $0$.

Hope that will explain some intuition about how penalized regression works in the space of parameters.

-
I think starting with a classical picture, as you've done, is a good start. To really understand this, I think it would be helpful to describe how the contours relate to the problem. In particular, we know in both cases, that the smaller we make our penalty, the closer we'll get to the OLS solution, and the larger it gets, the closer to a pure-intercept model we'll get. One question to ask is: How does this manifest itself in your figure? –  cardinal Jun 15 '12 at 14:37
By the way, your painting skills seem just fine. –  cardinal Jun 15 '12 at 14:41
Thanks for your comment! Everything is intuitively simple here: the larger penalty, the "more narrow" blue contours we get (and then the point two plots meet come closer to zero). An vise-versa: the smaller the penalty: the closer to the center of the red circle the plots will meet (OLS). –  Dmitry Laptev Jun 15 '12 at 14:43
You're getting closer to my intended point: Relating the contours to the penalty in the prose of the answer would help communicate what exactly you're showing and how it changes as the penalization varies. It may not be clear to others that the admissible contours expand as the penalty shrinks. Also, something slightly different happens once the admissible contours overtake the center point of the ellipsoid. –  cardinal Jun 15 '12 at 14:51
I updated the answer, introduced a more exact notation: $g(x)$ for the blue plots, $\lambda$ for the penalization paramater. Does that explain your concern better now? –  Dmitry Laptev Jun 15 '12 at 15:07