I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through them, by ordinary least squares, I can fit numerous planes with 0 residual. In this case, p=3 and n=2.

By implementing penalty, with ridge (L1) and lasso (L2), how are the $\beta$s going to be restricted? I am having difficulties understand these concepts because in both cases, the residual can be 0 in OLS, so when I am adding penalty terms, to minimize the penalized least squares, I only need to set the undetermined coefficients ($\beta$) to be 0.

For example, I want to fit a plane through (0,1,0) and (0,0,1), then the planes are $y = 1+\beta_1x_1-x_2+\epsilon$, with $\beta_1$ undetermined. For lasso, to minimize $[\epsilon^2+\lambda(|\beta_1|+1)]$, I set $\epsilon=0,\beta_1=0$, so that I get only one plane. For ridge, to minimize $[\epsilon^2+\lambda(\beta_1^2+1)]$, I set $\epsilon=0,\beta_1=0$, and I also get only one plane. However, I heard that for ridge regression, coefficient is shrunk towards 0 but not exactly 0, so how does this happen?

  • $\begingroup$ Possibly the question is poorly titled, because the full text indicates the question seems more about "Why does ridge give a 0 coefficient here?". On this point, it is just a quirk of the particular data, which has $x_1=0$ for all points. $\endgroup$ – GeoMatt22 Apr 18 '17 at 1:37

You have three parameters in your model: $\beta_0,\beta_1,\beta_2$. As you noted correctly there's an infinite number of planes that go through two points, because you need at least three points to nail the plane.

Now, imagine that you have a very extreme form of shrinkage, where your penalty on the intercept is super crazy high: any deviation from $\beta_0=0$ is heavily penalized. This would basically make you choose $\hat\beta_0=0$. Now you only need to define two remaining parameters, and two observations is just enough to do it. Essentially you forced your plane to go through the origin, i.e. the origin was that missing third point that you needed to nail the plane.

I think that you took the right approach by considering such an extreme case to strip the shrinkage to its bare bones. The shrinkage does exactly what we just did: it shrinks your potential solution space. So, in this case from an infinite number of solutions it shrank the space to exactly one.

  • $\begingroup$ For penalized regression, the intercept $\beta_0$ is not included in the regularization term right? The regularization term has the form $$ \lambda\sum_{j=1}^{p} \beta_j^{m}$$ From my understanding, only the coefficients of predictors are penalized and the intercept is not penalized. $\endgroup$ – S.Wang Apr 18 '17 at 14:52
  • $\begingroup$ It doesn't matter for the argument, you could fix $\beta_2=0$, then the slope on the second variable is fixed, so you fit all planes with that slope, and there's only one of them left once you plug two observations. $\endgroup$ – Aksakal Apr 18 '17 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.