# Why regularization/shrinkage method works for p>n?

I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through them, by ordinary least squares, I can fit numerous planes with 0 residual. In this case, p=3 and n=2.

By implementing penalty, with ridge (L1) and lasso (L2), how are the $\beta$s going to be restricted? I am having difficulties understand these concepts because in both cases, the residual can be 0 in OLS, so when I am adding penalty terms, to minimize the penalized least squares, I only need to set the undetermined coefficients ($\beta$) to be 0.

For example, I want to fit a plane through (0,1,0) and (0,0,1), then the planes are $y = 1+\beta_1x_1-x_2+\epsilon$, with $\beta_1$ undetermined. For lasso, to minimize $[\epsilon^2+\lambda(|\beta_1|+1)]$, I set $\epsilon=0,\beta_1=0$, so that I get only one plane. For ridge, to minimize $[\epsilon^2+\lambda(\beta_1^2+1)]$, I set $\epsilon=0,\beta_1=0$, and I also get only one plane. However, I heard that for ridge regression, coefficient is shrunk towards 0 but not exactly 0, so how does this happen?

• Possibly the question is poorly titled, because the full text indicates the question seems more about "Why does ridge give a 0 coefficient here?". On this point, it is just a quirk of the particular data, which has $x_1=0$ for all points. Commented Apr 18, 2017 at 1:37
• The geometric characterization of Ridge Regression at the end of my post at stats.stackexchange.com/a/164546/919 makes this immediately obvious, because it interprets regularization as formulating a version of the original problem with $n$ replaced by $n+p \gt p.$
– whuber
Commented Sep 29, 2023 at 13:00

You have three parameters in your model: $\beta_0,\beta_1,\beta_2$. As you noted correctly there's an infinite number of planes that go through two points, because you need at least three points to nail the plane.

Now, imagine that you have a very extreme form of shrinkage, where your penalty on the intercept is super crazy high: any deviation from $\beta_0=0$ is heavily penalized. This would basically make you choose $\hat\beta_0=0$. Now you only need to define two remaining parameters, and two observations is just enough to do it. Essentially you forced your plane to go through the origin, i.e. the origin was that missing third point that you needed to nail the plane.

I think that you took the right approach by considering such an extreme case to strip the shrinkage to its bare bones. The shrinkage does exactly what we just did: it shrinks your potential solution space. So, in this case from an infinite number of solutions it shrank the space to exactly one.

• For penalized regression, the intercept $\beta_0$ is not included in the regularization term right? The regularization term has the form $$\lambda\sum_{j=1}^{p} \beta_j^{m}$$ From my understanding, only the coefficients of predictors are penalized and the intercept is not penalized. Commented Apr 18, 2017 at 14:52
• It doesn't matter for the argument, you could fix $\beta_2=0$, then the slope on the second variable is fixed, so you fit all planes with that slope, and there's only one of them left once you plug two observations. Commented Apr 18, 2017 at 15:02

Consider also that $$p = o(C^n), C>1$$ is the threshold growth rate for a deterministic limit

$$\hat\beta_n \rightarrow E[ b | \mathcal{D}_T ] \\ = \mathrm{constant~for~} p=o(C^n)\\ = \mathrm{random~for~} p > O(C^n)$$

where $$\mathcal{D}_T$$ is the training data or in the limiting case, the tail field induced by it.