Why regularization/shrinkage method works for p>n?

Question

I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through them, by ordinary least squares, I can fit numerous planes with 0 residual. In this case, p=3 and n=2.

By implementing penalty, with ridge (L1) and lasso (L2), how are the $\beta$s going to be restricted? I am having difficulties understand these concepts because in both cases, the residual can be 0 in OLS, so when I am adding penalty terms, to minimize the penalized least squares, I only need to set the undetermined coefficients ($\beta$) to be 0.

For example, I want to fit a plane through (0,1,0) and (0,0,1), then the planes are $y = 1+\beta_1x_1-x_2+\epsilon$, with $\beta_1$ undetermined. For lasso, to minimize $[\epsilon^2+\lambda(|\beta_1|+1)]$, I set $\epsilon=0,\beta_1=0$, so that I get only one plane. For ridge, to minimize $[\epsilon^2+\lambda(\beta_1^2+1)]$, I set $\epsilon=0,\beta_1=0$, and I also get only one plane. However, I heard that for ridge regression, coefficient is shrunk towards 0 but not exactly 0, so how does this happen?

Answer 1

You have three parameters in your model: $\beta_0,\beta_1,\beta_2$. As you noted correctly there's an infinite number of planes that go through two points, because you need at least three points to nail the plane.

Now, imagine that you have a very extreme form of shrinkage, where your penalty on the intercept is super crazy high: any deviation from $\beta_0=0$ is heavily penalized. This would basically make you choose $\hat\beta_0=0$. Now you only need to define two remaining parameters, and two observations is just enough to do it. Essentially you forced your plane to go through the origin, i.e. the origin was that missing third point that you needed to nail the plane.

I think that you took the right approach by considering such an extreme case to strip the shrinkage to its bare bones. The shrinkage does exactly what we just did: it shrinks your potential solution space. So, in this case from an infinite number of solutions it shrank the space to exactly one.

Answer 2

Consider also that $p = o(C^n), C>1$ is the threshold growth rate for a deterministic limit

$\hat\beta_n \rightarrow E[ b | \mathcal{D}_T ] \\ = \mathrm{constant~for~} p=o(C^n)\\ = \mathrm{random~for~} p > O(C^n)$

where $\mathcal{D}_T$ is the training data or in the limiting case, the tail field induced by it.