I am having trouble visualizing regularization/shrinkage method for the case of p>n. If I have only two data point, but I want to fit a plane ($y = \beta_0+\beta_1x_1+\beta_2x_2+\epsilon$) through them, by ordinary least squares, I can fit numerous planes with 0 residual. In this case, p=3 and n=2.
By implementing penalty, with ridge (L1) and lasso (L2), how are the $\beta$s going to be restricted? I am having difficulties understand these concepts because in both cases, the residual can be 0 in OLS, so when I am adding penalty terms, to minimize the penalized least squares, I only need to set the undetermined coefficients ($\beta$) to be 0.
For example, I want to fit a plane through (0,1,0) and (0,0,1), then the planes are $y = 1+\beta_1x_1-x_2+\epsilon$, with $\beta_1$ undetermined. For lasso, to minimize $[\epsilon^2+\lambda(|\beta_1|+1)]$, I set $\epsilon=0,\beta_1=0$, so that I get only one plane. For ridge, to minimize $[\epsilon^2+\lambda(\beta_1^2+1)]$, I set $\epsilon=0,\beta_1=0$, and I also get only one plane. However, I heard that for ridge regression, coefficient is shrunk towards 0 but not exactly 0, so how does this happen?