In the context of trying to understand regularization and how it works for ridge regression vs. lasso regression, I've come across two ideas:
- Both of these methods attempt to improve generalization error by penalizing a model for complexity.
- In Lasso, the coefficients in a model can go to zero, so it operates as a variable selection procedure as well. In Ridge regression on the other hand, the coefficients can be small but can't go all the way to zero.
What I can't understand is how is Ridge Regression penalizing for complexity then? The number of coefficients remains the same, even if the values go down.
Isn't the model $\hat{y}=0.01x_1+2x_2+0.03x_3$ just as complex as $\hat{y}=5x_1+4x_2+7x_3$?
How exactly is complexity being evaluated in the case of ridge regression?