How do Shrinkage Methods change flexibility of a model?

Question

While working through An Introduction to Statistical Learning, I had difficulty clarifying how flexibility relates to Ridge Regression and Lasso. I recognize that both impose penalties on the coefficient estimates using either L1 or L2 norms, but how do they ultimately affect the flexibility of the model?

Answer 1

The term "model complexity" or "high variance model" are abstract statements to say the model has "many parameters" and can do some complicated tasks (regression or classification). For example, a linear model is not as complex as a 5th order polynomial model.

Here I will explain why we need shrinkage/ regularization first then explain why adding L1 and L2 norm constraints can reduce the model complexity.

Part 1.

Suppose you have some data, the data is not complex, do you want fit it with a simple model? Or fit with with a complex model but regularize it.

Please check following demo I am fitting a line, 2nd order, 3rd order, and 5 order polynomial to same data. You can clear see, a line is under fitting and 5 order is over fitting. A good fit may be a 3rd order.

BUT, I can do 5 order with regularization / shrinkage !! The following curve is 5th order, but with regularization. you can observe we controlled the over fitting and the model complexity close to 3rd order, but more "flexible" than 3rd order at same time not too over fit.

The idea is using complicate model with regularization would be better than using simple model, in many modern data.

Part 2.

Intuitively you can use the "degree of freedom" idea to think about why adding L1 or L2 constraints can reduce the model complexity. Please check following 2d example, without constraint the weights can be any value, but with L2 constraint, the weights must be in a circle and with L1 constraint (shown in green), the weights must be in a diamond (shown in red).

Now come back to the polynomial fit example, for 5th order model, there are many parameters, but we limit the values of the parameters to be certain range. That is why the model can act as simpler model.

Answer 2

LASSO and ridge regression are typically written in the Lagrangian form, with a penalty on the $l_1$ or squared $l_2$ norms. But, there's an equivalent form with a constraint on the norms instead of a penalty. For ridge regression:

$$\underset{w}{\min} \|y - Xw\|_2^2 \quad \text{s.t.} \quad \|w\|_2^2 \le c$$

For LASSO:

$$\underset{w}{\min} \|y - Xw\|_2^2 \quad \text{s.t.} \quad \|w\|_1 \le c$$

We can interpret these constraints geometrically, by thinking of the weight vector as a point in the space of all possible choices of weights. In the case of ridge regression, the constraint $\|w\|_2^2 \le c$ means that the weight vector $w$ is restricted to lie within a hypersphere of radius $\sqrt{c}$. Similarly, the $l_1$ constraint in LASSO means that the weights are restricted to lie within a polytope whose size scales with $c$ (the vertices of the polytope lie along the axes and $c$ gives the distance of each vertex from the origin).

Informally, a more flexible model is able to represent a wider variety of functional forms. Each form corresponds to a particular choice of parameters. For LASSO and ridge regression, we can see that the set of allowable parameters shrinks as we decrease $c$, because the hypersphere/polytope becomes smaller. Decreasing $c$ corresponds to increasing the penalty term (often called $\lambda$) in the Lagrangian form. Therefore, tightening the constraint (or increasing the penalty) corresponds to decreasing the model flexibility.